KR102460671B1 - Adaptation method of sensory effect, and adaptation engine and sensory device to perform it - Google Patents

Abstract

A method for sensory effect adaptation performed by an adaptation engine, the method comprising: identifying first metadata associated with an object in a virtual world for describing the object; converting the first metadata related to the identified object into second metadata for application to a sensory device in the real world, wherein the second metadata includes: The sensory effect adaptation method may be converted based on a scene determined according to the gaze of the user in the virtual world.

Description

Sensory effect adaptation method and adaptive engine and sensory device performing the same

The following embodiments relate to a sensory effect adaptation method, an adaptation engine for performing the same, and a sensory device, and more particularly, to a sensory effect adaptation technology based on a scene viewed by a user in a virtual world and an object included in the scene. .

In order to provide realistic multimedia content, 4D movie theaters provide sensory effect services that stimulate the five senses, such as smell and touch. When such sensory effects are provided together, user satisfaction and immersion may be increased.

A method in which a preset sensory effect is provided according to time (for example, a conventional 4D cinema provides a preset sensory effect corresponding to the time of the video being played regardless of the user's gaze) is the user's gaze in the virtual world. Regardless, the sensory effect may be provided, which may lower user satisfaction and immersion.

Therefore, the scene viewed by the user changes according to the user's gaze in the virtual world, and there is a need for research on a sensory effect service in which a sensory effect corresponding to the changed scene is provided to the user.

According to an embodiment, it may be possible to reproduce a sensory effect according to a change in the gaze of the user in the virtual world.

According to an embodiment, a sensory effect may be reproduced according to a user's preference, environment information, or performance of a sensory device.

According to one aspect, there is provided a sensory effect adaptation method performed by an adaptation engine, the method comprising: identifying first metadata related to an object in a virtual world for describing the object; converting the first metadata related to the identified object into second metadata for application to a sensory device in the real world, wherein the second metadata includes: The sensory effect adaptation method may be converted based on a scene determined according to the gaze of the user in the virtual world.

The second metadata may be a sensory effect adaptation method that is converted in consideration of not only the scene determined according to the user's gaze, but also the user's environment information, the user's preference, sensory device performance, and sensor performance. .

The converting into the second metadata may include: generating a control command XML for controlling the sensory device; generating a SyncTable for synchronizing the time for the video and the time for the control command XML; The method may further include transmitting the control command XML to a sensory device when the control command XML corresponding to the time of the reproduced image is found in the SyncTable.

According to one aspect, there is provided a sensory effect adaptation method performed by a sensory device, the method comprising: identifying control command XML generated based on second metadata converted from first metadata; parsing the identified control command XML to generate a binary-type control command, and controlling the sensory device using the generated binary-type control command, and the second metadata may be a sensory effect adaptation method in which the first metadata related to the object for describing the object in the virtual world is converted based on a scene determined according to the gaze of the user in the virtual world.

The second metadata may be a sensory effect adaptation method that is converted in consideration of not only the scene determined according to the user's gaze, but also the user's environment information, the user's preference, sensory device performance, and sensor performance. .

An adaptation engine, comprising: a processor, configured to: identify first metadata associated with the object for describing an object in a virtual world; and generate the first metadata associated with the identified object converted into second metadata for application to a sensory device in the real world, wherein the second metadata is converted based on a scene in which the first metadata is determined according to a user's gaze in the virtual world It can be an adaptive engine that becomes

The second metadata is an adaptation engine that is converted in consideration of not only the scene determined according to the user's gaze, but also the surrounding environment information of the user's real world, the user's preference, sensory device performance, and sensor performance. can

The processor generates a control command XML for controlling the sensory device, generates a SyncTable for synchronizing the time for the video and the time for the control command XML, and sets the time of the video to be played back. When the corresponding control command XML is retrieved from the SyncTable, the adaptive engine may further transmit the control command XML to a sensory device.

According to one aspect, in a sensory device, the sensory device includes a processor, wherein the processor identifies a control command XML generated based on second metadata converted from first metadata; Parsing the identified control command XML to generate a binary type control command, the sensory device is controlled using the generated binary type control command, and the second metadata describes an object of a virtual world. The sensory device may be a sensory device in which the first metadata related to the object to be performed is converted based on a scene determined according to a user's gaze in the virtual world.

The second metadata may be a sensory device that is converted in consideration of not only the scene determined according to the user's gaze, but also the user's real world surrounding environment information, the user's preference, sensory device performance, and sensor performance.

According to one aspect, in a computer-readable recording medium recording data for sensory effects, the data includes information related to an object for describing an object in a virtual world, and the information related to the object includes: i) an object ID flag indicating the existence of an object ID, and (ii) an object ID for identifying an object in the virtual world. A list of sensory effects related to the object identified using the object ID is included in the sensory effect list, , Whether to use the sensory effects included in the sensory effect list may be a recording medium that is determined through a sensory effect list flag.

When the object ID flag is 1, it may indicate that an object corresponding to the object ID exists, and when the object ID flag is 0, it may be a recording medium indicating that an object corresponding to the object ID does not exist.

According to one aspect, in a computer-readable recording medium recording data for sensory effects, the data includes a sensory effect list for describing an object in a virtual world, and the sensory effect The list may indicate a list of sensory effects related to the object in the virtual world, and whether to use the sensory effects included in the sensory effect list may be a recording medium including a determination through a sensory effect list flag.

When the sensory effect list flag is 1, the sensory effect included in the sensory effect list is used, and when the sensory effect list flag is 0, the sensory effect included in the sensory effect list is not used. It may be a recording medium.

1 shows the overall system architecture of the MPEG-V framework, according to an embodiment.
2 shows an embodiment of reproducing a motion chair according to an embodiment.
3A illustrates the use of an Arrayed Light effect simulating a fireworks visual effect in conjunction with a detonation event occurring in a movie with two Arrayed Light actuators in a 4D theater, according to one embodiment. show examples.
3B illustrates a use case of an Arrayed Light effect for visually augmenting a vehicle passing through a tunnel with color lighting in a 4D theater according to an embodiment.
4 illustrates information adaptation from a virtual world to a real world according to an embodiment.
5 illustrates information adaptation from the real world to the virtual world according to an embodiment.
6 illustrates information exchange between virtual worlds according to an embodiment.
7 illustrates a sensory effect authoring apparatus for authoring sensory effects, an adaptation engine for converting authored sensory effects to be reproduced in the sensory apparatus, and a sensory apparatus for reproducing sensory effects, according to an embodiment.
8 illustrates that sensory device performance, user preference, sensor performance, and environment information are considered in the adaptation engine, according to an embodiment.
9 is a diagram illustrating an XML representation syntax according to an embodiment.
10 is a diagram illustrating an XML representation syntax for an identification type according to an embodiment.
11 is a diagram illustrating an XML representation syntax for a VWO sound list type according to an embodiment.
12 is a diagram illustrating an XML representation syntax for a VWO fragrance list type according to an embodiment.
13 is a diagram illustrating an XML representation syntax for a VWO control list type according to an embodiment.
14 is a diagram illustrating an XML representation syntax for a VWO event list type according to an embodiment.
15 is a diagram illustrating an XML representation syntax for a VWO behavior model list type according to an embodiment.
16 is a diagram illustrating an XML representation syntax for a VWO sound type according to an embodiment.
17 is a diagram illustrating an XML representation syntax for a VWO fragrance type according to an embodiment.
18 is a diagram illustrating an XML representation syntax for a VWO control type according to an embodiment.
19 is a diagram illustrating an XML representation syntax for a VWO event type according to an embodiment.
20 is a diagram illustrating an XML representation syntax for a VWO immersive effect type according to an embodiment.
21 is a diagram illustrating an XML representation syntax for a VWO behavior model type, according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” to another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<abbreviated terms>

CT - common types

DIA - digital item adaptation (ISO/IEC 21000-7)

SEDL - sensory effects description language

SEM - sensory effect metadata

SEV - sensory effects vocabulary

UMA - universal multimedia access

UME - universal multimedia experience

REL - rights expression language

URI - Uniform Resource Identifier

URL - Uniform Resource Locator

XML - Extensible Markup Language

XSI - XML streaming instructions

<terms and definitions>

Device Command - A description of what controls the actuator used to create the sensory effect.

Sensed Information - Information obtained from sensors

Sensor - A device capable of collecting user input or environmental information. For example, temperature sensor, distance sensor, motion sensor, etc.

Sensor Adaptation Preferences - Descriptor and description schemes for indicating (user's) preferences with respect to adapting sensed information.

Sensor Capability - A description that characterizes a sensor in terms of its performance, such as accuracy or detection range.

Sensory Device - A consumer device on which the corresponding sensory effect can be created. Real-world devices may include all combinations of sensors and actuators in one device.

Sensory Device Capability - A description that describes the characteristics of the actuators used to generate sensory effects in terms of the capabilities of a given device.

Sensory effect - An effect that enhances perception by stimulating human senses in a specific scene. For example, fragrance, wind, lighting, haptic [kinesthetic-force, stiffness, weight, friction, texture, widget (e.g. button, slider, joystick, etc.), tactile: air-jet, suction pressure, thermal, current, vibration, etc.]

Sensory Effect Metadata - Metadata defining description schemes and descriptors representing sensory effects.

User's Sensory Preferences - A descriptive scheme and descriptor for expressing (user's) preferences with respect to rendering of sensory effects.

<Use of prefixes>

"xsi:" represents an element in the http://www.w3.org/2001/XMLSchema-instance namespace.

The prefix "xml:" binds to http://www.w3.org/XML/1998/namespace by definition. The prefix "xmlns:" is only used for namespace binding, not itself binding to a namespace name.

<System Architecture>

1 shows an overall system architecture of an MPEG-V framework, according to an embodiment.

The system architecture of the MPEG-V framework can be used to provide three media exchanges. There are two types of media exchange occurring between the real world and the virtual world: information exchange from the real world to the virtual world and information exchange from the virtual world to the real world. Another type of media exchange is the exchange of information between virtual worlds.

Adaptation Engine uses SEM (Sensory Effect Metadata: Light effect, Flash effect, Temperature effect, Wind effect, Vibration effect, Spraying effect, Scent effect, Fog effect, Color correction effect, Rigid body motion effect, Passive kinesthetic motion effect, Passive kinesthetic force effect, Active kinesthetic effect, Tactile effect, Parameterized tactile effect, Bubble effect, Arrayed Light Effect) may be provided.

According to an embodiment, the Adaptation Engine may use SEMBaseType as shown in Table 1 below to provide SEM for the expression of 18 sensory effects.

<complexType name="SEMBaseType"abstract="true">
<complexContent>
<restrictionbase="anyType">
<attributename="id"type="ID"use="optional"/>
</restriction>
</complexContent>
</complexType>

According to SEMBaseType, 18 sensory effects can be distinguished by ID. Accordingly, sensory effects corresponding to the ID may be activated and/or deactivated based on the time the media is played. For example, a lighting effect corresponding to id="Light1" may be activated and/or deactivated while the media is playing.

In this case, one ID may represent one sensory effect, and different IDs may be used to represent several identical sensory effects or multiple different sensory effects. Also, when two or more identical sensory effects are expressed like each other sensory effect, two or more IDs may be used. For example, as shown in Table 2 below, sensory effects can be distinguished by ID, and sensory effects corresponding to a predetermined condition (eg, duration, activate, location, intensity-value, intensity-range, etc.) at a predetermined time. effect can be expressed.

<sedl:Effect xsi:type="sev:TemperatureType" id=“Temp1” si:pts="8100"duration="2000"activate="True"location=":WCS:Left:Top:Back" intensity-value ="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:LightType" id=“Light1” si:pts="8100"duration="3000"activate="True"location=":WCS:Left:Top:Back"color="#FF0000"intensity-value="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:LightType" id=“Light2” si:pts="16750"duration="2000"activate="True"location=":WCS:Left:Top:Front"color="#0000FF"intensity-value="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:FlashType" id=“Flash1” si:pts="16750"duration="1000"activate="True"location=":WCS:Left:Top:Back"frequency="255"color="#FFFFFF"intensity-value="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:ScentType" id=“Scent1” si:pts="7900"duration="3000"activate="True"location=":WCS:Left:Top:Back"scent=":SCENT:rose"intensity-value="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:AirjetType" id=“Air1” si:pts="5050"duration="500"activate="True"location=":WCS:Left:Top:Back" intensity-value ="100"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:WindType" id=“Wind1” si:pts="1500"duration="3000"activate="True"location=":WCS:CenterRight:Middle:Front" intensity-value ="55"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:VibrationType" id=“Vib1” si:pts="9100"duration="3000"activate="True"location=":WCS:Center:Middle:Midway" intensity-value ="50"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:WaterSprayerType" id=“Water1” si:pts="14100"duration="300"activate="True"location=":WCS:CenterRight:Bottom:Front" intensity-value ="30"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:FogType" id=“Fog1” si:pts="0"duration="3000"activate="True"location=":WCS:Center:Middle:Front" intensity-value ="50"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:BubbleType" id=“Bubble1” si:pts="7600"duration="5000"activate="True"location=":WCS:Center:Bottom:Front" intensity-value ="250"intensity-range="0100"/>
<sedl:Effectxsi:type="sev:ArrayedLightType"updateRate="2"activate="true"si:pts="2">
<sev:LightSamplesmpeg7:dim="612">
#000000#0000FF#0000FF#0000FF#0000FF#000000
#000000#0000FF#000000#000000#000000#000000
#000000#0000FF#0000FF#0000FF#0000FF#000000
#000000#0000FF#000000#000000#000000#000000
#000000#0000FF#000000#000000#000000#000000
#000000#0000FF#0000FF#0000FF#0000FF#000000
#0000FF#0000FF#0000FF#0000FF#000000#000000
#0000FF#000000#000000#000000#000000#000000
#0000FF#0000FF#0000FF#0000FF#000000#000000
#0000FF#000000#000000#000000#000000#000000
#0000FF#000000#000000#000000#000000#000000
#0000FF#000000#0000FF#0000FF#000000#000000
</sev:LightSamples>
</sedl:Effect>

According to an embodiment, a sensory effect related to a scene viewed by a user may be reproduced by using information on time when media is played and information on objects inside a scene. That is, when a user in virtual reality or augmented reality is watching a specific scene from among the 360-degree images, the sensory effect can be provided using the time the image is played and the object included in the scene the user is looking at. have.

For example, in a situation in which a user in virtual reality or augmented reality is looking at the front from among the 360-degree images, it is not desirable to reproduce the 3D lighting effect on the left side of the user's view when considering the user's current view. not. For another example, if a user in virtual reality or augmented reality is watching a scene in which a ski jump is being directed, it is preferable to move a motion chair capable of expressing a ski jump according to the scene.

According to an embodiment, ID and object reference information for expressing sensory effects may be expressed as shown in Table 3 below. Here, the reference information of the object may include "objectIDRef". The ObjectIDRef indicates an ID reference of another effect referenced with respect to a sensory effect. As an example, the ObjectIDRef may have trajectory information related to the ID of the current sensory effect as a reference.

<complexType name="SEMBaseType"abstract="true">
<complexContent>
<restrictionbase="anyType">
<attributename="id"type="ID"use="optional"/>
<attributename="objectIdRef"type="anyURI"use="optional"/>
</restriction>
</complexContent>
</complexType>

When the scene is continuous, information about the trajectory in which the object moves can be set in advance. And, it is possible to connect the current sensory effect and its trajectory information. If the trajectory is partially overlapped in the screen area of the user's gaze, the trajectory data may be utilized so that the sensory effect can be changed to a sensory effect corresponding to the direction in which the user's gaze points. That is, when sensory effect metadata for a motion chair is generated from an image obtained from a 360-degree camera or a 360-degree VR camera, the motion of the motion chair may vary according to a scene entering the user's gaze. To this end, information about the trajectory in which the object moves may be additionally stored.

For example, FIG. 2 shows an embodiment of a motion chair reproduction according to an embodiment. Information on the trajectory for the movement of the object performing ski jumping shown in FIG. 2 may be stored along the movement trajectory of the object. However, when the user's gaze is looking to the left, the motion of the motion chair may be controlled using information about the trajectory that is modified according to the user's left gaze. That is, the information on the trajectory may be modified according to the user's gaze, and the movement of the motion chair may be controlled by the modified information.

<Arrayed light effects>

From the point of view of rich media, photorealistic media coupled/assisted with the target device is very useful to the user as the media consumption experience can be greatly improved. 4D cinema can be enriched by connecting media and devices together. Multiple devices can be used to enhance the realistic experience of simulating the effects of lighting, wind, flash, etc.

When creating a 4D movie, light effects can be used to represent the atmosphere or warmth of a space, but they represent complex light sources with an arrangement of multiple light display actuators to complement the events of the story at a much higher resolution. An arrayed light effect may be used.

3A illustrates a use case of an arranged lighting effect simulating a fireworks visual effect in conjunction with a detonation event occurring in a movie with two arranged lighting actuators in a 4D theater, according to one embodiment. At this time, the 4D theater provides a large main screen in front of the 4D chair, and two lighting display actuators are installed on the left and right.

While the fireworks video is played on the large main screen, the arranged lighting effects displayed in a sequence of an m by n matrix are simulated by a lighting display actuator according to the timeline of the movie content. The source of the arranged lighting effect may be an m by n matrix mapped to a lighting display actuator or a lighting sequence consisting of a video or image file consisting of an m by n matrix.

Therefore, when a wall or space is configured as a light wall using a sequence of lights instead of using single or multiple lights to reproduce the sensory effect, the lighting effect changes according to the gaze of the user in the virtual world. can be In this process, the surrounding lighting sequence may be changed according to the movement of the object included in the scene the user is viewing in the image or the change in the color of the object. That is, the arranged lighting effect may be composed of a sequence of RGB matrices.

3B illustrates a use case of arranged lighting effects for visually augmenting a vehicle passing through a tunnel with color lighting in a 4D theater according to one embodiment. When a car passes through a tunnel in red, blue and green tones, the effect editor can add arranged lighting effects so that you can experience the same lighting effect as the car's occupants.

Arranged lighting effect metadata may be authored by editors who are unaware of the number or capabilities of lighting display actuators installed in the theater. The adaptation engine will then generate device commands for actuating the appropriate lighting actuators by analyzing and extending the arranged lighting effect metadata.

For example, the editor may author the arranged lighting effect metadata without being aware that the number of lighting display actuators installed in a 4D theater is four. As an example, an editor may create an arranged lighting effect metadata assuming the number of lighting display actuators is two.

Accordingly, the adaptation engine can reproduce an optimized image in a theater in which four lighting display actuators are installed by analyzing and converting the arranged lighting effect metadata generated by the author.

According to an embodiment, the syntax of the arranged lighting effect is shown in Table 4 below.

According to an embodiment, a syntax for the performance of the arranged lighting devices is shown in Table 5 below.

According to an embodiment, a syntax for user preference of arranged lighting is shown in Table 6 below.

According to one embodiment, the syntax (Syntax) for the device control command of the arranged lighting is shown in Table 7 below.

<Three types of media exchange>

i) Adaptation of information from the virtual world to the real world

4 illustrates information adaptation from a virtual world to a real world according to an embodiment.

Sensory effect metadata, VW object characteristics, sensory device capability (actuator capability), device commands, user's sensory preferences), and a V => R adaptation engine that generates output data based on input data.

The virtual world within the framework may be an entity that acts as a source of sensory effect metadata and VW object properties, such as broadcasters, content creators/distributors or service providers. V => R Adaptation Engine accepts sensory effect metadata, sensory device (actuator) performance, and sensory preferences of the user as inputs, and generates device commands based on these inputs to enable a valuable and informative experience for the consumer. The entity that controls the device.

A real-world device (sensory device) is an entity that acts as a sink of device commands and acts as a source of sensory device (actuator) performance. Also, entities that provide a user's sensory preferences for the RoSE engine are collectively referred to as real-world devices. Sensory devices (actuators) are a subset of real-world devices, including fans, lights, wcent devices, and human input devices such as TV sets with remote controls (e.g. preferences). set).

Actual sensory effect metadata provides a means for representing the so-called sensory effect, that is, an effect of augmenting a feeling by stimulating human sensory organs in a specific scene of a multimedia application. Examples of sensory effects are fragrance, wind, lighting, etc. A means of transmitting this kind of metadata is called a sensory effect delivery format, and of course an audio/visual delivery format such as MPEG-2 transport stream, file format, Real-time Transport Protocol (RTP) payload format, etc. can be combined with

The sensory device capability defines a sensory effect representing the characteristics of a sensory device (actuator) and a description format representing a performance method. A sensory device (actuator) is a consumer device capable of producing a corresponding sensory effect (eg, lighting, fan, heater, etc.). Device commands are used to control sensory devices (actuators). As for sensory effect metadata, also for sensory device (actuator) capabilities and device commands, the corresponding means for carrying these assets are referred to as sensory device capability/commands delivery forma, respectively. do.

Finally, the sensory preference of the user makes it possible to describe the preference of the real (end) user with respect to the rendering of the sensory effect to the delivery format.

ii) Adaptation of information from the real world to the virtual world

5 illustrates information adaptation from the real world to the virtual world according to an embodiment.

VW represents an R2V adaptation comprising an R => V adaptation engine that generates output data based on VW object properties, sensed information, sensor performance, sensor adaptation preferences, and input data.

an entity that processes information sensed in the real world for consumption in the context of the virtual world; an entity that retrieves sensed information with or without sensor performance from a sensor, sensor adaptation preferences from a user and/or virtual world object properties from the virtual world; An entity that controls virtual world object properties or adapts sensed information by adapting sensed information based on sensor performance and/or sensor adaptation preferences.

There are two possible implementations for adapting information from the real world to the virtual world. In the first system implementation, R => V adaptation takes the sensor performance as input, the sensed information from the sensor, the sensor adaptation preference from the user, and adapts the sensed information based on the sensor performance and/or sensor adaptation preference. dapt

In the second system implementation, R => V adaptation takes the sensor performance from the sensor, the sensor adaptation preference from the user, and/or the sensed information from the virtual world with or without virtual world object characteristics, and the sensor performance and/or sensor adaptation preference Controls the virtual world object properties that adapt the sensed information based on

ii) information exchange between virtual worlds (adaptation)

6 illustrates information exchange between virtual worlds according to an embodiment.

Represents an information exchange that includes VW object properties that create exchangeable information in the virtual world.

V => V adaptation adapts proprietary virtual world object properties to VW object properties in the virtual world and transfers VW object properties from one virtual world to another virtual world to support interoperability. Based on the data provided in the virtual world object properties, the virtual world internally adapts its own representation to the virtual object/avatar.

7 illustrates a sensory effect authoring apparatus for authoring sensory effects, an adaptation engine for converting authored sensory effects to be reproduced in the sensory apparatus, and a sensory apparatus for reproducing sensory effects, according to an embodiment.

According to an embodiment, the sensory effect authoring apparatus 710 may acquire ( 711 ) a 360-degree image using a VR camera or a 360 camera. The sensory effect authoring apparatus 710 may generate ( 712 ) sensory effect metadata based on the object included in the 360-degree image. The sensory effect authoring apparatus 710 may generate ( 713 ) sensory effect metadata based on an ID corresponding to a scene viewed by the user from among the 360-degree images.

The generated sensory effect metadata 712 and 713 and the obtained 360-degree image 711 may be downloaded and transmitted in the form of files that can be transmitted and received through wired/wireless Internet, Wifi, and Zigbee.

The adaptation engine 730 may receive the 360-degree image 711 and sensory effect metadata 712 and 713 . The adaptation engine 730 may convert the received sensory effect metadata 712 and 713 into a form suitable for reproduction in a user environment.

The adaptation engine 730 may obtain ( 731 ) the user's environment information using a sensor. Here, the user's acquired environment information 731 may be used to effectively reproduce the authored sensory effect metadata 712 and 713 . For example, when the data measured by the illuminance meter among the environmental information acquired by the sensor is bright, the sensory effect corresponding to “brightness” included in the sensory effect metadata may not need to be expressed.

The adaptation engine 730 may acquire ( 732 ) the user's preference. Here, the sensory effect disliked by the user may be removed or transformed according to the user's preference. For example, if pregnant women do not like strong stimuli (eg, sound, lighting, fragrance, etc.), if user preference for strong stimuli is set to "weak" or "ignore" can hardly be reproduced.

The adaptation engine 730 may acquire 733 sensory device capabilities. The authored sensory effect metadata may be reproduced to suit the performance of the sensory device. For example, the wind may be expressed using a fan among sensory devices. When the wind is expressed as 100% using a fan, the control command for the fan may be 3 stages or 5 stages depending on the performance of the fan.

The adaptation engine 730 may obtain ( 734 ) gaze information of the user. Here, the user's gaze information may be used to determine a specific scene viewed by the user from among the 360-degree images. This is because the user cannot see the entire 360-degree video, but only a part of it.

The adaptation engine 730 may determine ( 735 ) a specific scene by using the obtained user's gaze information ( 734 ). The adaptation engine 730 may extract objects included in the determined scene and obtain trajectory information on the extracted objects ( 736 ). In this case, the acquired trajectory information 736 may be used to determine a scene according to the movement of the user's gaze.

The adaptation engine 730 may convert the sensory effect metadata 712 and 713 generated by the sensory effect authoring apparatus 710 into a form suitable for reproduction in a user environment through the sensory effect adaptation 737 .

After the sensory effect adaptation 737 is performed, the adaptation engine 730 may generate 738 a control command XML for controlling the sensory device 750 . The adaptation engine 730 may generate ( 739 ) a SyncTable for synchronizing the video time and the control command XML time to reproduce the sensory effect according to the time the video is played. Accordingly, when the control command XML time in the SyncTable corresponding to the video time is retrieved while the video is being played, the control command XML corresponding to the retrieved control command XML time may be transmitted ( 740 ) to the sensory device.

The sensory device 750 may receive the control command XML, parse the received control command XML (751), and generate (752) a control command in a binary form other than XML, and may be configured in a wired/wireless network/serial/Bluetooth or the like. The sensory device may be controlled 753 using the interface.

According to an embodiment, the sensory device may provide all the senses that the user feels in the real world to the user in the virtual world. Such sensory effects may be provided differently according to a scene according to the user's view and an object included in the scene. To this end, the adaptation engine may determine a scene according to the user's moving distance and direction and the user's field of view based on 6DoF (yaw/pitch/roll 3DoF + user space location change) data.

8 illustrates that an adaptation engine considers a user's preference according to an embodiment.

According to an embodiment, when an author or editor intends to author a lighting effect, all environments in which the authored lighting effect is reproduced cannot be considered. Accordingly, an author or editor may determine a lighting effect based on a scene of a 360-degree image at the time of authoring or an object included in the scene.

For example, when the author tries to express the red color included in the fireworks scene, the author sets the red color to 300 on the premise that the performance of the sensory device can be expressed from 0 to 500, and the Sensory effect metadata may be transmitted.

At this time, sensory device performance can express 'red' 'blue' 'green', and environmental information is measured as 'white' from a sensor capable of measuring 'white''red''blue''green', Users may not prefer 'red'. Then, the adaptation engine may transmit the control command XML indicating 'No operation' close to 'ignore'.

However, in the same situation, if user preference does not favor 'green', 'red' may be expressed as the author intended at the time of writing.

<virtual world object metadata>

The specificity of Virtual Environments (VEs) with respect to other multimedia applications lies in the representation of virtual world objects within the environment. "Virtual world objects" can be classified into two types: avatars and virtual objects. The avatar can be used as a (visual) representation of the user in the environment. These virtual world objects serve different purposes:

i) characterize the various types of objects within the VE; ii) provide interaction with the VE;

Even if some components of an object are related to the virtual environment (e.g. an avatar wearing a medieval suit in a modern style VE might be inappropriate), create and import the object once in other VEs. /should be available For the latter purpose, it should be possible to control the object from an external application (e.g., emotions that one avatar exposed in VE can obtain by processing the relevant user's physiological sensors). Current virtual world object properties XSDs describe objects by considering three main requirements:

It should be possible to i) easily create importers/exporters from various VEs implementations, ii) it should be easy to control objects within a VE, iii) data contained in virtual world object properties files. should be able to modify an object's proprietary template (virtual world only) using

Once an object is created with an authoring tool native to VW, it can be used in other VWs as well. In the case of an avatar, the user can have a unique presentation inside every VW, just like in real life. He can change and upgrade his avatar (i.e. "virtual self") in one VW, and all updated attributes can be reflected in all other VWs. The avatar itself contains expression and animation functions, but also contains higher-level semantic information. However, each VW may have an internal structure for handling avatars. All relevant characteristics of the avatar (including associated actions) can be exported from VW and then imported into another VW. Similarly, any virtual object you create can be exchanged between VW by exporting and importing the relevant properties of the object. In the case of an interface between the virtual world and the real world, the sensed real-world information can be processed to obtain meaningful data that can be used as control parameters for the relevant properties of the object in VW. With respect to the avatar, the captured gestures of the user can be used to control the avatar's gestures in VW by updating the avatar's relevant characteristics. Similarly, avatar motions created in the virtual world can be mapped onto real robots for use in hazardous areas, maintenance work, support for people with disabilities and/or the elderly, and the like.

There are basic types of properties and characteristics of virtual world objects that are shared by both avatars and virtual objects.

The basic types of virtual world object properties consist of the following types of data:

i) Identity: Contains an identification descriptor. ii) Sound: Includes sound resources and related properties. iii) Scent: Includes attributes related to fragrance resources. iv) Control: contains a set of descriptors for controlling the operating characteristics of an object, such as translation, orientation, and scaling. ⅴ) Event: Contains a set of descriptors that provide input events from mouse, keyboard, etc. ⅵ) Behavior Model: Includes a set of descriptors that define behavior information of an object according to an input event. ⅶ) id: includes a unique identifier for identifying individual virtual world object information.

The virtual world object basic type may be inherited from both avatar metadata and virtual object metadata to extend specific aspects of each metadata.

<Virtual world object base type>

Here, a complex type of VWOBaseType that avatar characteristic information and virtual object characteristic information should inherit is defined. 9 is a diagram illustrating an XML representation syntax according to an embodiment.

In this case, the source is shown in Table 8 below, and the binary representation syntax is shown in Table 9 below. And the semantics at this time are shown in Table 10 below.

VWOBaseType{ Number of bits Mnemonic IdentificationFlag One bslbf DescriptionFlag One bslbf VWOCFlag One bslbf BehaviorModelListFlag One bslbf IdFlags One bslbf if(IdentificationFlag) { identification IdentificationType } if(DescriptionFlag) { Description UTF-8 } if(VWOCFlag) { SoundListFlag One bslbf ScentListFlag One bslbf ControlListFlag One bslbf EventListFlag One bslbf SensoryEffectListFlag One bslbf if(SoundListFlag) { SoundList VWOSoundListType } if(ScentListFlag) { ScentList VWOScentListType } if(ControlListFlag) { ControlList VWOControlListType } if(EventListFlag) { EventList VWOEventListType } If(SensoryEffectListFlag) { SensoryEffectList VWOSensoryEffectListType } } if(BehaviorModelListFlag) { BehaviorModelList VWOBehaviorModelListType } if(IdFlag) { id UTF-8 } } AvatarBaseType { VWOBase VWOBaseType } VirtualObjectBaseType { VWOBase VWOBaseType }

Name Description VWOBaseType A basic type that describes common attributes and elements of avatars and virtual objects. DescriptionFlag Represented by a binary, this field signals the presence of a description element. "1" means the element is used, "0" means the element is not used. VWOCFlag Represented by a binary, this field signals the presence of a VWOC element containing a list of sounds, fragrances, controls, and events. "1" means the element is used, "0" means the element is not used. IdFlags Represented by the binary, this field signals the presence of the id attribute. "1" means the element is used, "0" means the element is not used. IdentificationFlag Represented by binary, this field signals the presence of an identifying element. "1" means the element is used, "0" means the element is not used. SoundListFlag Represented by binary, this field signals the presence of a list of sound elements. "1" means the element is used, "0" means the element is not used. ScentListFlag Represented by binary, this field signals the presence of a list of Scent elements. "1" means the element is used, "0" means the element is not used. ControlListFlag Represented by a binary, this field signals the presence of a Control element list. "1" means the element is used, "0" means the element is not used. EventListFlag Represented by binary, this field signals the presence of an Event element list. "1" means the element is used, "0" means the element is not used. SensoryEffectListFlag Represented by a binary, this field signals the presence of a SensoryEffect element list. "1" means the element is used, "0" means the element is not used. BehaviorModelListFlag Represented by the binary, this field signals the existence of a list of BehaviorModel elements. "1" means the element is used, "0" means the element is not used. identification Describes the identification of virtual world objects. Description Contains a description of the virtual world object. VWOC Describes a set of properties of a virtual world object. SoundList Describes a list of sound effects associated with virtual world objects. ScentList Describes a list of fragrance effects associated with virtual world objects. ControlList Describes the list of controls associated with the virtual world object. EventList Describes a list of input events associated with virtual world objects. SensoryEffectList Describes a list of sensory effects associated with virtual world objects. BehaviorModelList Describes the list of motion models associated with virtual world objects. id A unique identifier for identifying individual virtual world object information. AvatarBaseType A type that provides a characteristic description of the personal avatar. VirtualObjectBaseType A type that provides a characteristic description of an individual virtual object.

<identification type>

10 is a diagram illustrating an XML representation syntax for an identification type according to an embodiment.

In this case, the source is shown in Table 11 below, and the binary representation syntax is shown in Table 12 below. And the semantics at this time are shown in Table 13 below.

IdentificationType { Number of bits Mnemonic UserIDFlag One bslbf OwnershipFlag One bslbf RightsFlag One bslbf CreditsFlag One bslbf nameFlag One bslbf familyFlag One bslbf if(UserIDFlag) { UserID UTF-8 } if(OwnershipFlag) { Ownership AgentType } if(RightsFlag) { NumRights vluimsbf5 for(k=0; k< NumRights; k++){ Rights[k] See ISO/IEC 21000-16:2005 LicenseType } } if(CreditsFlag) { NumCredits vluimsbf5 for(k=0; k< NumCredits; k++){ Credits[k] AgentType } } if(nameFlag) { name See ISO/IEC 10646 UTF-8 } if(familyFlag) { family See ISO/IEC 10646 UTF-8 } } AgentType{ Number of bits Mnemonic mpeg7:AgentType UTF-8 } LicenseType{ Number of bits Mnemonic r:LicenseType UTF-8 }

Name definition IdentificationType Describes the identification of virtual world objects. UserIDFlag Represented by binary, this field signals the presence of a UserID element. "1" means the element is used, "0" means the element is not used. OwnershipFlag Represented by the binary, this field signals the existence of an ownership element. "1" means the element is used, "0" means the element is not used. RightsFlag Represented by binary, this field signals the existence of the rights element. "1" means the element is used, "0" means the element is not used. CreditsFlag Represented by binary, this field signals the presence of a credit element. "1" means the element is used, "0" means the element is not used. nameFlag Represented by the binary, this field signals the presence of the name attribute. "1" means the element is used, "0" means the element is not used. familyFlag Represented by the binary, this field signals the presence of a family attribute. "1" means the element is used, "0" means the element is not used. UserID Contains the user ID associated with the virtual world object. Ownership In binary representation, "AgentType" is UTF-8 encoded. NumRights This field is represented by binary and specifies the number of rights information. Rights In binary representation, "LicenseType" is encoded as UTF-8. NumCredits This field is represented by binary and specifies the number of credit information. Credits In binary representation, "AgentType" is UTF-8 encoded. name Describes the name of the virtual world object. family Describes relationships with other virtual world objects.

<VWO Sound List Type (VWOSountListType)>

11 is a diagram illustrating an XML representation syntax for a VWO sound list type according to an embodiment.

In this case, the source is shown in Table 14 below, and the binary representation syntax is shown in Table 15 below. And the semantics at this time are shown in Table 16 below.

VWOSoundListType { Number of bits Mnemonic NumVWOSoundType vluimsbf5 for(k=0; k< NumVWOSoundType; k++){ Sound[k] VWOSoundType } }

Name definition VWOSoundListType A wrapper element type that allows the generation of multiple sound effects associated with virtual world objects. NumVWOSoundType Represented by binary, this field specifies the number of sound information contained in the sound list type. Sound Describes sound effects associated with virtual world objects.

<VWO fragrance list type (VWOScentListType)>

12 is a diagram illustrating an XML representation syntax for a VWO fragrance list type according to an embodiment.

In this case, the source is shown in Table 17 below, and the binary representation syntax is shown in Table 18 below. And the semantics at this time are shown in Table 19 below.

VWOScentListType { Number of bits Mnemonic NumVWOScentType vluimsbf5 for(k=0; k< NumVWOScentType; k++){ Scent[k] VWOScentType } }

Name definition VWOScentListType A wrapper element type that can allow the generation of multiple fragrance effects associated with a virtual world object. NumVWOScentType This field is expressed in binary expression format and specifies the number of fragrance information included in the fragrance list type. Scent Describe the fragrance effect associated with virtual world objects.

<VWO ControlListType>

13 is a diagram illustrating an XML representation syntax for a VWO control list type according to an embodiment.

In this case, the source is shown in Table 20 below, and the binary representation syntax is shown in Table 21 below. And the semantics at this time are shown in Table 22 below.

VWOControlListType { Number of bits Mnemonic NumVWOControlType vluimsbf5 for(k=0; k< NumVWOControlType; k++){ Control[k] VWOControlType } }

Name definition VWOControlListType A wrapper element type that allows multiple occurrences of controls related to virtual world objects. NumVWOControlType This field is expressed in binary representation format and specifies the number of control information included in the control list type. Control Describes controls associated with virtual world objects.

<VWO Event List Type (VWOEventListType)>

14 is a diagram illustrating an XML representation syntax for a VWO event list type according to an embodiment.

In this case, the source is shown in Table 23 below, and the binary representation syntax is shown in Table 24 below. And the semantics at this time are shown in Table 25 below.

VWOEventListType { Number of bits Mnemonic NumVWOEventType vluimsbf5 for(k=0; k< NumVWOEventType; k++){ Event[k] VWOEventType } }

Name definition VWOEventListType A wrapper element type that allows multiple occurrences of input events associated with virtual world objects. NumVWOEventType This field is expressed in binary representation format and specifies the number of event information included in the event list type. Event Describes input events related to virtual world objects.

<VWOBehaviorModelListType>

15 is a diagram illustrating an XML representation syntax for a VWO behavior model list type according to an embodiment.

In this case, the source is shown in Table 26 below, and the binary representation syntax is shown in Table 27 below. And the semantics at this time are shown in Table 28 below.

VWOBehaviorModelListType { Number of bits Mnemonic NumVWOBehaviorModelType vluimsbf5 for(k=0;k<NumVWOBehaviorModelType;k++){ BehaviorModel[k] VWOBehaviorModelType } }

Name definition VWOBehaviorModelListType A wrapper element type that allows multiple occurrences of a behavioral model associated with a virtual world object. NumVWOBehaviorModelType This field is expressed in binary representation format and specifies the number of behavior model information included in the behavior model list type. BehaviorModel Describes the behavioral model associated with virtual world objects.

<VWO Sound Type (VWOSoundType)>

16 is a diagram illustrating an XML representation syntax for a VWO sound type according to an embodiment.

In this case, the source is shown in Table 29 below, and the binary representation syntax is shown in Table 30 below. And the semantics at this time are shown in Table 31 below.

VWOSoundType{ Number of bits Mnemonic SoundIDFlag One bslbf IntensityFlag One bslbf DurationFlag One bslbf LoopFlag One bslbf NameFlag One bslbf ResourcesURL See ISO/IEC 10646 UTF-8 if(SoundIDFlag) { soundID See ISO/IEC 10646 UTF-8 } if(IntensityFlag) { intensity 32 fsbf } if(DurationFlag) { duration 32 uimsbf } if(LoopFlag) { loop 8 uimsbf } if(NameFlag) { name See ISO/IEC 10646 . UTF-8 } }

Name definition VWOSoundType A descriptive type of sound effect associated with a virtual world object. SoundIDFlag Represented by binary, this field signals the presence of the ID attribute of the sound. "1" means that the attribute is used, and "0" means that the attribute is not used. IntensityFlag Represented by binary, this field signals the presence of an intensity attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. DurationFlag Represented by binary, this field signals the existence of a duration attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. LoopFlag Represented by binary, this field signals the existence of the loop attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. NameFlag Represented by the binary, this field signals the existence of the name attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. SoundResources URL An element containing a link to a sound file, usually an MP4 file. soundID A unique identifier for an object sound. intensity Sound intensity (volume) duration How long the sound lasts. The default unit is ms. loop Playback option indicating the number of repeats (default: 1, 0: repeat indefinitely, 1:once, 2:twice, ..., n: n times) name The name of the sound.

According to an embodiment, in the case of Table 32 below, a description of sound information related to an object having the following semantics is shown. A sound resource named "BigAlarm" is stored in "http://sounddb.com/alarmsound_0001.wav", and the identifier of the value of soundID is "SoundID3". The sound is 30 seconds long. The sound can be played repeatedly at a volume of intensity = "50%".

<VWO fragrance type (VWOScentType)>

17 is a diagram illustrating an XML representation syntax for a VWO fragrance type according to an embodiment.

In this case, the source is shown in Table 33 below, and the binary representation syntax is shown in Table 34 below. And the semantics at this time are shown in Table 35 below.

VWOScentType{ Number of bits Mnemonic ScentIDFlag One bslbf IntensityFlag One bslbf DurationFlag One bslbf LoopFlag One bslbf NameFlag One bslbf ResourcesURL See ISO/IEC 10646 UTF-8 if(ScentIDFlag) { scentID See ISO/IEC 10646 UTF-8 } if(IntensityFlag) { intensity 32 fsbf } if(DurationFlag) { duration 32 uimsbf } if(LoopFlag) { loop 8 uimsbf } if(NameFlag) { name See ISO/IEC 10646 UTF-8 } }

Name definition VWOScentType A type containing a description of the fragrance effect associated with the virtual world object. ScentIDFlag This field, represented by the binary, signals the presence of the ID attribute of the fragrance. "1" means that the attribute is used, and "0" means that the attribute is not used. IntensityFlag Represented by binary, this field signals the presence of an intensity attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. DurationFlag Represented by binary, this field signals the existence of a duration attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. LoopFlag Represented by binary, this field signals the existence of the loop attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. NameFlag Represented by the binary, this field signals the existence of the name attribute. "1" means that the attribute is used, and "0" means that the attribute is not used. ScentResources URL An element containing a link to a fragrance file. scentID A unique identifier for an object's fragrance. intensity the intensity of the fragrance. duration How long the scent lasts. The default unit is ms. loop Playback option indicating the number of repeats (default: 1, 0: repeat indefinitely, 1:once, 2:twice, ..., n: n times) name The name of the fragrance.

According to an embodiment, as shown in Table 36 below, a description of fragrance information related to an object is displayed. A fragrance resource named "rose" is stored at "http://scentdb.com/flower_0001.sct", and the identifier for the value of scentID is "ScentID5". The intensity is 20% and the duration is 20 seconds.

<VWO ControlType>

18 is a diagram illustrating an XML representation syntax for a VWO control type according to an embodiment.

In this case, the source is shown in Table 37 below, and the binary representation syntax is shown in Table 38 below. And the semantics at this time are shown in Table 39 below.

VWOControlType { Number of bits Mnemonic ControlIDFlag One bslbf MotionFeatureControl MotionFeatureControlType if(ControlIDFlag) { controlID See ISO/IEC 10646 UTF-8 } } MotionFeaturesControlType{ PositionFlag One bslbf OrientationFlag One bslbf ScaleFactorFlag One bslbf if(PositionFlag) { Position Float3DVectorType } if(OrientationFlag) { Orientation Float3DVectorType } if(ScaleFactorFlag) { ScaleFactor Float3DVectorType } }

Name definition VWOControlType A descriptive type of control associated with the virtual world object. ControlIDFlag Represented by binary, this field signals the presence of a ControlID element. "1" means that the attribute is used and "0" means that the attribute is not used. MotionFeatureControl A set of elements that control the position, orientation, and scale of a virtual object.
MotionFeatureControlType - A type that provides three types of control: Position Control, Orientation Control, and Scaling Control.
PositionFlag - Appearing only in binary representation, this field signals the presence of a position element. "1" means the property should be used, "0" means the property should not be used.
OrientationFlag - This field, present only in binary representation, signals the presence of an orientation element. "1" means the property should be used, "0" means the property should not be used.
ScaleFactorFlag - Appears only in binary representation, this field signals the presence of a scalefactor factor. "1" means the property should be used, "0" means the property should not be used.
Position - The position of the object in the scene using a 3D floating point vector (x, y, z).
Orientation - Orientation of objects in the scene using 3D floating point vectors with Euler angles (yaw, pitch, roll).
ScaleFactor - The scale of an object in the scene is represented as a 3D floating point vector (Sx, Sy, Sz). controlID A unique identifier for the control.

When two controllers relate to the same object but to different parts of the object, and these parts are hierarchical (parent and child relationships), the controllers perform relative actions in child relationships. If the controllers are related to the same part, the controller performs scaling adjustments or similar effects on the entire object.

According to an embodiment, in the case of Table 40 below, descriptions of object control information having the following meanings are shown. A motion feature control that changes position is given, and the identifier of controlID is "CtrlID7". The object is located at X = "122.0", Y = "150.0" and Z = "40.0".

<VWO EventType (VWOEventType)>

19 is a diagram illustrating an XML representation syntax for a VWO event type according to an embodiment.

In this case, the source is shown in Table 41 below, and the binary representation syntax is shown in Table 42 below. And the semantics at this time are shown in Table 43 below.

VWOEventType { Number of bits Mnemonic MouseFlag One bslbf KeyboardFlag One bslbf UserDefinedInputFlag One bslbf if(MouseFlag) { NumOfMouse vluimsbf5 for (k=0; k<NumOfMouse; k++) { Mouse[k] MouseEventCS } } if(KeyboardFlag) { NumOfKeyboard vluimsbf5 for(k=0; k< NumOfKeyboard; k++ ) { keyCodeFlag[k] One bslbf if(keyCodeFlag[k]) { keyCode 8 uimsbf } event[k] 2 bslbf } } if(UserDefinedInputFlag) { NumOfUserDefinedInput vluimsbf5 for(k=0; k<NumOfUserDefinedInput; k++ ) { UserDefinedInput[k] See ISO/IEC 10646 UTF-8 } } eventID See ISO/IEC 10646 UTF-8 }

Name definition VWOEventType A descriptive type of input events associated with the virtual world object. MouseFlag Represented by binary, this field signals the presence of a mouse element. "1" means the element is used and "0" means the element is not used. KeyboardFlag Represented by binary, this field signals the presence of a keyboard element. "1" means the element is used and "0" means the element is not used. UserDefinedInputFlag Represented by binary, this field signals the existence of a UserDefinedInput element. "1" means the element is used and "0" means the element is not used. NumOfMouse Represented by binary, this field specifies the number of mouse events contained in the VWOEventType. Mouse Name - Element (4 bits) - Descriptio
click - 0000 - Describes the event of clicking the left button of the mouse (click)
doubleclick - 0001 - Describes the event of double clicking the left button of the mouse (doubleclick).
leftBtnDown - 0010 - Describes an event that occurs while the left button of the mouse is pressed. (LeftButtonDown)
leftBtnUp - 0011 - Describes an event that occurs when the left mouse button is released. (LeftButtonUP)
rightBtnDown - 0100 - Describes an event that occurs while the right button of the mouse is pressed. (RightButtonDown)
rightBtnUp - 0101 - Describes an event that occurs when the right mouse button is released. (RightButtonUP)
move - 0110 - Describes an event that occurs while changing the mouse position. (move)
() - 0111 ~ 1111 - Reserved NumOfKeyboard Represented by binary, this field specifies the number of keyboard events contained in the VWOEventType. keyCodeFlag Represented by a binary, this field signals the presence of a keycode input element. "1" means the element is used and "0" means the element is not used. keyCode The corresponding key code (0-255) of each key is described. event Describes a keyboard event (pressed, clicked, or released). In a binary representation, a keyboard event is represented as follows. (pressed: 00, clicked: 01, released: 10, and reserved: 11) NumOfUserDefinedInput Represented by binary, this field specifies the number of user-defined input events contained in the VWOEventType. UserDefinedInput Describes a user-defined input event. eventID A unique identifier for the event.

According to an embodiment, in the case of Table 44 below, the description of the input event is shown in the following meaning. The mouse, which is the input device, creates a new input value "click". To identify this input, the value of eventID is "EventID1".

According to another embodiment, in the case of Table 45 below, the description of the input event is shown in the following meaning. The input device, the keyboard, creates a new input value that presses the key code of "65". To identify this input, the value of eventID is "EventID2".

According to another embodiment, in the case of Table 46 below, the description of the input event is shown in the following meaning. The keyboard creates a new input event when the two keys "shift" + "a" are pressed. One of the keyboard events is the "pressed" event of the "shift" key with code "16" and the other is the "pressed" event of the "a" key with code "65". To identify this input, the value of eventID is "EventID3".

<VWO realistic effect type (VWOSensoryEffectType)>

20 is a diagram illustrating an XML representation syntax for a VWO immersive effect type according to an embodiment.

In this case, the source is shown in Table 47 below, and the binary representation syntax is shown in Table 48 below. And the semantics at this time are shown in Table 49 below.

VWOSensoryEffectType { Number of bits Mnemonic EffectFlags One bslbf GroupOfEffectsFlag One bslbf If(EffectFlag) { EffectBaseType EffectBaseType } If(GroupOfEffectsFlag) { GroupOfEffectsType GroupOfEffectType } }

Name definition VWOSensoryEffectType A type containing a description of the sensory effect associated with the virtual world object. EffectFlags Represented by binary, this field signals the presence of an effect element. "1" means the element is used and "0" means the element is not used. GroupOfEffectsFlag Represented by binary, this field signals the presence of a GroupOfEffect element. "1" means the element is used and "0" means the element is not used. Effect Describe sensory effects. GroupOfEffects Describe groups of sensory effects. The purpose of grouping is to remove some redundancy in child elements. All properties included here are inherited as child elements.

According to an embodiment, in the case of Table 50 below, a description of the VWO sensory effect list having the following meaning is shown.

<VWO behavior model type (VWOBehaviourModelType)>

21 is a diagram illustrating an XML representation syntax for a VWO behavior model type, according to an embodiment.

In this case, the source is shown in Table 51 below, and the binary representation syntax is shown in Table 52 below. And the semantics at this time are shown in Table 53 below.

VWOBehaviorModelType{ Number of bits Mnemonic BehaviorInput BehaviorInputType BehaviorOutput BehaviorOutputType } BehaviorInputType{ EventIDRefFlag One bslbf if(EventIDRefFlag){ eventIDRef See ISO/IEC 10646 UTF-8 } } BehaviorOutputType{ SoundIDFlag One bslbf ScentIDFlag One bslbf AnimationIDFlag One bslbf ControlIDFlag One bslbf if(SoundIDFlag) { SoundIDRefs See ISO/IEC 10646 UTF-8 } if(ScentIDFlag) { ScentIDRefs See ISO/IEC 10646 UTF-8 } if(AnimationIDFlag) { AnimationIDRefs See ISO/IEC 10646 UTF-8 } if(ControlIDFlag) { ControlIDRefs See ISO/IEC 10646 UTF-8 } }

Name Description VWOBehaviorModelType A type that describes a container for input events and associated output object behavior. BehaviorInput An input event to create an object behavior. BehaviorInputType See input event ID.
Element - Information
eventIDRef - input event ID
EventIDRef Flag - This field, represented by the binary, signals the presence of an eventIDRef element. "1" means the element is used, "0" means the element is not used. BehaviorOutput Object behavior output according to input events. BehaviorOutputType See the list of object behavior outputs.
Element - Information
SoundFlag - Represented by binary, this field signals the presence of a sound element. "1" means the element is used, "0" means the element is not used.
ScentFlag - This field, represented by binary, signals the presence of a fragrance element. "1" means the element is used, "0" means the element is not used.
AnimationFlag - Represented by binary, this field signals the presence of an animation element. "1" means the element is used, "0" means the element is not used.
ControlFlag - Represented by binary, this field signals the presence of a control element. "1" means the element is used, "0" means the element is not used.
soundIDRefs - Provides the object's sound effects by referencing the soundID.
scentIDRefs - Provides the scent effect of an object by referencing the scentID.
animationIDRefs - Provides animation clips of objects by referencing animationID.
controlIDRefs - Provides control of the object by referencing the ControlID.

According to an embodiment, in the case of Table 54 below, a description of the VWO behavior model having the following meaning is shown. If eventID = "EventID1" is given as a BehaviorInput, then BehaviorOutput is executed with respect to soundID = "SoID5" and animationID = "AniID4".

<Light effect>

The syntax for the light effect is shown in Table 55 below, and the binary representation syntax is shown in Table 56 below. And the semantics at this time are shown in Table 57 below.

<complexType name="LightType">
<complexContent>
<extension base="sedl:EffectBaseType">
<attribute name="color"type="ct:colorType"use="optional"/>
<attributename="intensity-value"type="sedl:intensityValueType"use="optional"/>
<attributename="intensity-range"type="sedl:intensityRangeType"use="optional"/>
</extension>
</complexContent>
</complexType>

LightType { Number of bits Mnemonic EffectBaseType EffectBaseType colorFlag One bslbf intensityValueFlag One bslbf intensityRangeFlag One bslbf if(colorFlag) { color colorType } if(intensityValueFlag) { intensity-value 32 fsfb } if(intensityRangeFlag) { intensity-range[0] 32 fsfb intensity-range[1] 32 fsfb } }

Name definition LightType A tool to describe the Light effect colorFlag Represented by binary, this field indicates the presence of the color attribute. If 1, the color property exists; otherwise, the color property does not exist. intensityValueFlag Represented by binary, this field indicates the presence of an intensity-value attribute. If 1, the intensity-value property is present; otherwise, the intensity-value property does not exist. intensityRangeFlag Represented by binary, this field indicates the presence of an intensity-range attribute. If 1, the intensity-range property is present; otherwise, the intensity-range property is not present. color Example: urn:mpeg:mpeg-v:01-SI-ColorCS-NS:alice_blue represents the Alice Blue color. intensity-value Describe the intensity of the light effect in terms of illumination in lux. intensity-range Describe the range of intensity values. Example: [10.0-5 lux, 130.0 klx].

According to an embodiment, in the case of Table 58 below, a description of the light effect having the following meaning is shown. Intensity represents 50,0 lux within the range of (10-5,32000) lux, i.e. approximately color #FF0000 for a family living room. The light effect is activated at si: pts = "0" and deactivated at si: pts = "28".

According to another embodiment, in the case of Table 59 below, a description of a light effect using a color scheme in which a color is defined as a reference to a color classification scheme is shown.

According to another embodiment, in the case of Table 60 below, a description of a light effect for disabling the left:middle:front light effect is shown.

<Scent effect>

The syntax for the sent effect is shown in Table 61 below, and the binary representation syntax is shown in Table 62 below. And the semantics at this time are shown in Table 63 below.

<complexType name="ScentType">
<complexContent>
<extension base="sedl:EffectBaseType">
<attribute name="scent"type="mpeg7:termReferenceType"
use="optional"/>
<attributename="intensity-value"type="sedl:intensityValueType"
use="optional"/>
<attributename="intensity-range"type="sedl:intensityRangeType"
use="optional"/>
</extension>
</complexContent>
</complexType>

ScentType { Number of bits Mnemonic EffectBaseType EffectBaseType scentFlag One bslbf intensityValueFlag One bslbf intensityRangeFlag One bslbf if(scentFlag) { scented 9 bslbf } if(intensityValueFlag) { intensity-value 32 fsfb } if(intensityRangeFlag) { intensity-range[0] 32 fsfb intensity-range[1] 32 fsfb } }

Name definition ScentType A tool for explaining the Scent effect. scentFlag Represented by binary, this field indicates that the Scent attribute is present. If 1, the Scent attribute is present; otherwise, there is no Scent attribute. intensityValueFlag Represented by binary, this field indicates the presence of an intensity-value attribute. If 1, the intensity-value property is present; otherwise, the intensity-value property does not exist. intensityRangeFlag Represented by binary, this field indicates the presence of an intensity-range attribute. If 1, the intensity-range property is present; otherwise, the intensity-range property is not present. scented Explain whether or not Scent is used. intensity-value The intensity of the Scent effect in ml/h. intensity-range Describe the range of intensity values. Example: [0.0, 10.0] ml/h.

According to an embodiment, in the case of Table 64 below, the description of the Scent effect is shown in the following meaning. Scent is lilac according to the classification system, with an intensity of 0,1 ml/h [within the range of (0.0,10.0) ml/h] for 10 seconds and the effect starts at si:pts = "0".

<Arrayed Light effect>

The syntax for the Arrayed Light effect is shown in Table 65 below, and the binary representation syntax is shown in Table 66 below. And the semantics at this time are shown in Table 67 below.

<!-- ################################################ ## -->
<!-- SEV Arrayed Light type -->
<!-- ################################################ ## -->
<complexType name="ArrayedLightType">
<complexContent>
<extension base="sedl:EffectBaseType">
<sequence>
<choice>
<element name="LightSamples"type="sev:colorMatrixType"/>
<elementname="LightResource"type="sev:LightResourceType"/>
</choice>
</sequence>
<attributename="updateRate"type="positiveInteger"use="optional"/>
</extension>
</complexContent>
</complexType>

<complexTypename="colorMatrixType">
<simpleContent>
<extensionbase="sev:colorVector">
<attributeref="mpeg7:dim"use="required"/>
</extension>
</simpleContent>
</complexType>

<simpleTypename="colorVector">
<listitemType="ct:colorType"/>
</simpleType>

<complexTypename="LightResourceType">
<attributename="type"type="string"use="required"/>
<attributename="ref"type="anyURI"use="required"/>
</complexType>

ArrayedLightType { Number of bits Mnemonic EffectBaseType EffectBaseType LightSamplesFlag One bsblf updateRateFlag One bslbf if(LightSamplesFlag) { NbrOfLightSampleMatrix 16 uimsbf SizeOfLightSamplesRow 16 uimsbf SizeOfLightSamplesColumn 16 uimsbf for(j=0;j<NbrOfLightSampleMatrix;j++) { for(k=0;k<( SizeOfLightSamplesRow * SizeOfLightSamplesColumn);k++) { LightSamples[k] ColorType } } } else { NbrOfLightResources 16 uimsbf for(k=0;k< NbrOfLightResources;k++) { LightResource LightResourceType } } If(updateRateFlag) { updateRate 16 uimsbf } } LightResourceType { Number of bits Mnemonic typeLength vluimsbf5 Type typeLength *8 bslbf Ref UTF-8 }

Name definition ArrayedLightType A tool to describe the Arrayed light effect. Arrayed light effects are effectively represented by LightSamples or LightResources. LightResources can consist of mxn color matrices that map to light display actuators, such as light LED curtain displays, flexible LED arrays. LightResources are defined as video or images of color (including shades of gray) formed by m × n pixels and matching m × n Arrayed light actuators. LightSamplesFlag This field specifies the selection of an arrayed light source, if represented by binary. If 1, there are LightSamples, otherwise there are LightResources. NbrOfLightSampleMatrix Describes the number of LightSamples matrices. SizeOfLightSamplesRow Describes the column size of LightSamples. SizeOfLightSamplesColumn Describes the row size of LightSamples. LightSamples Describe m color samples of the array lighting effect used by the Arrayed light actuator. LightSamples are specified in taxonomy terms or colorType units that refer to RGB values. The m × n color matrix of LightSamples expands linearly according to nbrOfColumns and nbrOfRows among the functions of the arranged light actuators. For example, suppose an array lighting effect is specified by 100x100 color samples and the number of rows and columns of array lighting actuators is 200 and 300, respectively. The adaptation engine then linearly scales the 100 x 100 array lighting effect to a 200 x 300 matrix of array lighting actuators. LightResource Describes the type of light resource. The m × n color pixels of a video or image (e.g. array lighting effect) scale linearly according to the nbrOfRows and nbrOfColumns of the array fluorescence actuator function. The matrix adaptation is the same as that of LightSamples. upateRateFlag Represented by the binary, this field indicates the presence of the updateRate attribute. If the value is 1, the updateRate attribute is present; otherwise, there is no updateRate attribute. updateRate Describes the number of data updates per second. An example value of 2 means that the arrayed lighting units change the value 2 times every second.

According to an embodiment, as in Table 68 below, the first example shows the description of array lighting effects using a LightSample with a 6 x 12 color palette. The arrayed lighting effect is activated at si:pts = "2" and shows two 6x6 color matrices per second due to updateRate = "2". The second example shows the description of an array of lighting effects using a LightResource with the following meaning: The light array information set is provided by the light source movie file and given 30 frames per second of arranged lighting information. Arranged lighting effects are activated at si: pts = "5".

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components or substituted or substituted by equivalents to achieve the appropriate result.

Claims (14)

A method for sensory effect adaptation performed by an adaptation engine, comprising:
identifying first metadata associated with the object for describing the object in the virtual world;
converting the first metadata related to the identified object into second metadata for application to a sensory device in the real world;
including,
The second metadata is converted based on a scene in which the first metadata is determined according to a user's gaze in the virtual world, and includes a sensory effect applied to the real world;
wherein the sensory effect includes an arrayed light effect defined based on ArrayedLightType.
According to claim 1,
The second meta data is
A sensory effect adaptation method in which conversion is performed considering not only the scene determined according to the user's gaze, but also the user's environment information, the user's preference, sensory device performance, and sensor performance.
According to claim 1,
The converting into the second metadata includes:
generating a control command XML for controlling the sensory device;
generating a SyncTable for synchronizing the time for the image and the time for the control command XML;
transmitting the control command XML to a sensory device when the control command XML corresponding to the time of the reproduced video is found in the SyncTable;
A sensory effect adaptation method further comprising a.
A sensory effect adaptation method performed by a sensory device, the method comprising:
identifying the control command XML generated based on the second meta data converted from the first meta data;
parsing the identified control command XML to generate a binary type control command, and controlling the sensory device using the generated binary type control command
including,
The second metadata is converted based on a scene in which the first metadata related to the object for describing the object in the virtual world is determined according to the gaze of the user in the virtual world, and sensory effect applied to the real world including,
wherein the sensory effect includes an arrayed light effect defined based on ArrayedLightType.
5. The method of claim 4,
The second meta data is
A sensory effect adaptation method in which conversion is performed in consideration of not only the scene determined according to the user's gaze, but also the user's environment information, the user's preference, and the performance of a sensory device.
In the adaptive engine,
wherein the adaptation engine comprises a processor;
The processor is
Identifies first metadata related to the object for describing an object in the virtual world, and as second metadata for applying the first metadata related to the identified object to a sensory device in the real world convert,
The second metadata is
The first metadata is converted based on a scene determined according to the gaze of the user in the virtual world, and includes a sensory effect applied to the real world;
wherein the sensory effect comprises an arrayed light effect defined based on an ArrayedLightType.
7. The method of claim 6,
The second meta data is
An adaptation engine that is converted in consideration of not only the scene determined according to the user's gaze, but also the surrounding environment information of the user's real world, the user's preference, sensory device performance, and sensor performance.
7. The method of claim 6,
The processor is
A control command XML for controlling the sensory device is generated, a SyncTable is generated for synchronizing a time for an image and a time for the control command XML, and the control command corresponding to a time of a reproduced video. and transmitting the control command XML to a sensory device when XML is retrieved from the SyncTable.
A sensory device comprising:
The sensory device includes a processor,
The processor is
Identifies the control command XML generated based on the second metadata converted from the first metadata, parses the identified control command XML to generate a binary type control command, and the generated binary type control command The sensory device is controlled using
The second meta data is
The first metadata related to the object for describing the object in the virtual world is converted based on a scene determined according to the gaze of the user in the virtual world, and includes a sensory effect applied to the real world,
The sensory effect includes an arrayed light effect defined based on ArrayedLightType.
10. The method of claim 9,
The second meta data is
A sensory device converted in consideration of not only the scene determined according to the user's gaze, but also the surrounding environment information of the user's real world, the user's preference, sensory device performance, and sensor performance.
In the recording medium on which data for a sensory effect readable by a computer is recorded,
The data includes information related to the object for describing the object in the virtual world,
The information related to the object includes (i) an object ID flag indicating the existence of an object ID and (ii) an object ID for identifying the object in the virtual world,
A list of sensory effects related to the object identified using the object ID is included in the sensory effect list, and whether to use the sensory effects included in the sensory effect list is determined through a sensory effect list flag.
12. The method of claim 11,
When the object ID flag is 1, it indicates that an object corresponding to the object ID exists,
A recording medium indicating that an object corresponding to the object ID does not exist when the object ID flag is 0.
In the recording medium on which data for a sensory effect readable by a computer is recorded,
The data includes a sensory effect list for describing an object in the virtual world,
The sensory effect list represents a list of sensory effects related to the object in the virtual world,
Whether to use the sensory effects included in the sensory effect list is determined through a sensory effect list flag.
14. The method of claim 13,
When the sensory effect list flag is 1, it indicates that the sensory effect included in the sensory effect list is used;
A recording medium indicating that the sensory effect included in the sensory effect list is not used when the sensory effect list flag is 0.

Images