KR20130107917A - Method and apparatus of processing media file for augmented reality services - Google Patents

Abstract

PURPOSE: A method for processing a media file and an apparatus thereof can store in advance data for an area on a video in which a virtual object is displayed, effectively providing augmented reality service. CONSTITUTION: A media file analyzer (210) analyzes a media file including a three-dimensional object track and a marker track to be used in providing augmented reality service. An image playing part (212) extracts video and/or audio data included in a media file and generates video data necessary for video playing. A virtual object playing unit (216) extracts virtual object data included in a media file and generates virtual object data for displaying the virtual object. An image synthesizing unit synthesizes video data and virtual object data and forms a final video. The image synthesizing unit plays the final video on a screen. [Reference numerals] (202) Media file; (204) User input; (210) Media file analyzer; (212) Image playing part; (214) AR processing unit; (216) Virtual object playing unit; (218) Image playing part; (220) Final image

Description

METHOD AND APPARATUS OF PROCESSING MEDIA FILE FOR AUGMENTED REALITY SERVICES}

The present invention relates to the processing of media files, and more particularly, to a method and apparatus for providing and processing media files for Augmented Reality (AR) services.

Augmented Reality (AR) refers to a technology in which a virtual object (that is, an AR object) is superimposed on a real world that a user sees. It is a concept of providing augmented information service by synthesizing a virtual object or related information with real-time video / audio information, and it is also referred to as mixed reality (MR) in terms of extending human sense and perception. In particular, the spread of mobile terminals and smartphones with various sensors such as cameras and global positioning systems (GPS) is spreading, and augmented reality services using mobile devices are rapidly spreading as various convergence services using high-speed mobile Internet are introduced. It is becoming.

The International Organization for Standardization (ISO) Media File Format (FF) defines the general structure for time-based multimedia files, such as video and audio, and includes MP4 (Moving Picture Experts Group) -4 (MEPG) -4 or 3GP ( It is used as the basis for other file formats, such as the 3rd Generation Partnership Project (3GPP) file format.

1 illustrates a logical configuration of an ISO-based media file according to the prior art. As illustrated, the media file 100 includes a file header area 102, a metadata area 104, and a media data area 106. It consists of.

The file header area 102 includes basic information of the content included in the media file. For example, information such as a content identifier, a content producer, a production time, and the like may be included in the file header. When the media file is divided into a plurality of tracks or streams, the media file may include track configuration information.

The metadata area 104 includes individual information about a plurality of media objects of the content included in the media file. Various profile information for decoding the media object and information about the location of the media object are included. In this case, the media object is the minimum unit of the content. In the case of a video, one image frame displayed on the screen may be a media object, and in the case of voice, one audio frame may be a media object. have. There may be a plurality of media objects for each track, and information necessary to play these media objects is included in the metadata area 104.

Media data area 106 is an area where media objects are actually stored.

The physical organization of an ISO based media file consists of boxes. Individual boxes exist as container boxes composed of related data and sub-boxes or composed only of sub-boxes. For example, the tracks illustrated in FIG. 1 are physically stored as track boxes, which are container boxes composed of various sub-boxes that store track header information, media information, media decoding information, and the like.

Conventional ISO-based media files do not define any meta-information needed to provide augmented reality services, nor do they provide a method for synthesizing multimedia content included in different layers, that is, an instruction method for superimposing images and virtual objects. Do not. Therefore, the conventional ISO-based media file has a lot of limitations to use for augmented reality service.

In the related art, a device for playing a media file analyzes an image currently being played in real time, extracts an area for displaying a virtual object, displays a virtual object in the corresponding area, and then synthesizes the final image with the playing image. to provide. Extracting the area in which the virtual object is to be displayed in the image is classified as a marker-based technology and a non-marker-based technology as a key element technology of augmented reality. In marker-based augmented reality, the device recognizes an image containing a specific image such as a black and white pattern, a barcode, etc., extracts the relative coordinates of the area where the virtual object is to be displayed, and displays the virtual object based on the non-marker-based augmentation. In reality, it directly identifies objects in the image and obtains relevant information.

The former has the advantage of providing a relatively accurate position where the virtual object is displayed compared to the non-marker technology, but it is impossible to provide a natural image because a marker must always be included in the image. In the latter case, since marker insertion is not required in the image, augmented reality service can be provided for media files created without considering augmented reality, as well as more natural than marker-based images. However, a technique for extracting feature points on which a virtual object is to be displayed by accurately recognizing an object in an image in real time is relatively inferior to a marker-based technique. In addition, most non-marker-based feature point extraction methods currently proposed require significant computational processing at the receiving terminal, which ultimately depends on the feature extraction algorithm and the computing power of the receiving device (or playback device). This can be influenced.

Meanwhile, in the conventional augmented reality service, the virtual object and the additional information shown to the user are determined by an application program providing the corresponding service. For example, in the case of an augmented reality service that provides a logo of a related company in the middle of an image to promote a product, the logo displayed to a user is entirely determined by an application, and thus is irrelevant to the actual content of the image being played. Since the advertisement image is displayed, the effect of the advertisement may be degraded.

The present invention provides a method and apparatus for providing an augmented reality service using a media file.

The present invention provides a method and apparatus for providing a media file for providing an augmented reality service.

The present invention provides a method and apparatus for processing a media file for providing an augmented reality service.

The present invention provides a method and apparatus for playing a media file for providing an augmented reality service.

The present invention embeds meta information necessary for providing an augmented reality service in a media file, that is, a virtual object and additional information necessary for displaying the same on a screen, and displays a correlation of a plurality of media in order to reproduce the image and the virtual object. It provides a method and apparatus.

A method according to a preferred embodiment of the present invention comprises: A method of processing a media file for an augmented reality service, the method comprising: analyzing a media file including a marker track and a three-dimensional (3D) object track for use in providing an augmented reality service, and a video included in the media file; And / or extracting audio data to generate image data necessary for reproducing an image, extracting virtual object information included in the media file to generate virtual object data for displaying the virtual object, and generating the image data. And synthesizing the virtual object data to form a final image, and reproducing the final image on a screen.

According to another embodiment of the present invention, there is provided a method comprising: A method of providing a media file for an augmented reality service, the method comprising: storing a media file including a marker track and a three-dimensional (3D) object track for use in providing an augmented reality service, and providing the media file It includes.

Apparatus according to a preferred embodiment of the present invention; An apparatus for processing a media file for an augmented reality service, the apparatus comprising: a media file analyzer configured to analyze a media file including a marker track and a 3D object track for use in providing an augmented reality service; An image reproducing unit for extracting included video and / or audio data to generate image data necessary for image reproduction, and a virtual object data for generating virtual object data for displaying virtual objects by extracting virtual object information included in the media file An object reproducing unit and an image synthesizing unit composes a final image by synthesizing the image data and the virtual object data, and reproduces the final image on a screen.

According to another embodiment of the present invention, there is provided an apparatus comprising: An apparatus for providing a media file for an augmented reality service, the apparatus comprising: a storage unit for storing a media file including a marker track and a 3D object track for use in providing an augmented reality service, and providing the media file It includes a control unit.

According to the disclosed embodiment of the present invention, various augmented reality services can be effectively provided by previously storing information about a virtual object used in augmented reality and a region on an image on which the virtual object is displayed in a media file. In addition, the receiver can overcome the physical resource constraints such as battery and computing power required to obtain the information needed to provide AR services in real time, and can also reuse various types of media files that already exist in AR services. have.

By utilizing the disclosed embodiments of the present invention, the advertisement effect may be further maximized by providing the user with a logo of a company related to the image to be played. For example, when a drama is played, a logo of a company producing accessories worn by the lead actor may be displayed, and in the case of a car racing broadcast, the logo of a finished car company may be displayed.

1 illustrates a logical configuration of an ISO based media file according to the prior art.
2 is a diagram illustrating a media file playing apparatus supporting an extended portion of an ISO media file format according to an embodiment of the present invention.
3 is a diagram illustrating an extended portion of an ISO media file format (ISO FF) according to an embodiment of the present invention.
4 is a diagram illustrating an ISO media file format (ISO FF) according to another embodiment of the present invention.
5 is a flow chart briefly illustrating a process of a conventional marker-based augmented reality.
6 is a diagram illustrating an example in which a marker section is defined based on a moving speed of a virtual object and a moving position on a straight line.
7A and 7B are diagrams illustrating the main boxes in a marker track and the correlation between these boxes in accordance with one embodiment of the present invention.
8 is a diagram illustrating another example in which a marker section is defined based on a moving speed of a virtual object and a moving position on a straight line.
9 is a diagram illustrating major boxes in a 3D object track and correlations between these boxes according to an embodiment of the present invention.
10 is a flowchart illustrating a procedure of synthesizing and playing a virtual object with an image in a marker section according to an embodiment of the present invention.
FIG. 11 is a diagram illustrating an example of applying a marker track and a 3D object track according to an embodiment of the present invention. FIG.
12 and 13 illustrate the contents of main boxes of a marker track and a 3D object track according to one embodiment of the invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions of the present invention, and these may be changed according to the intention of the user, the operator, or the like. Therefore, the definition should be based on the contents throughout this specification.

Hereinafter, an extended portion of an ISO media file format for storing information of meta information necessary for providing augmented reality service in an ISO-based media file, and a device for playing a media file created in the ISO media file format will be described. ,

2 is a diagram illustrating a media file playing apparatus supporting an extended portion of an ISO media file format according to an embodiment of the present invention. The media file reproducing apparatus 200 may be provided in a receiving device configured to process a media file and provide an augmented reality service.

Referring to FIG. 2, the media file analyzer 210 analyzes the media file 202 given as an input and extracts various information necessary for playing a media file, including meta information necessary for providing an augmented reality service. The augmented reality processor 214 activates or deactivates an augmented reality service based on the user input 204. The image player 212 extracts video / audio data included in the media file and processes the same to generate final data necessary for image reproduction. Similarly, the virtual object reproducing unit 216 extracts the virtual object information included in the media file and processes it to generate final data for displaying the virtual object (ie, AR object) on a screen (not shown). The image synthesizing unit 218 receives the image and the virtual object data to be displayed on the screen from the image reproducing unit 212 and the virtual object reproducing unit 216 to synthesize the final image 220, and reproduces the image on the screen. do. In particular, the image synthesizing unit 218 interoperates with the media file analysis unit 210 to correlate the image and the virtual object, that is, reproduce the image and the virtual object, reproduce only the image, or reproduce only the virtual object. Decide if

Although not shown, an embodiment of the present invention stores a media file of an ISO media file format according to an embodiment of the present invention, and provides the media file to a receiving device or an intermediate device capable of processing and playing the media file. A media file providing apparatus (eg, a media file server) is disclosed.

3 is a diagram illustrating an extended portion of an ISO media file format (ISO FF) according to an embodiment of the present invention. As shown in FIG. 3, the ISO media file format 302 represents multimedia content existing in an existing ISO media file. In addition to the video track 306 and the audio track 304, a three-dimensional (3D) object track 310 and a marker track are newly defined to provide an augmented reality service.

The marker track 308 includes the relative coordinates of the area on the screen where the virtual object will be displayed and additional information needed to represent the virtual object. Therefore, in the media file playback apparatus, information included in a marker track without going through a process of acquiring marker coordinates in a conventional marker-based augmented reality service, that is, a marker image recognition and a relative marker coordinate calculation based on a camera coordinate system. Use only to know the area where the virtual object will be displayed.

The 3D object track 310 is a track for storing virtual objects to be displayed on the screen, and serves as a container for storing expressions of virtual objects to accommodate various augmented reality technologies. In particular, a plurality of 3D object tracks may exist in the media file. Individual 3D object tracks can provide augmented reality service independent of an application by storing virtual objects for providing different types of additional information or by providing separate 3D object tracks for each augmented reality service provider.

In addition to defining meta information for augmented reality, a method of representing correlations between media for indicating screen composition for augmented reality will be described. For example, referring to FIG. 3, a screen synthesis instruction such as playing a superimposed virtual object used in augmented reality on top of an image provided by a video track, or playing only an image or a virtual object alone is indicated.

4 is a diagram illustrating an ISO media file format (ISO FF) according to another embodiment of the present invention. As shown, the ISO media file format 402 may direct the synthesis of the image with the audio track 404 and the video track 406, along with other media tracks 408, including additional video tracks.

1) marker track

The main role of the marker track is to provide the relative coordinates of the area where the virtual object is to be displayed on the screen and additional information required for representing the virtual object, for example, light source information. That is, assuming that there is a virtual marker in the image, all information provided by the marker is stored in the media file, thereby providing the same effect as the marker included in the actual image.

5 is a flowchart schematically illustrating a process of a conventional marker-based augmented reality. Referring to FIG. 5, the apparatus recognizes a marker image from an input image in step 502, calculates a three-dimensional position (ie, coordinate) and orientation of the marker image using camera coordinates in step 2 504, In step 506, the marker image is compared with the template stored in the device. In step 508, the image is converted into a three-dimensional object using the calculated coordinates. In step 510, the image and the virtual object are synthesized. Display on the screen.

Coordinate and direction information, which is one of the main information stored in the marker track, corresponds to the execution result of the process of steps 3 (502, 504, 506) in step 1 of FIG. As a result, the device uses only the information contained in the marker track without performing the process of acquiring the marker coordinates in the existing marker-based augmented reality service, that is, recognizing the marker image and calculating the relative marker coordinates based on the camera coordinate system. The area in which the virtual object is to be displayed is known.

When an image is played back, the virtual object needs to move continuously along the time axis along with the image. To this end, the device refers to the marker sections on the time axis defined based on characteristics (eg, moving speed, light source, etc.) generated when the virtual object moves, and places the virtual markers at the start and end points of each section. Store the virtual marker information in a media file.

The representation of the virtual object at the midpoint of the marker interval is calculated using the marker information of the start and end points. For example, if the brightness of the light shining on the virtual object at the start of the interval is 1,000 lux and the illuminance at the end point is 2,000 lux, then when the virtual object is moved continuously from the start point to the end point in the time axis, the light brightness is The virtual object is displayed on the screen gradually increasing from 1,000 lux to 2,000 lux. Therefore, the marker section may be described as a section in which each characteristic of the virtual object gradually increases or decreases.

6 is a diagram illustrating an example of defining a marker section based on a moving speed of a virtual object and a moving position on a straight line.

Referring to FIG. 6, the virtual object 600 is displayed on the screen for 15 minutes 10 seconds to 15 minutes 22 seconds after image playback, and moves from the upper left to the lower right of the screen. In detail, the virtual object 600 moves at a constant speed from the upper left to the lower right of the screen in the marker section # 1 602 (15:10:00 to 15:15:00), and the marker section # 2 (604). ) (15: 15: 00 ~ 15: 20: 00) moves from the bottom right of the screen to the bottom left. In marker section # 3 (606) (15:20:00-15:22:00), the moving line of the object is the same as the marker section # 2 (that is, the lower left to the lower left of the screen), but the moving speed is the marker section # 2 Is different from Therefore, the marker section # 3 606 is defined as a section separate from the marker section # 2 604.

The information of the virtual markers at the start point and the end point of each section distinguished as described above is stored in the media file. Therefore, the narrower the range, the more natural the synthesis of the virtual object and the image.

FIG. 7A illustrates the major boxes in a marker track and the correlation between these boxes in accordance with one embodiment of the present invention.

Referring to FIG. 7A, the marker track 700 has an extended form of a track box defined in an existing ISO media file format, and thus includes a tkhd (Track Header) box 702 and an mdia (Media) box 704. do. The mdia box 704 includes an mdhd (Media Header) box 706, a hdlr (Handler Reference) box 708, a minf (Media Information) box 710, and a dinf (Data Information) box 712.

The hdlr box 708, which stores the current track type, is expanded as shown in FIG. 7B to support marker tracks and 3D object tracks. The handler_type that stores the type of the current track in the hdlr box 708 after expansion is the video track 'vide', the sound track 'soun', the hint track 'hint', the timed metadata track 'meta', and the temporary video. It may be set to one of an auxiliary video track 'auxv', a location track 'loct', and a 3D object track 'objt'.

When handler_type of hdlr box 708 is set to 'loct' or 'objt', the Media Information Header box in minf box 710 is set to nmhd (Null Media Header) box.

The marker track 700 includes a mtbl (Marker Table) 714 to store the relative coordinates of the area where the virtual object is to be displayed on the screen and additional information necessary for representing the virtual object as described above. The mtbl box 714 is a container box and is composed of one mkhd (Marker Header) box 716 and zero or more mgrp (Marker Group) boxes 718. One of the main roles of the mkhd box 716 is to store information about the reference virtual marker to which the virtual marker used in the marker section is compared.

The marker image may exist in various forms such as a rectangle and a circle. Therefore, the virtual marker information (vmki) box 716a in the mkhd box 716 stores virtual marker information that is a reference for the virtual markers used in each marker section. do. In particular, it is possible to extract the enlargement or reduction information of the virtual marker by comparing the marker information used in individual marker sections with the reference virtual marker information included in the vmki, which ultimately allows the enlargement or reduction of the virtual object displayed on the virtual marker. Let's do it.

When a virtual object used in any two marker sections uses the same virtual object stored in the 3D object track, the two marker sections are included in the same marker group. Referring to the example of FIG. 8, the marker section # 1 802 and the marker section # 3 806 are included in the same group, and the marker section # 2 804 and the marker section # 4 808 are also the same group. Included in Since the final appearance of the virtual object may change depending on the three-dimensional movement and zooming of the virtual marker, the determination of the marker group is not determined by the shape of the virtual object displayed on the actual screen. It is based on a virtual object stored in the 3D object track. That is, when the two marker sections are displayed on the screen using the same virtual object stored in the 3D object track, the two marker sections belong to the same group. This is similar to identifying images included in a marker in a conventional marker based augmented reality to find a virtual object that should be displayed on that marker.

The marker interval is represented by a mint (Marker Interval Information) box 718a in the mgrp box 718, and all marker intervals included in the same group are included in the mgrp box 718. The related sample information (rsmp) box 718b in the mint box 718a stores sample information used in the corresponding marker section.

The following shows an example of the format of a mtbl (Marker Table Box).

aligned (8) class MarkerTable extends Box ('mtbl') {

}

The following shows an example of the format of a marker header box (mkhd).

aligned (8) class MarkerHeader extends Box ('mkhd') {

unsigned int  (32) related _ track _ ID ;

unsigned int  (32) next _ marker _ group _ ID ;

unsigned int  (32) marker _ group _ entry _ count ;

VirtualMarkerInformation ();

}

Here, related_track_ID represents the ID of a video track displayed by overlapping virtual objects, next_marker_group_ID is a marker_group_ID value to be used next, and an initial value is set to 1, and no mgrp box using the same marker_group_ID exists. The marker_group_entry_count represents the number of mgrp boxes included in the current track.

The following shows an example of the format of a virtual marker information box (vmki). The marker image may exist in various forms such as a rectangle and a circle. The vmki box described below is an example of displaying a virtual marker composed of rectangles.

aligned (8) class VirtualMarkerInformation extends Box ('vmki') {

unsigned int  (32) x1, y1 , z1 ;

unsigned int  (32) x2, y2 , z2 ;

unsigned int  (32) x3, y3 , z3

unsigned int  (32) x4, y4 , z4 ;

}

Where x1, y1, z1 represent the x, y, z coordinates of the upper left corner of the virtual marker, x2, y2, z2 represent the x, y, z coordinates of the upper right corner of the virtual marker, x3, y3, z3 Denotes the x, y, z coordinates of the lower left corner of the virtual marker, and x4, y4, z4 denotes the x, y, z coordinates of the lower right corner of the virtual marker.

The following shows an example of a mgrp (Marker Group Box).

aligned (8) class MarkerGroup extends Box ('mgrp') {

unsigned int  (32) marker _ group _ ID ;

unsigned int  (32) next _ marker _ interval _ ID ;

unsigned int  (32) marker _ interval _ entry _ count ;

for  (i = 1; i < marker _ interval _ entry _ count ; ++ i) {

MarkerIntervalInformation ();

}

}

Here, marker_group_ID represents the ID of the marker group, next_Marker_interval_ID is the value of marker_interval_ID to be used next, and the initial value is set to 1, and there is no mint box using the same marker_interval_ID in the given mgrp box. The marker_interval_entry_count represents the number of mint boxes included in the mgrp box.

The following shows an example of the format of a mint (Marker Interval Information Box).

aligned (8) class MarkerIntervalInformation extends Box ('mint') {

unsigned int  (32) marker _ interval _ ID ;

Region start _ region ;

Region end _ region ;

RelatedSampleInformation ();

}

Where marker_interval_ID indicates the ID of the marker section, start_region indicates detailed information about the start point of the marker section, and end_region ?? Detailed information about the end point of the marker section is shown.

The following shows an example of the format of a region (Region Box). The Region box stores the relative coordinates of the area on the screen where the virtual object will be displayed and additional information needed to represent the virtual object (for example, a light source). The example below includes only the position information of the virtual object.

aligned (8) class Region extends Box ('rinf') {

VirtualMarkerInformation ();

}

The following shows an example of the format of the rsmp (Related Sample Information Box).

aligned (8) class RelatedSampleInformation extends Box ('rsmp') {

unsigned int  (32) first _ sample ;

unsigned int  (32) sample _ count ;

}

Here, first_sample represents the first sample number of the marker section, and sample_count represents the total number of samples included in the marker section.

2) 3D object track

The 3D object track describes the virtual object used in the marker section. In particular, to support various augmented reality content and service scenarios, a plurality of 3D object tracks may exist in a media file, and each 3D object track may have a marker_group_ID defined in a marker track and a virtual object defined in a 3D object track. Provides mapping information with. In addition, the 3D object track stores information about the virtual marker that is a reference point and how the virtual object should be displayed on the screen at the virtual marker. That is, the movement, rotation, and zoom information of the virtual object displayed on the reference virtual marker specified in the vmki box defined in the mkhd box of the marker track is stored.

9 is a diagram of the major boxes in the 3D object track and the correlation between these boxes in accordance with one embodiment of the present invention.

Referring to FIG. 9, the 3D object track 900 has an extended form of a track box defined in an existing ISO media file format like the marker track 700, and thus, the tkhd (Track Header) box 902, mdia ( Media) box 904. The mdia box 904 includes an mdhd (Media Header) box 906, a hdlr (Handler Reference) box 908, a minf (Media Information) box 910, and a dinf (Data Information) box 912. Extension of existing boxes required to support the 3D object track is the same as the description of the marker track 700 described above.

The 3D object track 900 includes an otbl (Object Table) box 914 for representing a virtual object used in each marker section in addition to the existing box defined in the above-described ISO media file. The otbl box 914 is a container box, which is composed of one othd (Object Table Header) box 914a, one mtbl (Mapping Table) box 914b, and zero or more odes (Object Description) box 914c. . The mtbl box 914b provides mapping information for the actual virtual object used in the individual mgrp box of the Maker track, and the odes box 914c provides information representing the virtual object. odes box 914c serves as a container to support various augmentation techniques for virtual object representation.

There may be a plurality of 3D object tracks in an ISO-based media file, and individual tracks exist independently for each type of service or service provider. Therefore, when augmented reality is provided, a service is provided by connecting only one 3D object track, and virtual objects existing in multiple 3D object tracks cannot be mixed and used.

The following shows an example of the format of otbl (3D Object Table Box).

aligned (8) class 3DObjectTable extends Box ('otbl') {

}

The following shows an example of the format of an othd (Object Table Header Box).

aligned (8) class ObjectTableHeader extends Box ('othd') {

unsigned int  (32) next _ object _ ID ;

unsigned int  (32) object _ entry _ count ;

unsigned int  (32) service _ provider _ ID ;

string description ;

}

Here, next_object_ID is the Object_ID value to be used next. The initial value is set to 1, and in order to use the same Object_ID, the odes box type value must be different. object_entry_count represents the number of odes boxes included in the current track, service_provider_ID represents the ID of the service provider providing the 3D object track, and description represents an additional description of the current track.

The following shows an example of the format of mtbl (Mapping Table Box).

aligned (8) class MappingTable extends Box ('mtbl') {

unsigned int  (32) mapping _ entry _ count ;

for  (i = 1; i < mapping _ entry _ count ; ++ i) {

unsigned int  (32) marker _ group _ ID ;

unsigned int  (32) object _ ID ;

}

}

Here, object_entry_count indicates the number of mapping information, marker_group_ID indicates marker_group_ID defined in the mgrp box of the marker track, and object_ID indicates the ID of the virtual object to be used in the marker section identified by marker_group_ID. object_ID is defined in the odes box.

The following shows an example of the format of the odes (Object Description Box).

aligned (8) class ObjectDescription extends Box ('odes') {

unsigned int  (32) object _ ID ;

unsigned int  (32) type ;

string object _ desc

template  int (32) [9] transformation _ matrix ;

template  int (32) [9] rotation _ matrix ;

template  int (32) [9] scale _ matrix ;

}

Here, object_ID indicates the ID of the virtual object, and type indicates the technology (e.g., VRML (Virtual Reality Modeling Language), 3D XML (3D Extensible Markup Language), etc.) used to represent the virtual object. There may exist virtual objects described in different types using the same object_ID. In this case, the media file reproducing apparatus determines which technology the virtual objects represented by. object_desc represents a virtual object represented by a technique called type, transformation_matrix represents a matrix used to calculate the movement of the virtual object used to represent the virtual object over the reference virtual marker, and rotation_matrix represents a virtual object over the reference virtual marker. The scale_matrix represents a matrix used to calculate the rotation of the virtual object, and scale_matrix represents a matrix used to calculate the zoom of the virtual object used to represent the virtual object on the reference virtual marker.

The information represented by the odes box is as follows: When the reference virtual marker defined in the marker track is recognized, the virtual object defined in the object_desc of the odes box is virtualized using three matrix values (ie, transformation_matrix, rotation_matrix, and scale_matrix) that store three-dimensional movement and scale information. Determine how the object should be displayed relative to the reference virtual marker.

10 is a flowchart illustrating a procedure of synthesizing and playing a virtual object with an image in a marker section according to an embodiment of the present invention.

Referring to FIG. 10, in operation 1002, the apparatus extracts and arranges marker sections based on a sample. In operation 1004, a relative position of a virtual object to be displayed on the screen using a start point, an end point, and a reference virtual marker of each marker section, The virtual object to be displayed on the screen is extracted using the market_group_ID of the marker section and the mapping information in the 3D object track in step 1006, and the additional information for displaying the corresponding object is calculated in step 1008. The position, direction, and magnification information of the virtual object is calculated and the position, direction, and magnification information of the reference virtual marker defined in the odes box in which the object is defined to calculate the final position and appearance of the virtual object on the screen. In operation 1010, the device synthesizes the virtual object on the associated image and plays it on the screen.

11 is a diagram illustrating an example of applying a marker track and a 3D object track according to an embodiment of the present invention.

Referring to FIG. 11, each marker section 1102, 1104, 1106 is set based on a linear movement and a moving speed of a virtual object. In particular, the marker section # 1 1102 and the marker section # 3 1106 display the same virtual object (smile mark in the illustrated example), so that the two marker sections 1102 and 1106 belong to the same marker group. .

The contents of the main box of the marker track and the 3D object track, that is, the mtbl box 1202 and the otbl box 1302 for FIG. 11 are as shown in FIGS. 12 and 13, respectively. Referring to FIG. 13, there are two odes boxes 1304a and 1304b having an object_ID of 1, and each of the boxes 1304a and 1304b represents the same virtual object using different techniques. As an example, the determination of which technology is represented by the virtual object may be made by the media file reproducing apparatus.

The provision and processing of the media file according to embodiments of the present invention may be realized in the form of hardware, software or a combination of hardware and software. Such media files and / or the software may be, for example, volatile or nonvolatile storage devices such as Read Only Memory (ROM) or the like, whether random or rewritable, or random access memory (RAM), for example, Memory such as a memory chip, device or integrated circuit, or a machine (eg, optically or magnetically recordable) such as, for example, a compact disk (CD), a digital versatile disc (DVD), a magnetic disk or a magnetic tape Computer).

In addition, the media file providing and processing of the present invention may be implemented by a computer or a portable device including a control unit and a memory, the memory suitable for storing a program or programs containing instructions for implementing embodiments of the present invention. An example is a machine-readable storage medium.

Accordingly, the present invention includes a program comprising code for implementing the apparatus or method described in one claim of the present specification and a storage medium readable by a machine (computer, etc.) storing such a program. In addition, such a program may be electronically transported through any medium such as a communication signal transmitted via a wired or wireless connection, and the present invention appropriately includes the same.

In addition, the media file reproducing apparatus of the present invention may receive and store the media file and / or the program from a media file providing apparatus connected by wire or wirelessly. The media file providing apparatus includes a memory for storing a media file according to the present invention, a program including instructions for causing the media file reproducing apparatus to perform a predetermined media file processing, information for processing a media file, and the like. At least one of a memory, a communication unit for performing wired or wireless communication with the media file reproducing apparatus, and a control unit for automatically transmitting a request or a corresponding program to the communication unit by the media file reproducing apparatus.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Therefore, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

Claims (8)

In the method of processing a media file for augmented reality service,
Analyzing a media file including a marker track and a 3D object track for use in providing augmented reality service;
Extracting video and / or audio data included in the media file to generate image data necessary for playing the image;
Generating virtual object data for displaying a virtual object by extracting virtual object information included in the media file;
And synthesizing the image data and the virtual object data to form a final image, and reproducing the final image on a screen.
In the method of providing a media file for augmented reality service,
Storing a media file including a marker track and a three-dimensional (3D) object track for use in providing augmented reality service;
Providing the media file.
3. The marker track according to claim 1 or 2, wherein the marker track includes a marker table (mtbl) box for storing relative coordinates of an area of the screen on which the virtual object is to be displayed and additional information necessary for representing the virtual object.
The mtbl box defines a plurality of marker sections on a time axis classified based on a feature occurring when the virtual object moves.
A marker header (mkhd) box for storing information of a reference virtual marker to be compared with the virtual markers used in each marker section, and a marker group for storing information of marker sections included in the same group among the plurality of marker sections ( mgrp) box.
The 3D object track according to claim 1 or 2, wherein the 3D object track stores a virtual object to be displayed on a screen and is used in each of a plurality of marker sections on a time axis classified based on a characteristic generated when the virtual object moves. An object table (otbl) box for representing the virtual object being
The otbl box may include: an object table header (othd) box, a mapping table (mtbl) box for providing mapping information on an actual virtual object used in a marker section included in the same group among the plurality of marker sections, and And an object description (odes) box representing the type of augmented reality technology for representing the virtual object.
In the apparatus for processing a media file for augmented reality service,
A media file analyzer for analyzing a media file including a marker track and a 3D object track for use in providing augmented reality service;
An image reproducing unit for extracting video and / or audio data included in the media file to generate image data necessary for image reproduction;
A virtual object reproducing unit configured to extract virtual object information included in the media file and generate virtual object data for displaying the virtual object;
And an image synthesizer configured to synthesize the image data and the virtual object data to form a final image and to reproduce the final image on a screen.
In the apparatus for providing a media file for augmented reality service,
A storage unit for storing a media file including a marker track and a three-dimensional (3D) object track for use in providing augmented reality service;
And a control unit for providing the media file.
7. The marker track according to claim 5 or 6, wherein the marker track includes a marker table (mtbl) box for storing relative coordinates of an area of the screen on which the virtual object is to be displayed and additional information necessary for representing the virtual object.
The mtbl box defines a plurality of marker sections on a time axis classified based on a feature occurring when the virtual object moves.
A marker header (mkhd) box for storing information of a reference virtual marker to be compared with the virtual markers used in each marker section, and a marker group for storing information of marker sections included in the same group among the plurality of marker sections ( mgrp) box.
The method according to claim 5 or 6, wherein the 3D object track is used in each of a plurality of marker sections on a time axis that store virtual objects to be displayed on the screen and are classified based on characteristics generated when the virtual objects move. An object table (otbl) box for representing the virtual object being
The otbl box may include: an object table header (othd) box, a mapping table (mtbl) box for providing mapping information on an actual virtual object used in a marker section included in the same group among the plurality of marker sections, and And an object description (odes) box representing a type of augmented reality technology for representing a virtual object.

Images