KR20220020103A - Server, method and computer program for prividing real-life type media contents - Google Patents

Abstract

Provided is a realistic media content providing server, which includes: a learning data generation unit for generating learning data based on two-dimensional coordinate information for an object in an image for learning and an image for learning for the object to which a tracker is attached; a learning unit for training a tracking model based on the generated learning data; an output unit for outputting realistic media content and projecting the content onto an object through a projector which interworks with the realistic media content providing server; a tracking unit for tracking the two-dimensional coordinate information for the moving object through the learned tracking model, when the object moves; and an adjustment unit adjusting the realistic media content based on the two-dimensional coordinate information for the tracked object. The output unit may output the realistic media content adjusted through the projector and project the content onto the object.

Description

SERVER, METHOD AND COMPUTER PROGRAM FOR PRIVIDING REAL-LIFE TYPE MEDIA CONTENTS}

The present invention relates to an apparatus, method and computer program for providing immersive media content.

As 5G, the next-generation mobile communication, is used, various immersive media services such as AR/VR are attracting attention. In the case of the existing AR/VR service, it is inconvenient for the user to wear smart glasses or HMD device to view AR/VR contents.

To overcome the limitations of these AR/VR services, projector-based XR (Extended Reality)/MR (Mixed Reality) types of immersive media services have emerged.

Recently, a method for providing content that increases immersion by tracking a projected object by using a tracker, a depth camera, etc. in a projector-based XR/MR immersive media service method is being developed.

However, these content providing methods require an external device (a plurality of tracker devices) capable of precisely tracking the posture of an object, and accordingly, the purchase cost of the external device is high. In addition, in order to provide a projector-based XR/MR immersive media service, additional work such as performing calibration between sensors installed in a space is required. These restrictions not only increase the production cost of content constituting the sensory media service, but also block the spread of the projector-based XR/MR sensory media service.

On the other hand, although an object posture estimation method using a depth camera is being developed, the depth camera has a lower resolution than a general camera and contains a lot of noise (noise), which lowers the precision of object tracking and lowers the realism of the projected content. There is this.

Meanwhile, by applying the CNN network, which is widely used for image processing, to RGB cameras in recent years, it is possible to acquire edge information of objects (objects on which the content image is projected), which could only be obtained through conventional depth cameras, through RGB cameras. However, for this purpose, it is generally necessary to take an image and label the edges of the object included in the image.

In general, labeling data for the edge of an object is obtained through the manual labor of many people. Such manual work may also cause problems in the quality of training data depending on the skill level of the labeler.

Although the labeling operation is sometimes performed through an optical camera and a marker system, since the marker image is captured in the image obtained by this method, the operation of removing the marker image from the corresponding image is additionally required. In addition, when acquiring training data in a projection situation, there is a problem in that the quality of the optical tracker is deteriorated due to the projected light.

Japanese Laid-Open Patent Publication No. 2003-500766 (published on July 7, 2003)

The present invention is to solve the problems of the prior art described above, the tracking model is trained based on the training data generated based on the two-dimensional coordinate information about the object in the training image and the training image for the object to which the tracker is attached, Based on the two-dimensional coordinate information of the moving object tracked through the learned tracking model, it is intended to adjust the realistic media content projected to the object.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the immersive media content providing server according to the first aspect of the present invention provides a training image for an object to which a tracker is attached and two-dimensional coordinate information for the object in the training image. a learning data generation unit generating learning data based on the learning data; a learning unit for learning a tracking model based on the generated training data; an output unit for outputting immersive media content and projecting it onto an object through a projector that interworks with the immersive media content providing server; a tracking unit for tracking two-dimensional coordinate information on the moving object through the learned tracking model when the object moves; and an adjustment unit configured to adjust the immersive media content based on the two-dimensional coordinate information on the tracked object, wherein the output unit outputs the adjusted immersive media content through the projector and projects it onto the object. can

The method for providing immersive media content performed by the immersive media content providing server according to the second aspect of the present invention includes a training image for an object to which a tracker is attached and two-dimensional coordinate information for the object in the training image. generating learning data based on the; training a tracking model based on the generated training data; outputting the immersive media content through a projector interworking with the immersive media content providing server and projecting the immersive media content onto an object; tracking two-dimensional coordinate information on the moving object through the learned tracking model when the object moves; The method may include adjusting the immersive media content based on the two-dimensional coordinate information on the tracked object, and outputting the adjusted immersive media content through the projector and projecting it on the object.

A computer program stored in a computer-readable recording medium including a sequence of instructions for providing immersive media content, which is performed by the immersive media content providing server according to the third aspect of the present invention, when executed by a computing device, causes the tracker to Generates training data based on a training image for an attached object and 2D coordinate information on the object in the training image, trains a tracking model based on the generated learning data, and provides the immersive media content server Outputs immersive media content through a projector in conjunction with and projects it on an object, and when the object moves, 2D coordinate information on the moving object is tracked through the learned tracking model, and the and a sequence of commands for adjusting the immersive media content based on the two-dimensional coordinate information for the object, and outputting the adjusted immersive media content through the projector and projecting it on the object.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, the present invention learns a tracking model based on the training data generated based on the two-dimensional coordinate information for the object in the training image and the training image for the object to which the tracker is attached, and , it is possible to adjust the immersive media content projected to the object based on the two-dimensional coordinate information for the moving object tracked through the learned tracking model.

Through this, the present invention can configure learning data for tracking an object using an image taken by a general camera, so labeling manual or optical camera and marker system for the vertices of an existing object labeling operation production cost can be reduced.

In addition, the present invention can lower the cost of configuring an immersive media service system by using a general camera, and thus can expect an effect of expanding the market for immersive media services.

1 is a block diagram of a system for providing immersive media content according to an embodiment of the present invention.
2 is a block diagram of the immersive media content providing server shown in FIG. 1 according to an embodiment of the present invention.
3A to 3F are diagrams for explaining a method of generating learning data according to an embodiment of the present invention.
4 is a diagram for explaining a method of training a tracking model according to an embodiment of the present invention.
5 is a view for explaining a method of deriving two-dimensional coordinate information for an object according to the movement of the object through a tracking model, according to an embodiment of the present invention.
6 is a flowchart illustrating a method of providing immersive media content according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware.

Some of the operations or functions described as being performed by the terminal or device in this specification may be instead performed by a server connected to the terminal or device. Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the server.

Hereinafter, detailed contents for carrying out the present invention will be described with reference to the accompanying configuration diagram or process flow diagram.

1 is a block diagram of a system for providing immersive media content according to an embodiment of the present invention.

Referring to FIG. 1 , the immersive media content providing system may include a immersive media content providing server 100 , a projector 110 , and a camera 120 . However, since the immersive media content providing system of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1, and may be configured differently from FIG. 1 according to various embodiments of the present invention. may be

In general, each component of the immersive media content providing system of FIG. 1 is connected through a network (not shown). A network refers to a connection structure in which information can be exchanged between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW: World). Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television networks, and the like. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, and the like are included, but are not limited thereto.

The immersive media content providing server 100 is a two-dimensional image of the object 130 derived from the learning image and the learning image for the object 130 received from the camera 120 that has photographed the object 130 to which the tracker is attached. Training data may be generated based on the coordinates.

The reason for attaching the tracker to the object 130 is to derive three-dimensional coordinate information for the vertices of each corner of the object 130 using coordinate information corresponding to the position of the tracker attached to the object 130, and the derived Deriving two-dimensional coordinates for the object 130 from the learning image using the three-dimensional coordinate information for the vertices of each corner of the object 130, the camera internal parameter information for the camera 120, and the camera external parameter information it is for

After the generation of the learning data is completed, the tracker attached to the object 130 is removed.

The immersive media content providing server 100 may train a tracking model for tracking information on the vertices of the corners of the object 130 based on the generated training data. Here, the tracking model may be generated differently for each object having a different shape.

For example, when the shapes of the first object and the second object are different from each other, the immersive media content providing server 100 sets each of the first learning image included in the first learning data and the first object in the first learning image. Each of the second object in the second learning image and the second learning image included in the second learning data and can be used to track the coordinates of the first object by learning the tracking model based on the two-dimensional information about the vertices of the corners A tracking model may be trained based on the two-dimensional information about the vertices of the corners and used for coordinate tracking of the second object.

The immersive media content providing server 100 may output the immersive media content 140 through the projector 110 interworking with the immersive media content providing server 100 and project it onto the object 130 .

When the object 130 moves, the immersive media content providing server 100 may track the two-dimensional coordinate information of the moving object 130 through the learned tracking model.

The immersive media content providing server 100 adjusts the immersive media content 140 based on the two-dimensional coordinate information on the tracked object 130 , and outputs the adjusted immersive media content through the projector 110 . Thus, it can be projected on the object 130 .

Hereinafter, the operation of each component of the immersive media content providing system of FIG. 1 will be described in more detail.

2 is a block diagram of the immersive media content providing server 100 shown in FIG. 1 according to an embodiment of the present invention.

Referring to FIG. 2 , the immersive media content providing server 100 includes a derivation unit 200 , a learning data generation unit 210 , a learning unit 220 , an output unit 230 , a tracking unit 240 , and an adjustment unit ( 250) may be included. However, the immersive media content providing server 100 shown in FIG. 2 is only one implementation example of the present invention, and various modifications are possible based on the components shown in FIG. 2 .

Hereinafter, FIG. 2 will be described with reference to FIGS. 3A to 5 .

The derivation unit 200 includes two-dimensional coordinate information for a marker derived from a calibration image for a marker photographed by a camera in a space in which at least one marker is installed, three-dimensional coordinate information for a marker measured from a receiver installed in the space, and The camera external parameter information may be derived based on the camera internal parameter information. Here, the calibration image for the marker is before shooting the learning image for generating learning data related to object tracking, the marker installed in the space in order to obtain parameter information about the camera that shoots a specific space through the camera It means a two-dimensional image taken. Here, the camera internal parameter may include, for example, a focal length of the camera lens, a center point of the camera lens, an asymmetry coefficient for the image sensor, and a distortion coefficient for the camera lens. Here, the focal length (f, focal length) of the camera lens means the distance between the center of the camera lens and the image sensor installed in the camera, and the central point (c, principal point) of the camera lens is the distance from the pinhole of the camera to the image sensor. It means the feet of the water line. The skew coefficient (α) for the image sensor means the degree of inclination of the y-axis with respect to the cell arrangement of the image sensor, and the distortion coefficient (p, distortion coefficient) for the camera lens is the degree of distortion of the camera lens. is shown. Here, the camera external parameter information means information on the position and posture of the camera in the three-dimensional coordinate system.

For example, referring to FIGS. 3A and 3B together, the derivation unit 200 captures at least one marker 32 installed in a specific space with a camera 120 interlocked with the immersive media content providing server 100 . In one case, the calibration image 301 of the marker 32 photographed from the camera 120 may be received, and 2D coordinate information of the marker 32 may be derived from the calibration image 301 .

The derivation unit 200 includes transmission time information transmitted by a plurality of receivers 30 installed in a specific space to at least one marker 32 hanging in a specific space, and a reflection reflected back by the marker 32 and transmission time information. Distance information between the marker 32 and the receiver 30 may be derived based on the reception time information at which the signal is received. In addition, the derivation unit 200 may derive 3D coordinate information for the marker 32 in a specific space based on the location information of the receiver 30 and the distance information between the marker 32 and the receiver 30 . there is.

The derivation unit 200 substitutes the camera internal parameter information 303, the two-dimensional coordinate information 305 for the marker 32, and the three-dimensional coordinate information 307 for the marker 32 into the determinant of FIG. External parameter information 309 may be derived. At this time, the camera external parameter information 309 is composed of three equations with 12 unknowns, and each equation is two-dimensional coordinate information 305 for the marker 32 and three-dimensional coordinate information for the marker 32 . (307) and camera internal parameter information (303), 12 equations that can solve 12 unknowns are established when 4 or more trackers (a device attached to an object to measure the position of the object) are used. Camera external parameter information 309 for the camera 120 in space may be derived.

The derivation unit 200 applies a transformation matrix for the posture of the tracker to the coordinate information corresponding to the position of the tracker attached to the object 130, and three-dimensional coordinates for the vertices of each corner of the object 130 to which the tracker is attached information can be derived.

Referring to FIG. 3C , after measuring the shape of the object 130 , the derivation unit 200 may select points constituting vertices of an easily identifiable edge of the object 130 in the image.

The derivation unit 200 determines the attachment location information 311 for attaching the tracker 150 to the object 130, and from the attachment location information 311 of the tracker 150 attached to the object 130 to the object ( 130) can measure the distance to the vertex of the edge. Here, the attachment position information 311 of the tracker 150 may be composed of three-dimensional coordinate information for the tracker 150 .

The derivation unit 200 converts the posture of the tracker 150 with respect to the X-axis, Y-axis, and Z-axis based on the posture information (attachment location information 311) of the tracker 150 attached to the object 130. A matrix 313 may be derived. At this time, the derivation unit 200 applies a translation value and a rotation value for the tracker 150 in the process of measuring the distance to the vertex of the corner of the object 130 , and a transformation matrix for the posture of the tracker 150 . (313) can be derived.

The derivation unit 200 may derive the transformation matrix 313, [Rt] to each edge forming the vertex of the corner of the object 130 by using the posture information of the tracker 150 . For example, when the tracker 150 is attached to the rear center of the object 130, the object 130 does not rotate on a plane, so the rotation matrix is set as an identity matrix, and X from the tracker 150 The rotation matrix can be used as a t-value by measuring the amount of change in the axis, Y-axis, and Z-axis.

The derivation unit 200 applies a transformation matrix 313 for the posture of the tracker 150 to coordinate information corresponding to the position of the tracker attached to the object 130 (attachment position information 311 of the tracker 150). While doing so, the vertices 315 forming the corners of the object 130 may be selected. Also, the derivation unit 200 may derive 3D coordinate information about the vertices of each corner of the object 130 by using the points 315 of the selected object 130 . Here, the reason for attaching the tracker to the object 130 is to derive three-dimensional coordinate information for the vertices of each corner of the object 130 through coordinate information corresponding to the position of the tracker attached to the object 130. .

3C and 3D together, the derivation unit 200 receives the learning image 321 from the camera 120 that has captured the learning image 321 including the object 130 to which the tracker 150 is attached. In one case, based on the camera internal parameter information 303 for the camera 120 that has captured the learning image 321 , the camera external parameter information 309 and the three-dimensional coordinate information 317 for the object 130 for learning Two-dimensional coordinate information 319 for the object 130 in the image 321 may be derived.

The derivation unit 200 substitutes the camera internal parameter information 303 for the camera 120, the camera external parameter information 309, and the three-dimensional coordinate information 317 for the object 130 into the determinant of FIG. 3D for learning Two-dimensional coordinate information 319 for the object 130 in the image 321 may be derived.

The training data generation unit 210 generates training data based on the two-dimensional coordinate information for the object 130 in the training image 321 and the training image 321 for the object 130 to which the tracker 150 is attached. can do.

Referring to FIG. 3E , the learning data generating unit 210 projects the immersive media content 140 to the object 130 through the projector 110 interworking with the immersive media content providing server 100 , and the corresponding object. When the learning image 321 including the object 130 on which the immersive media content 140 is projected is received from the camera 120 that has captured the image 130, the object on which the immersive media content 140 is projected ( Learning data may be generated based on the training image 321 for 130 , and 2D coordinate information about the object 130 in the training image 321 .

The derivation unit 200 includes the camera internal parameter information 303 for the camera 120, the camera external parameter information 309, and the object 130 according to the position change or posture change of the object 130 in the determinant of FIG. 3D. By substituting the three-dimensional coordinate information 317, the two-dimensional coordinates for the vertices of each corner of the object 130 in the learning image 321 for the object 130 photographed according to the position change or posture change of the object 130 Information 319 may be derived.

The learning data generating unit 210 generates learning data by mapping the two-dimensional coordinate information about the vertices of the learning image 321 and each corner of the object 130 that are changed according to the position change or the posture change of the object 130 . can do.

Referring to FIG. 3F , the learning data generating unit 210 is a two-dimensional coordinate for the vertices of each corner of the object 130 in the first learning image and the first learning image captured according to the first posture of the object 130 . The first training data may be generated by mapping the information ((x1, y1), (x2, y2), (x3 y3), (x4, y4)). In addition, the learning data generating unit 210 may include two-dimensional coordinate information ((x1) for the vertices of each corner of the object 130 in the second learning image and the second learning image captured according to the second posture of the object 130 . , y1), (x2, y2), (x3 y3), (x4, y4)) may be mapped to generate second training data.

The learning unit 220 may train the tracking model based on the generated training data. Here, the tracking model is a model for acquiring two-dimensional coordinate information about the vertices of each corner of the object, and may be generated differently for each object having a different shape.

For example, if the shapes of the first object and the second object are different from each other, the learning unit 220 may set the first learning image included in the first learning data and the vertex of each corner of the first object in the first learning image. It is possible to learn a tracking model based on the two-dimensional information about the first object and use it to track the coordinates of the first object, and to the vertices of each corner of the second object in the second learning image and the second learning image included in the second learning data. A tracking model can be trained based on the two-dimensional information about the object and used for tracking the coordinates of the second object.

Referring to FIG. 4 , the learning unit 220 inputs the training image to the tracking model 40 to output the two-dimensional coordinate information for the vertices of each corner of the object mapped to the training image 40. can learn For example, when the tracking model 40 receives the first training image, the learning unit 220 outputs two-dimensional coordinate information about the vertices of each corner of the object mapped to the first training image. ) can be learned. In addition, when the tracking model 40 receives the second training image, the learning unit 220 outputs the tracking model 40 to output two-dimensional coordinate information about the vertices of each corner of the object mapped to the second training image. can learn

After the generation of training data required for training of the tracking model 40 is completed, the tracker is removed from the object 130 .

The output unit 230 may output the immersive media content 140 through the projector 110 interworking with the immersive media content providing server 100 and project it onto the object 130 . Here, the object 130 may be an object to which a tracker is not attached.

When the object 130 moves, the tracker 240 may track 2D coordinate information on the moving object 130 through the learned tracking model 40 . The tracker 240 may track the two-dimensional coordinate information of the vertices of each corner of the object 130 from the captured image of the moving object 130 through the learned tracking model 40 .

The tracker 240 may estimate the posture of the object 130 by using the two-dimensional coordinate information about the vertices of each corner of the object 130 acquired at continuous time through the learned tracking model 40 . . Also, the tracker 240 may calculate a position value of the object 130 by using a posture value corresponding to the estimated posture of the object 130 .

Referring to FIG. 5, the tracking unit 240 inputs the first captured image of the moving object 130 (eg, the moving or rotating object 130) photographed at time t to the tracking model 40, Through the tracking model 40 , 2D coordinate information about the vertices of each corner of the object 130 may be tracked from the first captured image.

Then, the tracking unit 240 inputs the second captured image of the moving object 130 photographed at time t+1 into the tracking model 40, and the object ( 130), the two-dimensional coordinate information for the vertices of each corner can be tracked.

) 및 위치 정보(

)를 계산할 수 있다. As such, when the change of the two-dimensional coordinate information for the vertices of the corners of the object 130 between successive captured images is applied to [Equation 1], 3 of the object 130 at time t+1 compared to time t Dimension posture information (

) and location information (

) can be calculated.

[Equation 1]

The adjuster 250 may adjust the immersive media content 140 based on the two-dimensional coordinate information of the tracked object 130 . For example, the adjustment unit 250 is calculated according to the two-dimensional coordinate information for the object 130 tracked at time t and the two-dimensional coordinate information for the object 130 tracked at time t+1 versus time t The immersive media content 140 may be adjusted based on the 3D posture information and location information of the object 130 changed at +1 time.

The output unit 230 outputs the immersive media content 140 adjusted by the adjustment unit 250 through the projector 110 linked with the immersive media content providing server 100 to project it onto the object 130 . can

As described above, in the present invention, the immersive media content 140 is displayed on the object 130 in response to the 3D posture information and location information of the object 130 changed at time t+1 compared to time t through the projector 110 . By projecting, it is possible to provide a projector-based immersive media service without expensive equipment such as a separate tracker system.

Meanwhile, for those skilled in the art, the derivation unit 200 , the learning data generation unit 210 , the learning unit 220 , the output unit 230 , the tracking unit 240 , and the adjustment unit 250 are implemented separately, or among them It will be fully understood that one or more may be integrated and implemented.

6 is a flowchart illustrating a method of providing immersive media content according to an embodiment of the present invention.

Referring to FIG. 6 , in step S601 , the immersive media content providing server 100 provides training data based on the training image for the object 130 to which the tracker is attached and the two-dimensional coordinate information for the object 130 in the training image. can create After the generation of the training data is completed, the tracker may be removed from the object 130 .

In step S603, the immersive media content providing server 100 may train a tracking model based on the generated training data.

In step S605 , the immersive media content providing server 100 may output the immersive media content 140 through the projector 110 interworking with the immersive media content providing server 100 and project it onto the object 130 . there is.

In step S607 , when the object 130 moves, the immersive media content providing server 100 may track the two-dimensional coordinate information of the moving object 130 through the learned tracking model.

In step S609 , the immersive media content providing server 100 may adjust immersive media content based on the two-dimensional coordinate information on the tracked object 130 .

In step S611 , the immersive media content providing server 100 may output the immersive media content 140 adjusted through the projector 110 and project it onto the object 130 .

In the above description, steps S601 to S611 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: realistic media content providing server
110: projector
120: camera
130: object
140: immersive media content
200: derivation part
210: training data generation unit
220: study unit
230: output unit
240: tracking unit
250: adjustment unit

Claims (17)

In the immersive media content providing server,
a training data generator for generating training data based on a training image for an object to which a tracker is attached and two-dimensional coordinate information for the object in the training image;
a learning unit for learning a tracking model based on the generated training data;
an output unit for outputting immersive media content and projecting it onto an object through a projector that interworks with the immersive media content providing server;
a tracking unit for tracking two-dimensional coordinate information on the moving object through the learned tracking model when the object moves; and
Adjustment unit for adjusting the immersive media content based on the two-dimensional coordinate information of the tracked object
including,
The immersive media content providing server, wherein the output unit outputs the adjusted immersive media content through the projector and projects it on the object.
The method of claim 1,
a derivation unit for deriving two-dimensional coordinate information on the object in the learning image based on camera internal parameter information, camera external parameter information, and three-dimensional coordinate information on the object for the camera that captures the learning image;
Which will further include, immersive media content providing server.
3. The method of claim 2,
The derivation part
Two-dimensional coordinate information for the marker derived from a calibration image for the marker photographed by the camera in a space in which at least one marker is installed, three-dimensional coordinate information for the marker measured from a receiver installed in the space, and the The immersive media content providing server that derives the camera external parameter information based on the camera internal parameter information.
3. The method of claim 2,
The derivation part
By applying a transformation matrix for the posture of the tracker to the coordinate information corresponding to the position of the tracker attached to the object to derive three-dimensional coordinate information for the vertices of each corner of the object to which the tracker is attached, A server that provides realistic media content.
5. The method of claim 4,
The derivation part
Deriving two-dimensional coordinate information about the vertices of each corner of the object in the learning image for the object photographed according to the position change or the posture change of the object,
The learning data generation unit
The immersive media content providing server that generates the learning data by mapping two-dimensional coordinate information about the vertices of each corner of the object and the learning image that is changed according to a change in position or posture of the object.
6. The method of claim 5,
the learning unit
The immersive media content providing server is to learn the tracking model by inputting the training image to the tracking model and outputting two-dimensional coordinate information for vertices of each corner of the object mapped to the training image.
The method of claim 1,
the tracking unit
The immersive media content providing server that tracks the two-dimensional coordinate information for the vertices of each corner of the object from the captured image of the moving object through the learned tracking model.
The method of claim 1,
The tracking model is to be generated for each object having a different shape, immersive media content providing server.
A method of providing immersive media content performed by a immersive media content providing server, the method comprising:
Generating learning data based on the two-dimensional coordinate information for the object in the training image and the training image for the tracker is attached to the object;
training a tracking model based on the generated training data;
outputting the immersive media content through a projector interworking with the immersive media content providing server and projecting the immersive media content onto an object;
tracking two-dimensional coordinate information on the moving object through the learned tracking model when the object moves;
adjusting the immersive media content based on the two-dimensional coordinate information for the tracked object; and
outputting the adjusted immersive media content through the projector and projecting it on the object
A method for providing immersive media content comprising a.
10. The method of claim 9,
Deriving two-dimensional coordinate information for the object in the learning image based on the camera internal parameter information, the camera external parameter information, and the three-dimensional coordinate information about the object for the camera for capturing the learning image
The method further comprising a, immersive media content providing method.
11. The method of claim 10,
Two-dimensional coordinate information for the marker derived from a calibration image for the marker photographed by the camera in a space in which at least one marker is installed, three-dimensional coordinate information for the marker measured from a receiver installed in the space, and the deriving the camera external parameter information based on the camera internal parameter information
The method further comprising a, immersive media content providing method.
11. The method of claim 10,
Deriving three-dimensional coordinate information for the vertices of each corner of the object to which the tracker is attached by applying a transformation matrix for the posture of the tracker to coordinate information corresponding to the position of the tracker attached to the object
The method further comprising a, immersive media content providing method.
13. The method of claim 12,
The step of deriving the two-dimensional coordinate information for the object in the learning image,
Including the step of deriving two-dimensional coordinate information about the vertices of each corner of the object in the learning image for the object taken according to the change in the position or posture of the object,
The step of generating the learning data is,
Providing immersive media content, comprising the step of generating the learning data by mapping two-dimensional coordinate information on the vertices of each corner of the object and the training image that is changed according to a change in position or posture of the object method.
14. The method of claim 13,
The step of training the tracking model is,
The step of inputting the training image into the tracking model and learning the tracking model to output two-dimensional coordinate information for vertices of each corner of the object mapped to the training image by inputting the training image into the tracking model; How to provide.
10. The method of claim 9,
The step of tracking the two-dimensional coordinate information for the moving object,
The method of providing immersive media content, comprising the step of tracking two-dimensional coordinate information for vertices of each corner of the object from the captured image of the moving object through the learned tracking model.
10. The method of claim 9,
The method for providing immersive media content, wherein the tracking model is generated for each object having a different shape.
A computer program stored in a computer-readable recording medium comprising a sequence of instructions for providing immersive media content, which is executed by a immersive media content providing server, comprising:
When the computer program is executed by a computing device,
To generate training data based on the two-dimensional coordinate information for the object in the training image and the training image for the object to which the tracker is attached,
Train a tracking model based on the generated training data,
Output the immersive media content through a projector that interworks with the immersive media content providing server and project it onto an object,
When the object moves, tracking two-dimensional coordinate information about the moving object through the learned tracking model,
Adjust the immersive media content based on the two-dimensional coordinate information for the tracked object,
and a sequence of instructions for outputting the adjusted immersive media content through the projector and projecting it onto the object.

Images