KR102067823B1 - Method and apparatus for operating 2d/3d augument reality technology - Google Patents

Abstract

The present invention relates to a video image based 2D / 3D AR sensory experience method and apparatus. Particularly, the present invention is a dynamic 3D spatial arrangement of a 2D / 3D content photographed at two different sensor viewpoints using different 3D information corresponding to a 2D video image, and having different 3D spatial arrangements of a matcher in the AR content. And a method for providing high quality AR-based AR services in response to dynamic changes in service sensor viewpoints.

Description

 Video image based 2D / 3D AR realistic experience method and device {METHOD AND APPARATUS FOR OPERATING 2D / 3D AUGUMENT REALITY TECHNOLOGY}

The present invention relates to a method for performing augmented reality display through an augmented reality support smart terminal or a large mirror TV panel using augmented reality (AR) technology. In detail, the present invention relates to a method of virtually photographing a celebrity such as a star or experiencing a realistic interaction through an augmented reality display.

Conventional AR service method that can take a photo or experience realistic interaction with celebrities such as stars through augmented reality display using Augmented Reality (AR) technology, Three-Dimensional (2D) and Two-Dimensional (2D) methods can be considered depending on the processing method of AR content that is interactive and augmented.

In this case, the AR content may be a content prepared in advance for the AR experience of the experiencer, and may include a picture or a video of a star or a 3D character and motion information such as a star that the experienced person, such as photographing, will experience. The AR service may be a service that provides a hands-on experience to an experienced user by augmenting and synthesizing AR content through an image of a user who is input live and video of the experienced person or by surrounding space recognition.

First, the 3D approach mainly uses computer graphics technology such as animation characters to augment and synthesize 3D characters created by 3D authoring tools such as Maya and interact with the 3D information of the experienced person. The above-described method may facilitate the real-time 3D interaction processing when combined with a sensor that provides real-time 3D information of the user, such as the location or posture of the user, such as Kinect. In addition, since it is a 3D character in the digital space, it experiences realistic AR experience with the same quality as the existing digital content, depending on the 3D modeling, character rigging, and 3D rendering quality used for the AR service. Can be provided.

However, based on the above-described method, for the AR service with the real person (e.g K-Pop star), the 3D avatar of the real person must be generated through the 3D authoring tool through 3D scanning or 3D modeling. Creating 3D avatars requires a lot of work time, and the 3D digitization process such as 3D modeling, character rigging, and 3D rendering of many elements such as 3D precise shape expression, facial expression, and hairstyle Due to technical constraints, realistic augmentation synthesis is difficult.

In addition, in the 2D approach, video recording may be performed on an existing person or a background of an augmented synthetic object to be used for an AR service. In this case, the AR service can be provided by simply synthesizing 2D video separated from the augmented target part using a chroma key technique to a specific layer such as the front or the rear based on a predetermined experience position of the experienced person. have. For example, it may be a method applied to a current experience-type theme park. In this method, since the captured image of the above-described real person (e.g. K-Pop star) is synthesized as it is in the predetermined position in the AR content, the reality of the augmented target serviced through the display may be high from the experience point of view. However, 3D interaction with the experiencer has a difficult limitation. In addition, the best AR experience can only be achieved when the camera location during AR content shooting and the camera location during AR service are the same, and the location of the AR augmented synthesis target and the experiencer's location meet all the positions in the predefined content scenario. This may be possible. In other words, most AR contents that take a 2D approach have a limitation that the experiencer must adjust its position or posture according to the flow of content because the service is deployed with the location and story determined at the time of content creation regardless of the experience or location of the experiencer. .

An object of the present invention is to provide a video image-based AR experience experience method and apparatus.

An object of the present invention is to provide a video image-based 2D / 3D AR sensory experience method and apparatus.

The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

According to an embodiment of the present invention, an Augmented Reality (AR) service providing method may be provided. In this case, the AR service providing method may include acquiring 3D information and 2D video image information about a first object, acquiring 2D video image information and 3D interaction information of AR content based on the first object, Combining the 2D video image information and the 2D video image information of the AR content with respect to the first object, and outputting the combined 2D video image. In this case, 3D spatial information of the first object is determined based on 3D information and 2D video image information of the first object, 3D spatial information of the AR content is adjusted according to the determined 3D spatial information of the first object, and the AR content. The 2D video image information may be combined with the 2D video image information for the first object based on the 3D spatial information of the adjusted AR content.

In addition, according to an embodiment of the present invention, an apparatus for providing an AR service may be provided. In this case, the apparatus for providing an AR service may include a live RGBD processor, an AR content processor, a 3D interaction processor, and an AR rendering processor. At this time, the 3D information and the 2D video image information of the first object are obtained through the live RGBD processing unit, and the 3D spatial information of the first object is determined based on the 3D information and the 2D video image information of the first object. And acquire 2D video image information and 3D interaction information of the AR content through the AR content processor, adjust 3D spatial information of the AR content according to the determined 3D spatial information of the first object, and adjust the first through the AR rendering processor. The 2D video image information of the AR content and the 2D video image information of the AR content are combined, wherein the 2D video image information of the AR content is combined with the 2D video image information of the first object based on the 3D spatial information of the adjusted AR content. Can be.

In this case, according to an embodiment of the present invention, when the live RGBD input sensor and the AR content unit use different sensors, the 3D information and the 2D video image information of the first object are obtained based on the live RGBD input sensor coordinate system. 2D video image information and 3D interaction information of the AR content may be obtained based on the AR content unit coordinate system.

Further, according to an embodiment of the present invention, the live RGBD input sensor coordinate system and the AR content unit coordinate system are converted into the same experience space coordinate system, and 3D space information of the AR content is based on the experience space coordinate system. Can be adjusted accordingly.

Further, according to an embodiment of the present invention, the live RGBD input sensor coordinate system and the AR content unit coordinate system are converted into the same experience space coordinate system, and 3D space information of the AR content is based on the experience space coordinate system. Can be adjusted accordingly.

In addition, according to an embodiment of the present invention, an apparatus for providing an AR service may be provided. In this case, the device providing the AR service is a live RGBD input sensor unit, AR content unit,

It may include an AR experience processing unit and a display unit. In this case, the AR sensory experience processing unit obtains 3D information and 2D video image information on the first object from the live RGBD input sensor unit, and based on the 3D information and the 2D video image information on the first object. The 3D position of the 3D position is determined, and the AR sensory experience processing unit obtains 2D video image information and 3D interaction information of the AR content from the AR content unit, and adjusts the 3D spatial information of the AR content according to the determined 3D spatial information of the first object. The AR experience processing unit combines the 2D video image information of the AR content and the 2D video image information of the AR content, wherein the 2D video image information of the AR content is based on the adjusted 3D spatial information of the AR content. The 2D video image information is combined with the display unit, and the display unit may output the combined 2D video image information.

The following items may be equally applied to a method and an apparatus for providing an AR service.

According to an embodiment of the present invention, the 3D spatial information of the first object includes at least one or more of a 3D position, attitude, angle, direction, and size of the first object, and the 3D spatial information of the AR content is determined by the AR content. It may include at least one of the 3D position, posture, angle, direction, size.

At this time, according to an embodiment of the present invention, when the environmental information of the first sensor for obtaining information on the first object and the environmental information of the second sensor for obtaining information on the AR content are different from each other, 3D of the AR content The spatial information may be adjusted according to the determined 3D spatial information of the first object.

In addition, according to an embodiment of the present invention, the environmental information of the first sensor includes at least one or more of the position, angle, direction and distance information when the first sensor obtains information about the first object, The environment information of the second sensor may include at least one of position, angle, direction, and distance information when the second sensor acquires information on the AR content.

According to an embodiment of the present invention, the 3D interaction service for the first object may be provided based on the 3D interaction information of the AR content.

In this case, according to an embodiment of the present invention, the 3D interaction service may be a service provided in association with the combined 2D video image.

In addition, according to an embodiment of the present invention, the 3D information on the first object may be information obtained based on at least one or more of a depth image and skeleton information.

In this case, according to an embodiment of the present invention, when the first sensor and the second sensor are different sensors, the 3D information and the 2D video image information of the first object are obtained based on the first coordinate system, and the 2D of the AR content. Video image information and 3D interaction information may be obtained based on the second coordinate system.

In this case, according to an embodiment of the present invention, the first coordinate system and the second coordinate system are converted into the same third coordinate system, and the 3D spatial information of the first object and the 3D spatial information of the AR content may be determined based on the third coordinate system. have.

According to an embodiment of the present invention, an adjustment line may be generated based on the 3D spatial information of the first object, and the 3D spatial information of the AR content may be adjusted based on the generated adjustment line.

In this case, according to an embodiment of the present invention, when the 3D spatial information of the first object is updated, the 3D spatial information of the AR content may be automatically updated based on the updated 3D spatial information of the first object.

At this time, according to an embodiment of the present invention, the step of combining the 2D video image information and the AR content of the 2D video image information for the first object is the depth information of the 2D video image information and the 2D AR content of the first object And augmenting synthesis based on depth information of the video image information.

In this case, according to an embodiment of the present invention, the 2D video image information of the first object and the 2D video image information of the AR content may be augmented and synthesized in consideration of occlusion and visibility.

In addition, according to an embodiment of the present invention, the combined 2D video image may be displayed through a head mounted display (HMD).

According to the present invention, depth image information may be additionally acquired at the time of capturing AR content of an augmented synthesis target for an AR service based on a 2D approach.

According to the present invention, it is possible to provide a method of additionally processing 3D information of an augmented synthesis object based on depth information.

According to the present invention, it is possible to provide a method of linking 2D and 3D AR content information with the location and experience space of a camera in actual AR service, the location of the experience person, and the lighting environment.

According to the present invention, an augmented reality based high quality 3D realistic experience service that enables 3D interaction of a 3D approach by interworking with the 3D information of an experienced user even in augmented composite image quality of a 2D approach and 2D video augmented synthesis may be possible. have.

According to the present invention, 3D interaction with the experiencer is possible at the AR content production time and production cost for the existing 2D content-based AR service, and a 3D realistic experience service that supports natural augmentation synthesis according to the lighting environment of the experience space. May be possible.

According to the present invention, it is possible to provide a distinctive visual differentiation in a service provided to a user in preparation for a realistic experience based on a simple layer-based 2D video augmented synthesis or a 3D avatar-based augmented synthesis.

According to the present invention, the present invention can be applied to a service that provides 3D interaction with a 3D avatar-based experience based on the image quality of the existing 2D video-based approach.

According to the present invention, since there are no services provided by combining the advantages of 2D video and the advantages of 3D avatar through various embodiments of the present invention, the present invention can be applied through a service that is visually provided to an experienced person.

According to the present invention, it is possible to provide a minimum configuration and an implementable embodiment when providing the above-described service.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned above may be clearly understood by those skilled in the art from the following description. will be.

1 is a block diagram showing the configuration of video image-based 2D / 3D AR experience experience service.
2 is a diagram for explaining an example of RGBD information processing according to an embodiment of the present invention.
3 is a block diagram illustrating a detailed configuration of an R video image based 2D / 3D AR experience experience service according to an embodiment of the present invention.
FIG. 4 illustrates a 3D spatial transformation between an experience centered on an experience space coordinate system and an AR content coordinate system according to an embodiment of the present invention.
FIG. 5 illustrates an example of adjusting a 3D coordinate system of AR content using an experience-centered adjustment criterion according to an embodiment of the present invention.
6 illustrates, by way of example, 2D data planar bounding box based processing of an RGB image of AR content, according to an embodiment of the invention.
7 is a flowchart illustrating a method of providing an AR service according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention.

In the following description of embodiments of the present invention, when it is determined that a detailed description of a known structure or function may obscure the gist of the present invention, a detailed description thereof will be omitted. In the drawings, parts irrelevant to the description of the present invention are omitted, and like reference numerals denote like parts.

In the present invention, when a component is "connected", "coupled" or "connected" with another component, it is not only a direct connection, but also an indirect connection in which another component exists in between. It may also include. In addition, when a component "includes" or "having" another component, it means that it may further include another component, without excluding the other component unless otherwise stated. .

In the present invention, the terms "first" and "second" are used only for the purpose of distinguishing one component from other components, and do not limit the order or the importance between the components unless otherwise specified. Accordingly, within the scope of the present invention, a first component in one embodiment may be referred to as a second component in another embodiment, and likewise, a second component in one embodiment may be referred to as a first component in another embodiment. It may also be called.

In the present invention, the components distinguished from each other to clearly describe each feature, and does not necessarily mean that the components are separated. That is, a plurality of components may be integrated into one hardware or software unit, or one component may be distributed and formed into a plurality of hardware or software units. Therefore, even if not mentioned otherwise, such integrated or distributed embodiments are included in the scope of the present invention.

In the present disclosure, components described in various embodiments of the present disclosure are not necessarily required components, and some of them may be optional components. Therefore, an embodiment consisting of a subset of the components described in one embodiment is also included in the scope of the present invention. In addition, embodiments including other components in addition to the components described in the various embodiments are included in the scope of the present invention.

According to the present invention, depth image information may be additionally acquired at the time of capturing AR content of an augmented synthesis target for an AR service based on a 2D approach. In this case, we propose a method of additionally processing 3D information of an augmented composition object based on depth information and a method of linking the 2D / 3D AR content information with the location of the camera, the experience space, and the experience person in the actual AR service. . Through this, we can present a video image-based 2D / 3D AR experience experience that enables 3D interaction of 3D approach by interworking with 3D information of experienced person even in augmented synthesis image quality of 2D approach and 2D video augmentation synthesis. have.

In addition, as an example, the present invention to be described later may reflect the technical requirements of the market that is currently being commercialized in the experience hall or theme park by applying augmented reality technology that is currently in the spotlight in the field of digital content. Specifically, high quality augmented reality-based 3D realistic experience service may be possible through a short production time and low cost through the invention to be described later. Through this, the global VR and AR market analysis may be directly used in the AR market, and is not limited to the above-described embodiment.

On the other hand, from 2018, most smartphones have AR function at OS level through Google Tango-based smart device with built-in RGBD sensor that started commercial distribution in 2016 or Google AR Core and Apple ARKit released in 2017. You can get a supported infrastructure. It is determined that the primary application field of this infrastructure will be an entertainment providing content service including the field of experience described in the present invention, and thus it will have sufficient market demand.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention.

1 is a block diagram showing the configuration of video image-based 2D / 3D AR experience experience service. The present invention may be composed of a live RGBD input sensor unit 1000, a 2D / 3D AR content unit 2000, a 2D / 3D AR sensory experience processing unit 100, and a display unit 3000 as shown in FIG. 1.

 In addition, the present invention can be specified by the 2D / 3D AR sensory experience processing unit 100, the above-described live RGBD input sensor unit 1000, 2D / 3D AR content unit 2000 and the display unit 3000 is the present invention May be external components associated with the operation of. That is, the present invention may be an invention as a system including all of the above-described configurations. Alternatively, the present invention may be the invention only by the configuration of the 2D / 3D AR sensory experience processing unit 100 described above, but is not limited to the above-described embodiment.

In this case, referring to FIG. 1, the live RGBD input sensor unit 1000 may sense an RGB video image and a depth image of an experienced user input for AR service. The 2D / 3D AR content unit 2000 may include an RGB video input, which is an augmented synthesis target input from previously photographed 2D / 3D AR content, and 3D information for each frame of the video input. In addition, the 2D / 3D AR sensory experience processing unit 100 may be in charge of video image-based 2D / 3D AR sensory experience processing through inputs of the live RGBD input sensor unit 1000 and the 2D / 3D AR content unit 2000. . In addition, the display 3000 may present a final augmented composite image to a user.

In this case, as an example, the live RGBD input sensor unit 1000 may use a sensor that provides a real-time RGB color image and depth image such as a Microsoft Kinect sensor or an Intel Real Sensor. For the human body, skeleton information that describes the movement of the human skeletal system can be obtained directly from the sensor or by prior art (eg “J. Shoton et al. Efficient Human Pose Estimation from Single Depth Images, IEEE Trans.On PAMI 2821-2840). , 2013 ”), etc., to obtain it through Depth image analysis of the sensor. In addition, the RGB color image can be used instead of the RGB image of the RGBD sensor input of high-quality RGB image, such as DSLR camera or video cam. Depth image is a dual-camera system using two RGB cameras or a structure-from-motion or SLAM using one RGB camera, in addition to the pattern method or time of flight (TOF) method of the infrared region. It can also be obtained through such a method.

At this time, as an example, the RGB image, the depth image, and the skeleton information may be related to the relative rotation / movement relationship between each sensor or the information expression coordinate system, and the lens characteristic model of the sensor through a camera geometric calibration technique in the field of computer vision. It can be assumed that correction is performed for the back and the like. Such geometric correction assumptions can be typically implemented through various geometric correction techniques in the art. The geometric correction information can be used to calculate the 3D position of the pixel corresponding to the Depth image pixel in the color image corresponding to the Depth image pixel by using the Depth value for each pixel of the Depth image. Depth, Color, and Skeleton information can be combined in the RGBD sensor coordinate system to calculate 3D shape information represented in 3D space. In this case, the number of image frames acquired per second by the RGB sensor and the number of image frames acquired per second by the Depth sensor may be different.If the acquired frames of the Depth sensor are different, the Depth is not acquired by using the frame that simultaneously acquires RGB and Depth as key frame. The RGB frame can utilize the depth information of the adjacent key frame through an interpolation technique.

For example, referring to FIG. 2, 3D information of an experienced user input for AR service may be obtained through the live RGBD input sensor 1000. In this case, as described above, the live RGBD input sensor 1000 may acquire the experiencer's 3D information using at least one of the experiencer's RGB image, the depth image, and the 3D skeleton information. In this case, the 3D information may be information generated by integrating the 3D shape of the experienced person and the skeleton information. In addition, the 3D information may further include 3D location information of the experienced person in consideration of the RGBD sensor coordinate system, but is not limited to the above-described embodiment.

In addition, the experienced person may be an object sensed by the live RGBD input sensor 1000. That is, it may be an object as input information for performing sensing for a 2D / 3D AR sensory experience service. Hereinafter, the present invention will be described through the term experienced person for convenience of description, but the present invention is not limited thereto. That is, the object that can be sensed by the live RGBD input sensor 1000 may be equally applied in the following invention, and is not limited to the above-described embodiment.

Next, the 2D / 3D AR content 2000 uses the live RGBD input sensor 1000 to advance the actual person (eg K-Pop star, etc.) and the background that the AR content provider wants to provide to the experienced person through the AR service. You can shoot and save. In addition, as a result of processing the photographed picture, the depth information of the augmented synthesis target region for each frame of the 2D RGB video image and the video image may be stored. In this case, as an example, when the augmented synthesis target is a human body, the 2D / 3D AR content 2000 may include 3D information such as skeleton information for each frame of the human body.

In addition, as an example, an object for providing an AR service may be extended to another object as well as an actual person. That is, the same can be applied to the object that can be expressed as AR content, and is not limited to the above-described embodiment. For example, the AR content may be provided by an animal or another object. As a specific example, AR content may be provided as an object such as an animal experience, baseball, soccer, or the like. That is, the content that can provide the AR service can be equally applied to the invention described below, but is not limited to the above-described embodiment. However, in the following description, for convenience of explanation, the AR content is described based on a real person, but is not limited thereto.

Next, the AR sensory experience processing unit 100 calls in real time according to the AR service scenario. Even if the AR content is not a person, but a CG model such as a famous animated character, it is possible to render and use 2D RGB video images through high quality rendering in advance, and 3D such as a skeleton or a low-level mesh model of the character used for rendering. May contain information.

In addition, referring to FIG. 1, the 2D / 3D AR realistic experience processing unit 100 includes a live RGBD processing unit 200, a 2D / 3D AR content processing unit 300, a 3D interaction processing unit 400, and an AR rendering processing unit 500. It may include. In this case, the live RGBD processing unit 200 may process image information photographing the experienced person. In addition, the 2D / 3D AR content processing unit 300 may process the 2D / 3D AR content stored in advance according to the location of the experiencer. In addition, the 3D interaction processor 400 may control the 3D interaction between the experiencer and the AR content. In addition, the AR rendering processor 500 may finally transform and match the AR content according to the 3D information and the 3D interaction of the experiencer.

A detailed detailed configuration diagram of each detailed processing unit of the 2D / 3D AR sensory experience processing unit 100 may be configured with FIG. 3, which will be described later. In order to explain the detailed configuration of the 2D / 3D AR sensory experience processing unit 100, the situation of FIG. 4 will be described below by way of example. The experience space of FIG. 5 may be a fixed space arbitrarily set by the AR sensory experience service provider based on the fixed RGBD sensor 1000, which will be described later. In addition, the experience space may be a variable space in the front field of view (FOV) viewed by the sensor based on the RGBD sensor of the photographer who photographs the experiencer using a mobile mobile RGBD sensor, such as photographing. In the former case, the experiencer's coordinate system and the experience space coordinate system are different. In the latter case, the two coordinate systems can be the same or different and can be continuously moved using a smart terminal equipped with an RGBD sensor. The experience space coordinate system may be set at any position, but for convenience of explanation and understanding, the following description assumes that the experience space is set based on the floor of the space where the actual experiencer is located, but is not limited thereto. The range of the experience space may be predefined by the user's selection or upon production of AR content based on the sensing range of the RGBD sensor used for the experience.

The experiencer can be located at any location on the experience space defined in this way. The AR sensory experience service may acquire 3D information such as RGB video image, depth image, skeleton information, etc. of the experiencer based on the coordinate system of the experiencer RGBD sensor 1000 through the experiencer RGBD sensor 1000. In addition, the AR-experience-experienced content service provider may predetermine 2D / 3D AR content of the subject (eg, K-Pop star, etc.) based on the AR content RGBD sensor coordinate system by using an equivalent RGBD sensor such as the experiencer RGBD sensor 1000 in advance. Can be digitalized by photographing.

In this case, as an example, the above-described sensor of the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 may be the same model. As another example, the sensors of the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 may be different models, and are not limited to the above-described embodiment.

Also, as an example, the positions of the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 sensor may be the same based on the experience space coordinate system. As another example, the positions of the sensors of the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 may be differently assumed based on the experience space coordinate system, and are not limited to the above-described embodiment.

In this case, as an example, the data of the coordinate system of the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 may be expressed in one experience space coordinate system.

Referring to FIG. 4, it may be expressed in one experience space coordinate system using a coordinate system transformation relationship with an experience space coordinate system obtained through conventional camera geometric correction. In this case, when the two RGBD sensors used in the live RGBD input sensor 1000 and the 2D / 3D AR content 2000 are different from each other, the RGBD sensor used for the AR content shooting based on the sensor of the live RGBD input sensor 1000 is different. The process of converting information such as internal and external geometric correction factors and color spaces according to the live sensor can be performed.

In this case, referring to FIG. 4, when the experiencer's position is not forced to the AR content 3D position based on the experience space coordinate system, the experiencer's 3D position and the AR content's 3D position may be different from each other in the experience space. For example, this situation is frequently reproduced in AR realistic experience services based on 2D approaches such as stars and photography, which are currently serviced in experience-type theme parks. In other words, the experienced person directly adjusts the position to create a natural AR situation or limit the experience position to a specific point. Due to this 3D spatial mismatch, the interaction between the experiencer and the AR content may not be natural.

Therefore, in the following description, a method of realistically reconstructing and adjusting a 2D video image of the AR content that is finally augmented and synthesized according to the experiencer's 3D position and the experiencer's RGBD sensor position and a method capable of processing 3D interaction are described below. Suggest.

More specifically, referring to FIG. 3, the live RGBD processing unit 200 may include a 3D information detection unit 201 of an experienced person, a 3D data processing unit 202 of an experienced person, and a 2D RGB video image processing unit 203 of an experienced person. have. The live RGBD processor 200 may calculate an image area of the experiencer to be augmented and synthesized in the RGB image by using the 3D information of the experiencer in the experience space and the same. In addition, the live RGBD processing unit 200 may calculate the dense 3D information of the experiencer for 3D interaction with the AR content. As another example, the live RGBD processing unit 200 may calculate the simplified 3D information of the experiencer for 3D interaction with the AR content, and is not limited to the above-described embodiment.

Specifically, referring to the operation of each configuration, the 3D information detector 201 of the experienced person may receive 3D information 1001 such as the depth image and the skeleton information of the experienced person from the live RGBD input sensor unit 1000. In this way, the 3D information detector 201 of the user can detect the location of the user in the experience area. In this case, as described above, since the RGBD sensor is geometrically corrected, the experiencer's position may be calculated based on the 3D coordinate system of the RGBD sensor or based on the 3D coordinate system of the experience area. For example, in the case of expressing the 3D coordinate system of the experience area, the depth value and skeleton information value of each pixel of the Depth image of the RGBD sensor are converted through a transformation matrix of rotation / movement between the RGBD sensor coordinate system and the experience area coordinate system. Can be calculated

The experienced person's 3D information detector 201 may provide the detected person's location and attitude information 1201 in the experience space to the 3D coordinate system controller 301 of the 2D / 3D AR content processor 300. That is, the 3D information detector 201 of the experienced person may provide the posture information 1201 as information necessary for adjusting the information of the AR content based on the coordinate system at the time of capturing the AR content. The linkage relationship of the experienced person 3D information will be described later with reference to the 3D coordinate system adjusting unit 301.

Experiencer's 3D information processing unit 202 uses Depth information input from the RGBD sensor 1000 to determine where the experiencer is occupying in the experience space in the form of mesh, point cloud, or similar 3D information. Calculate closely according to HW computing power. The occupancy information may be used for 3D collision processing detection and interaction processing with AR content in the 3D interaction processing unit 400.

In this case, as an example, a computing capability may be limited in a smartphone. Thus, the dense occupancy information calculated for service in a limited computing environment can be combined with the input Skeleton information. After that, the bounding-box technique or the spherical-approximation technique can be used to approximate / simplify the dense occupancy of the experienced person. In this case, as an example, the bounding-box technique and the spherical-approximation technique may be techniques used for collision checking with surrounding characters or environments in character animation in the field of computer graphics. Through this, the 3D interaction processing unit 400 can reduce the amount of calculation and can easily perform the calculation.

 In addition, as an example, it may be calculated by simplifying to a 3D bounding box or a representative 3D plane including the experiencer. Furthermore, by using the input depth information and skeleton information, it is possible to approximate by modifying the joint length, posture, and 3D shape information of the standard character model.

The dense spatial occupancy information of the experienced person obtained by the experienced person's 3D information processing unit 202 may be provided to the experienced person's 2D RGB video image processing unit 203.

The experiencer's 2D RGB video image processing unit 203 uses the RGB video image input of the experiencer's input from the RGBD sensor and the dense spatial occupancy information input from the experience's 3D information processing unit 202 to view the experience's image area in the RGB image. The operation may be recognized as an image and the remaining image area may be identified as a background image. Dense spatial occupancy information input from the experienced person's 3D information processing unit 202 is used to distinguish the experienced person's image area in a complex and everyday background environment, rather than a chroma-key technique. When classifying, it can be used as the dictionary information of the foreground candidate area when constructing a tri-map using techniques such as Graph-Cut. The foreground image thus obtained may be used as an augmented reality service depending on the service type, or may be serviced by matching the experienced person's image to a mixed reality in a virtual space. Also, in the foreground image, occlusion may occur when the foreground image interacts with the AR content. In this case, when the occlusion occurs, it is possible to determine which information of the experiencer image and the AR content image to be displayed on the actual display unit 3000 in units of pixels. For example, reference information for determining occlusion or visibility may be provided using 3D location information obtained from a depth image corresponding to each foreground color pixel.

In addition, as an example, when providing a service through simple AR augmentation synthesis, the experienced 2D RGB video image processing unit 203 may pass the input RGB video image 1002 to the next step as it is without the above-described processing.

That is, as described above, when the augmented synthesis considering the mutual blindness between the experiencer and the AR content or the background of the AR content that is not the background of the AR service space is not synthesized on the basis of mixed reality (MR), input RGB video The image 1002 can be passed to the next step as it is without the above-described processing, but is not limited to the above-described embodiment.

The 2D / 3D AR content processor 300 may include a 3D coordinate system adjuster 301, a 2D / 3D data aligner 302, a 3D data processor 303, and a 2D data processor 304 of AR content. In this case, the 3D coordinate system adjusting unit 301 may adjust the 3D coordinate system. In addition, the 2D / 3D data alignment unit uses the AR content information and the geometric correction information of the RGBD sensor at the time of AR content shooting to live 3D information such as 2D RGB information (2002), depth information, skeleton information, etc. of the augmented composition target. The calculation may be performed as in the result of the RGBD processing unit 200 and registered in the operation memory. The 3D data processor 303 of the AR content may process data of the AR content. Also, the 2D data processor 304 may transform the RGB image information of the AR content based on the 3D information adjusted to the experiencer's location information. In this case, as illustrated in FIG. 6, 3D information for 2D video image and 3D interaction information of AR content may be calculated based on the experiencer's information, which will be described later.

In more detail, referring to FIG. 5, the 3D coordinate system adjusting unit 301 performs calculation using the 3D position of the experienced person calculated by the 3D information detection unit 201 of the experienced person and the 3D position of the AR content converted into the experience space coordinate system. Can be done. In this case, the 3D coordinate system adjusting unit 301 may calculate adjustment conversion information on how the AR content should be adjusted according to the AR content service scenario based on the experiencer's location through the above information.

As an example, an example of an AR content service scenario may be as follows. When the AR content is automatically located next to the experiencer and assumes a scenario in which a 3D interaction is taken according to the experiencer's posture and photographing, the 3D coordinate system adjusting unit 301 adjusts based on the experiencer's position as illustrated in FIG. 5. You can create a line. In this case, the 3D coordinate system adjusting unit 301 may calculate adjustment transformation information for allowing the AR content to be parallel to the experiencer and the adjusting line using the 3D position of the current AR content.

The 3D data processor 303 of the AR content may perform an update so that 3D information of the AR content may be located next to the experienced person in the actual digital experience space as shown in FIG. 5. At this time, the 3D data processing unit 303 of the AR content applies adjustment transformation information to 3D information such as a depth image and skeleton information, a skeleton-based spherical or bounding-box model, a 3D bounding box, and a representative 3D plane of the AR content stored in advance. The above-described update can be performed. Thereafter, the 3D data processor 303 of the AR content may provide the 2D data processor 304 with adjustment conversion information between the last updated 3D information and the input 3D information 2001 as a result. Through this, the 2D data processor 304 may use the above-described information for converting the RGB video 2002 information. In this process, information such as skeleton-based spherical or bounding-box models, 3D bounding boxes, representative 3D planes, low-resolution polygonal meshes, such as the 3D information processing unit 202 of the experienced user, is preliminarily included in the 3D information 2001 in the AR content. When not input, the 3D information processing unit 202 may simplify the 3D information for detecting and processing the 3D interaction with the experiencer in the same manner as the 3D information processing unit 202.

The 2D / 3D data aligner 302 uses the AR content information and the geometric correction information of the RGBD sensor at the time of AR content capture to display the 2D RGB information, the depth information, and the skeleton information of the augmented composition target with the result of the live RGBD processing unit 200. Similarly, the AR content can be calculated and registered as an operation memory based on the RGBD sensor coordinate system at the time of shooting. However, unlike the live RGBD processing unit 200, the result of the experiencer's 2D RGB video image processing unit 203 may be calculated and stored in advance and loaded into a memory. In addition, in order to improve AR quality of AR content, the quality of the foreground area of the AR composition may be improved by photographing in a controlled environment such as a chroma key environment instead of a general environment when the AR content is captured. As an example, the 2D data processor 304 may perform shooting in a controlled environment such as a chroma key environment when shooting a real person (e.g K-Pop star). In addition to extracting skeleton information using Depth information of RGBD sensor, high quality skeleton information can be obtained by linking with specialized motion capture equipment. The foreground area information of the 2D RGB image of the AR content may include alpha blending information for a natural augmentation synthesis process of the boundary surface of the K-Pop star's hair or the foreground silhouette.

The 2D data processor 304 may control the collision process based on the 3D information adjusted by the 3D data processor 303 of the AR content and the 3D information of the experience person according to the experiencer location information. In addition, the 2D data processor 304 may perform a preliminary work for 2D / 3D transformation 501 and augmentation synthesis 502 of AR content in accordance with the live sensor viewpoint corresponding to the 3D interaction. In addition, based on the information arranged through the 2D / 3D data aligning unit 302, as shown in FIG. 4, AR content may be expressed in 3D space using 2D video image and depth information for 3D transformation based on the experiential space coordinate system. have. The 2D data processor 304 may interwork with the 2D / 3D data and the ping unit 501 of the AR rendering processor 500 to perform 3D transformation for augmenting and synthesizing the 2D data from the experience sensor's live sensor point in 3D space. have. This operation may be performed in various ways in consideration of the computing power of the computing device and the display resolution being serviced.

The following describes how the AR content is processed at the live sensor's viewpoint by the combination of the 3D data processor 303, the 2D data processor 304, and the 2D / 3D data warping unit 501, through some embodiments. .

For example, the easiest implementation method may be a method of generating a virtual 3D plane and registering and rendering the image of each frame of the 2D video as a texture map in the plane. At this time, the selection of the position and size of the 3D plane is important, and the position may be the average position of the 3D space occupancy of the augmented synthesis object at the time of AR content shooting as shown in FIG. In addition, the size of the 3D plane is a 3D planar bounding box (hereinafter referred to as the 3D plane) on the 3D plane that most tightly includes the actual occupied size in the 3D space of the augmented composite object obtained based on the geometric correction information of the RGBD sensor at the time of shooting. ) And register the bounding box RGB image area of the bounding box projected by affine projection or the like according to the camera projection model of the RGBD sensor at the time of AR content shooting as a texture map of the 3D planar bounding box. Also, in order to maximize the amount of visualization information when setting the 3D plane, the 3D plane can be arranged perpendicular to the experience space and the planes can be placed facing each other. In this case, four vertices of the 3D plane generated for the registration of the texture map may be projected onto the RGB image of the AR content by using an inverse transformation relationship between the RGBD sensor coordinate system and the experience space coordinate system at the time of AR content shooting. You can register the area connecting the four projected vertices as a texture map. Also, if the 2D RGB image includes alpha map information for alpha blending for natural composition of the boundary surface during augmented synthesis, when the RGB image area is registered as a texture map, the alpha information is added to the texture map alpha map. Registering the mask information allows the 3D rendering engine to calculate the natural alpha blending effect when rendering the image. However, since there is a difference in the position between the experiencer and the augmented synthesis object, the position and orientation of the 3D plane are recalculated according to the 3D transform equation using the adjustment transformation information of the 3D data processing unit 303 of the AR content, and the 2D RGB image 1002 of the experiencer is calculated. Projection is performed using the inverse transformation relationship between the RGBD sensor and the experience space, and the RGB image of the AR content reflecting the image transformation by perspective can be obtained. The experienced person can see the AR image as if a real person (e.g K-Pop star) is next to the reality as shown in FIG. Such an implementation can realize high realistic augmented synthesis with a small amount of computation, but if the actual spatial occupancy is distributed vertically in the normal direction of the generated 3D plane according to the pose of the augmented composite object (eg, the sensor is The greater the difference between the position / angle of the RGBD sensor during AR content shooting and the position / angle of the RGBD sensor during AR service, the more the image conversion quality may be distorted.

In another implementation method, a 3D vertex is generated using 3D location information recalculated for each RGB pixel by the 3D data processor 303 for each pixel constituting the foreground image of the AR content and correlated with adjacent pixels. Using this information, we generated a 3D local mesh model consisting of triangles or squares and projected the topology of the mesh model onto the RGB image using the inverse transformation relationship between the RGBD sensor coordinate system and the experiential space coordinate system at the time of AR content shooting. Register the corresponding RGB image areas as a texture map. Next, by using the adjustment transformation information calculated by the 3D coordinate system adjusting unit 301, the positions of the vertices constituting the mesh model are 3D transformed, and the resultant mesh model is 3D rendered at the viewpoint of the RGB image of the experienced person and augmented and synthesized. The experienced person will experience the AR effect as if the augmented synthetic target is next to it, as shown in FIG. 5. In addition, if the 2D RGB image includes alpha map information for alpha blending in the same manner as above, when the RGB image is registered as the texture map, the 2D RGB image is registered to the alpha map mask information of the texture map. This implementation method increases the calculation amount compared to the former 3D plane approximation method, but it is not affected by the 3D space occupied form according to the pose or position of the augmented composite object when converting RGB images, and the sensor viewpoint difference between AR content shooting and live service. It is not affected by image distortion.

In this case, if the point of time when shooting AR content is different from the point of time of the sensor serving as a live service, when the 3D information of the 3D meshed AR content is rendered at the time of the live sensor, the information that was not visible at the time of AR content shooting is not displayed at the time of live content In this case, the experiencer sees a hole in AR content that is augmented and synthesized. In order to solve this problem, AR content creates one global mesh model using both RGB and depth images of each frame that are developed on the time axis, and the actual person (eg K-Pop star) in AR content for each frame. Correspondences that explain the deformation relationship between the global mesh model and the local mesh model for each frame according to the movement of the back can be found. Using this correspondence, color information of the mesh, which was not visible due to occlusion in a specific frame, for each mesh of the global mesh model can be obtained in a frame having different visibility on the time axis. Such a global mesh model may be calculated and provided by offline calculation after AR content shooting, or may be calculated in a live service if necessary.

By constructing such a global mesh model, if a hole occurs during augmented synthesis of AR content at the time of a live sensor, the color information of the pixel where the hole is generated is used to find a color value in the global mesh model and use it during augmented synthesis. The color of the AR content at that time can be seen.

As another example, Surfel rendering may be used. Surfel is created at the 3D position of each pixel constituting the foreground image of the AR content, and the normal information of the surfel can be set using the 3D information of neighboring pixels. Next, color texture information may be registered for each surfel by projecting an area of a surfel having a certain radius onto an RGB image. Next, when each surfel is 3D converted and rendered according to the live view of the experiencer, the experiencer can bring an augmentation effect that the AR content is interacting in the same space as the experiencer. The advantage of this method over the above two methods is that it can easily solve the image distortion problem when the AR content and the live sensor viewpoint difference are large due to the 3D planar approximation of the first method. The hole problem due to the viewpoint difference caused by the method based on the local mesh model of the second method can be solved by using optimization techniques to adjust the radius of the surfel and the normal to the viewpoint difference between the two sensors.

Another method may be using a light field sensor. A view and focus can be defined according to a live sensor view, and an image of a corresponding view can be generated and applied to the methodology above to serve.

As described above, the 2D data processor 304 corresponding to the preprocessor and the 2D / 3D data warping unit 501 corresponding to the post processor may be configured in various embodiments.

The 3D interaction processor 400 between the experiencer and the AR content transforms the 3D information of the AR content through the 3D coordinate system controller 301 and the 3D data processor 303 of the AR content according to the 3D spatial position of the experiencer, and then changes the 3D information of the experiencer. Interaction can be handled by comparing information and experience space under experience space coordinate system.

When the interaction processing unit 400 collides with the 3D collision processing detection unit 401 for checking whether the experience person artificially requires 3D interaction based on the 3D information of the AR content whose 3D information is adjusted based on the experience person, 3D interaction processing unit 402 for generating an interaction through a 3D interaction predefined in the AR service according to the intention of the experience.

First, the 3D impulse process detecting unit 401 is a skeleton-based spherical or bounding-box model, a 3D bounding box, a representative 3D plane, and the like generated by the 3D information processor 202 and the 3D data processor 303 of the AR content. You can check for conflicts. In this case, when there is a collision, the 3D interaction processing unit 402 may transmit the type information in which the 3D collision occurs. Collision checking can be confirmed by checking whether unit 3D objects, such as spheres or bounding boxes and low-resolution polygonal meshes, which are constructed by simplifying the appearance of the body based on a skeleton, have overlapping in 3D space. For example, if a 3D unit object that simplifies the left hand attached to the skeleton representing the skeleton of the experiencer's left arm collides with the right hand of the AR content, the type information indicates that the experiencer is the real person of the AR content (eg K-Star). You can decide that you want to interact with each other. Therefore, when the corresponding collision type information is sent to the 3D interaction processing unit 402, the interaction processing unit augments and synthesizes the content of taking pictures by holding hands among 2D / 3D reaction contents prepared in advance according to the type information. Can be. Through this, it is possible to provide a high quality image based 3D interaction with the 2D / 3D real person (e.g. K-Star) and the like in the actual AR content using the 2D video image.

The AR rendering processor 500 may perform a function of finally transforming, matching, and rendering AR content to the display 3000 in accordance with the 3D information and 3D interaction of the experienced person. In this case, the AR rendering processor 500 may include a 2D / 3D data mapping unit 501, a 2D / 3D data blending unit 502, and a 2D video matching unit 503.

The 2D / 3D data warping unit 501 has been described through various embodiments associated with the 2D data processing unit 304, as described above. The 2D / 3D data warping unit 501 arranges the 2D video image on the AR content in 3D space according to the 3D space position of the experience person and the live sensor viewpoint in the experience space, and transforms the position and direction in the 3D space according to the embodiment. It can play a role. The 2D / 3D data transformed in the 3D space can be arranged to correspond to the real space on a content authoring tool such as Unity3D and the 3D rendering engine.

The 2D / 3D data rendering unit 502 performs 3D rendering including lighting effects through environment map or lighting (Point / Directional light source) in which lighting information of a predefined experience space is placed in the content. Can be done. In this process, the 3D data processing unit 303 uses the information used in the 3D space arrangement process of the 2D / 3D data and the layout lighting information for a natural AR augmented synthesis effect, and an experience space of a real person (eg K-Star) in the AR content. You can create natural shadows according to your placement. For example, when placed on a 3D plane, alphamap information of 2D data may be used, and when using a local mesh model, realistic shadows may be generated by physically simulating lighting directly on the mesh model. In the case of 2D video image, the color space of the 2D video is simulated according to the lighting environment of the experience space using the color space conversion information and the lighting environment map information between the sensor that captured the AR content and the sensor for the experience service during the rendering process. Can be reflected in the 2D video of the AR content.

The 2D video matching unit 503 is a 2D video image of the AR content input from the live input sensor 1000 and processed by the live 2D data processing unit 203 and finally rendered by the 2D / 3D data blending unit 502. It can perform a function of augmented synthesis into a single image. Augmented synthesis occludes the actual experience of the two video information, the actual person (eg K-Star) in the AR content, and the pixel information of the shadow through a visibility test based on the depth information of the two images. May be augmented and synthesized.

The final augmented synthesized 2D video image is provided to the experienced person through the display unit 3000 so that the experienced person can see the real person of the actual AR content (eg K-Star) through 3D interaction through high quality video and 3D information of the 2D video image of the AR content. You can experience as if you are right next to).

When the display unit 3000 is a head mounted display (HMD) that supports optical see-through or video see-through, the AR rendering processor 500 performs stereo through two virtual cameras set according to the interspace distance of the user. The embodiment can be extended to provide a three-dimensional experience by performing a scopic rendering.

7 is a flowchart illustrating a method of providing an AR service.

The device providing the AR service may acquire 3D information and 2D video image information about the first target. In this case, as described above with reference to FIGS. 1 to 6, the apparatus for providing the AR service may be an AR experience processing unit 100, a live RGBD input sensor unit 1000, an AR content unit 2000, and The display 3000 may be an external device. As another example, an apparatus for providing an AR service may be a device including all of the above-described live RGBD input sensor unit 1000, AR content unit 2000, display unit 3000, and AR sensory experience processing unit 100. The present invention is not limited to the above-described embodiment.

Also, as an example, the first object may be the experienced person described above. That is, as described above, it may refer to an object that can be sensed through the live RGBD sensor unit 1000, and is not limited to the above-described embodiment. In this case, the AR service apparatus may obtain 3D information and 2D video image information about the first target (ie, the user). In this case, as an example, the 2D video image information may be RGB video image information of the first target. The 3D information may include at least one of depth image information and skeleton information.

Next, 2D video image information and 3D interaction information of AR content may be acquired based on the first object. In this case, as described above with reference to FIGS. 1 to 6, the AR content may be a real person. Also, the AR content may be content set by a provider, and is not limited to an actual person. In this case, for example, the AR content may be AR content obtained from the above-described AR content unit 2000. In this case, the AR content may include an RGB video input, which is an augmented synthesis target input from the previously obtained 2D / 3D AR content, and 3D information for each frame of the video input. That is, it may include 2D video image information and 3D interaction information. In this case, as an example, the 3D interaction information may be 3D information. That is, the information is not limited only to the interaction.

Next, the AR service providing apparatus may combine 2D video image information of the first object and 2D video image information of the AR content. As described above with reference to FIGS. 1 to 6, when a sensor (eg, a live RGBD input sensor) for a first object and a sensor for an AR content (eg, a sensor of an AR content unit) are different from each other, it is used in each sensor. The coordinate system can be different. At this time, it is necessary to convert the above-described coordinate system into the same coordinate system (e.g. experience space coordinate system) as described above. That is, there is a need to convert 3D spatial information of the AR content based on the 3D spatial information of the first object. For example, the 3D spatial information may be information determined based on at least one or more of a 3D position, posture, angle, direction, and size of the object. That is, the 3D space information may refer to information on the 3D space for the object, as described above. More specifically, as described above, it is necessary to determine 3D spatial information of the first object in order to receive the AR service regardless of the position, attitude, angle, etc. of the first object. Through this, even if the first target does not move to the designated location, the AR service may be provided. Also, as an example, the above-described information about the first object may be obtained through the first sensor, and the information about the AR content may be information about the second sensor. For example, the first sensor and the second sensor may include a camera. In this case, the first sensor and the second sensor may be different from each other, the environmental information that is information when obtaining information about each object. In this case, the environment information may be at least one or more of information about a position, an angle, a directional distance, etc. when the sensor acquires the target information. For example, the position of the camera, the distance and the direction between the angle camera and the object may be the above-mentioned environmental information, and are not limited to the above-described embodiment.

In addition, as an example, 2D video image information of the first object and 2D video image information of the AR content may also be combined based on the converted coordinate system.

Next, the AR service providing apparatus may output the combined 2D video image. In this case, as described above with reference to FIGS. 1 to 6, the AR service providing apparatus may output the combined 2D video image through the display 3000. Also, as an example, the AR service providing apparatus may provide a 3D interaction service together with a 2D video image. That is, the 3D interaction service related to the AR content may be provided to the first object (or experiencer) based on the AR content converted and combined based on the coordinate system. In this case, the 3D interaction service may be provided at a location adjacent to the first object, and is not limited to the above-described embodiment. In this way, the first object may be provided with a 3D interaction service based on the combined 2D video image.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, a software module, or a combination of the two executed by a processor. The software module may reside in a storage medium (ie, memory and / or storage) such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM. An exemplary storage medium is coupled to the processor, which can read information from and write information to the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

Exemplary methods of the present invention are presented as a series of operations for clarity of description, but are not intended to limit the order in which the steps are performed, and each step may be performed simultaneously or in a different order as necessary. In order to implement the method according to the present invention, the steps illustrated may include additional steps in addition to, may include other steps except for some steps, or may include additional other steps except for some steps.

The various embodiments of the present invention are not intended to list all possible combinations, but to describe representative aspects of the present invention, and the matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For hardware implementations, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), General Purpose It may be implemented by a general processor, a controller, a microcontroller, a microprocessor, and the like.

It is intended that the scope of the invention include software or machine-executable instructions (e.g., operating system, applications, firmware, programs, etc.), and such software or software, for causing operations in accordance with various embodiments of the methods to be executed on a device or computer. Instructions, and the like, including non-transitory computer-readable media that are stored and executable on a device or computer.

100: AR experience experience processing unit 200: live RGBD processing unit
300: 2D / 3D AR content processing unit 400: 3D interaction processing unit
500: AR rendering processing unit 1000: live RGBD input sensor
2000: AR content unit 3000: Display unit

Claims (19)

In the augmented reality (AR) service providing method,
Acquiring 3D information and 2D video image information on a first object;
Acquiring 2D video image information and 3D interaction information of AR content based on the first object;
Combining the 2D video image information of the first object and the 2D video image information of the AR content; And
And outputting the combined 2D video image.
Determine 3D spatial information of the first object based on the 3D information and the 2D video image information about the first object,
Adjust 3D spatial information of the AR content according to the determined 3D spatial information of the first object,
The 2D video image information of the AR content is combined with the 2D video image information for the first object based on the 3D spatial information of the adjusted AR content.
The 2D video image information of the AR content is an RGB video input that is an object of augmented synthesis inputted from 2D / 3D AR content photographed in advance.
3D interaction information of the AR content is characterized in that the 3D information per frame of the RGB video input,
How to provide AR services. The method of claim 1,
3D spatial information of the first object includes at least one of 3D position, posture, angle, direction, and size of the first object,
The 3D spatial information of the AR content includes at least one or more of the 3D position, attitude, angle, direction, size of the AR content. The method of claim 2,
When the environmental information of the first sensor obtaining the information on the first object and the environmental information of the second sensor obtaining the information on the AR content are different from each other, 3D spatial information of the AR content may be converted into And adjusting according to the determined 3D spatial information. The method of claim 3, wherein
The environment information of the first sensor includes at least one of position, angle, direction, and distance information when the first sensor obtains information on the first object,
And the environment information of the second sensor includes at least one of position, angle, direction, and distance information when the second sensor obtains information on the AR content. The method of claim 1,
And providing a 3D interaction service for the first object based on the 3D interaction information of the AR content. The method of claim 5,
The 3D interaction service is a service provided in conjunction with the combined 2D video image, AR service providing method. The method of claim 1,
The 3D information on the first object is information obtained based on at least one or more of a depth image and skeleton information, AR service providing method. The method of claim 3, wherein
When the first sensor and the second sensor are different sensors, the 3D information and the 2D video image information of the first object are obtained based on a first coordinate system.
2D video image information and 3D interaction information of the AR content is obtained based on a second coordinate system. The method of claim 8,
The first coordinate system and the second coordinate system are converted into the same third coordinate system, and the 3D spatial information of the first object and the 3D spatial information of the AR content are determined based on the third coordinate system. Way. The method of claim 1,
Generate an adjustment line based on the 3D spatial information of the first object,
And the 3D spatial information of the AR content is adjusted according to the 3D spatial information of the first object based on the generated adjustment line. The method of claim 10,
And when the 3D spatial information of the first object is updated, the 3D spatial information of the AR content is automatically updated based on the updated 3D spatial information of the first object. The method of claim 1,
Combining the 2D video image information for the first object and the 2D video image information of the AR content; depth information of the 2D video image information for the first object and the 2D video image of the AR content And augmented and synthesized based on depth information of the information. The method of claim 12,
And the 2D video image information of the first object and the 2D video image information of the AR content are augmented and synthesized in consideration of occlusion and visibility. The method of claim 1,
The combined 2D video image is displayed through a head mounted display (HMD), AR service providing method. In the device for providing augmented reality (AR) services,
A live RGBD processor;
AR content processing unit;
3D interaction processing unit; And
Including an AR rendering processor;
Acquire 3D information and 2D video image information on a first object through the live RGBD processor,
3D spatial information of the first object is determined based on the 3D information and the 2D video image information about the first object,
Acquire 2D video image information and 3D interaction information of AR content through the AR content processor,
Adjust 3D spatial information of the AR content according to the determined 3D spatial information of the first object,
Combine the 2D video image information of the AR and the 2D video image information of the AR content through the AR rendering processor,
The 2D video image information of the AR content is combined with the 2D video image information for the first object based on the 3D spatial information of the adjusted AR content.
The 2D video image information of the AR content is an RGB video input that is an object of augmented synthesis inputted from 2D / 3D AR content photographed in advance.
3D interaction information of the AR content is characterized in that the 3D information per frame of the RGB video input,
AR service providing device. The method of claim 15,
The live RGBD processing unit obtains the 3D information and the 2D video image information about the first object from a live RGBD input sensor,
The AR content processor obtains the 2D video image information and the 3D interaction information of the AR content from an AR content unit,
And 2D video image information combining the 2D video image information of the first object and the 2D video image information of the AR content is output through a display unit. The method of claim 16,
When the live RGBD input sensor and the AR content unit use different sensors, the 3D information and the 2D video image information of the first object are obtained based on the live RGBD input sensor coordinate system.
2D video image information and 3D interaction information of the AR content is obtained based on the AR content unit coordinate system. The method of claim 17,
The live RGBD input sensor coordinate system and the AR content unit coordinate system are converted into the same experience space coordinate system,
And the 3D spatial information of the AR content is adjusted based on the 3D spatial information of the first object based on the experience space coordinate system. In the device for providing augmented reality (AR) services,
A live RGBD input sensor unit;
AR content unit;
AR experience experience processing unit; And
Including; display unit,
The AR sensory experience processing unit obtains 3D information and 2D video image information about a first object from the live RGBD input sensor unit,
3D spatial information of the first object is determined based on the 3D information and the 2D video image information about the first object,
The AR experience processing unit obtains 2D video image information and 3D interaction information of AR content from the AR content unit,
Adjust 3D spatial information of the AR content according to the determined 3D spatial information of the first object,
The AR sensory experience processing unit combines the 2D video image information for the first object and the 2D video image information of the AR content,
The 2D video image information of the AR content is combined with the 2D video image information for the first object based on the 3D spatial information of the adjusted AR content,
The display unit outputs the combined 2D video image information,
The 2D video image information of the AR content is an RGB video input that is an object of augmented synthesis inputted from 2D / 3D AR content photographed in advance.
3D interaction information of the AR content is characterized in that the 3D information per frame of the RGB video input,
AR service providing device.

Images