KR20230010601A - Method for Providing Message-Based XR Mobile Services Using Augmented Reality and System Supporting It - Google Patents

Abstract

According to the present invention, a system for supporting message-based XR mobile service using augmented reality includes: a terminal which produces and provides a message containing XR content, and executes XR content included in the message provided from the outside in response to a predetermined condition; and an XR service platform for supporting an interface for producing and executing the XR contents in the terminal.

Description

Method for Providing Message-Based XR Mobile Services Using Augmented Reality and System Supporting It

The present invention relates to a method for providing a message-based XR mobile service using augmented reality and a system supporting the same.

In general, Augmented Reality (AR) technology refers to a technology that provides users with virtual objects created by computer graphics superimposed on objects existing in the real world, and recently, the implementation of augmented reality has been implemented on computers and mobile phones (smartphones). , As it extends to end points such as tablets and smart glasses, it corresponds to a technology in which research and development are being concentrated.

Augmented Reality is a technology that superimposes virtual objects on the real world that users see. It is called Mixed Reality (MR) because it combines the real world with the virtual world with additional information in real time and shows it as a single image.

Augmented reality, a concept that complements the real world with the virtual world, uses a virtual environment created by computer graphics, but the main character is the real environment. Computer graphics play a role in providing additional information necessary for the real environment.

Unlike the virtual reality technology composed of only the virtual environment, this augmented reality technology provides a mixed image of the real environment and virtual objects, so that the user can see the real environment to provide a better sense of reality and additional information.

Augmented reality technology is used for remote medical diagnosis, broadcasting, architectural design, and manufacturing process management. In particular, with the recent spread of smart phones, they have entered a full-fledged commercialization stage, and are developing various products in the game and mobile solution industries and education fields.

In addition, although various types of augmented reality are provided while moving using smart glasses, when the amount of augmented reality is large, unwanted augmented reality is also provided, making it difficult to easily find the desired augmented reality.

On the other hand, with the recent emergence of Internet services such as Facebook, YouTube, and Wikipedia, users do not just utilize already created content, but also create, freely post, and share multimedia content themselves, and market As can be seen from the rapid growth in <<, there is a clear tendency that the boundary between virtual and real spaces gradually disappears.

Publication No. 10-2021-0060474 (Title of Invention: Augmented Reality Content Sharing System and Method) Publication No. 10-2018-0058880 (Title of Invention: Method for Providing Augmented Reality Contents Based on Product Image Recognition)

An object of the present invention is to provide a message-based XR mobile service providing method using augmented reality that can solve the conventional problems and a system supporting the same.

A system for supporting a message-based XR mobile service using augmented reality according to an embodiment of the present invention for solving the above problems produces and provides a message containing XR content, and responds to a preset condition to provide an externally provided message. a terminal executing the XR content included in the message; and an XR service platform supporting an interface for creating and executing the XR content in the terminal.

In one embodiment, the terminal extracts at least one or more objects in the captured real image, processes an object designated by a user among the extracted objects into a 3D XR object, and then mixes the processed 3D XR object in the real image. It is characterized by generating an XR image.

In one embodiment, the terminal is characterized in that an image object specified by a user among extracted image objects is created or processed as XR contents.

In one embodiment, the terminal generates a 3D virtual image, replaces an image object specified by the user with one of various XR contents supported by the UI, or provides XR contents to be included in a captured image from the UI. It is characterized by being provided.

In one embodiment, the terminal includes a rendering unit that generates an XR image obtained by rendering the captured image and the created stereoscopic XR image object, and the rendering unit is interlocked with an interaction unit to execute the XR image object in the terminal. It is characterized in that a corresponding stereoscopic XR image is generated to include interaction information.

In one embodiment, the terminal is characterized in that it generates a message containing the XR image rendered by the rendering unit.

In one embodiment, the terminal includes an interaction unit for controlling an XR object in an XR video message received from the outside to react or execute under a preset condition, and the interaction unit controls the rendering unit to react or execute the XR object under a preset condition. It is characterized in that an interaction condition value for execution is given.

In one embodiment, the terminal detects the XR content in the stereoscopic XR image exposed on the screen under a preset detection condition, for example, when the XR content is located in the user's body (hand) or the user's gesture or motion. It is characterized by doing.

A mobile service providing method using message-based XR technology according to an embodiment of the present invention for solving the above problems includes receiving a captured image captured by an image capturing unit; generating XR contents to be rendered within the photographed image captured by the XR contents creation unit from a UI or generating XR contents similar to an image object specified by a user; generating an XR image in which the XR content in the captured image is rendered by a user-specified condition value in a rendering unit; generating an XR video message including an XR video in an XR video message generating unit; and when the XR video message generating unit generates an XR video message containing XR content, the input/output unit transmits or shares the XR video message to a number or SNS designated by the user.

A mobile service providing method using message-based XR technology according to another embodiment of the present invention for solving the above problems includes receiving an XR video message from an external terminal in an input/output unit; Executing XR content according to the interaction value set in the XR video message in an interaction unit; detecting, by an XR content detector, the XR content in the stereoscopic XR image exposed on the screen under a preset detection condition, for example, when the XR content is located within the user's body (hand) or whether there is a user's gesture or motion; and outputting an effect (vibration, sound, etc.) corresponding to the XR content in the effect unit.

Using the message-based XR mobile service providing method using augmented reality and a system supporting the same according to an embodiment of the present invention, the problem of simply selecting a gift and delivering it as a message as in the prior art, e.g., emotions It has the advantage of being able to solve problems that could not be conveyed through rich expression.

In other words, there is an advantage in that it can satisfy the emotional delivery part in conveying the heart compared to directly seeing and delivering the gift offline.

Through the advantages described above, users can experience the space they want to go or the space they want to deliver a gift to (for example, a space where they want to propose) through virtual content and avatars that are actually filmed. It has the advantage of providing pleasure.

In addition, you can present a different experience with realistic contents and gifts that are one level different from the feeling of receiving a mobile gift.

1 is a device configuration diagram of a system supporting a message-based XR mobile service using augmented reality according to an embodiment of the present invention.
2 is a flowchart illustrating a method for providing a message-based XR mobile service using augmented reality according to an embodiment of the present invention.
3 is a flowchart illustrating a method for providing a message-based XR mobile service using augmented reality according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, based on the accompanying drawings, a message-based XR mobile service providing method using augmented reality according to an embodiment of the present invention and a system supporting the same will be described in more detail.

1 is a device configuration diagram of a system supporting a message-based XR mobile service using augmented reality according to an embodiment of the present invention.

As shown in FIG. 1, the system 100 supporting a message-based XR mobile service using augmented reality according to an embodiment of the present invention may be a terminal running an XR service platform, and the XR service platform It may be configured to produce and provide a message including 3D XR content using the 3D XR content, and to display and acquire 3D XR content included in the externally provided message in response to a preset motion.

In addition, the terminal 100 extracts at least one or more objects in a real image captured using the XR service platform, processes an object designated by a user among the extracted objects into a stereoscopic XR object, and then processes the object in the real image. It may be configured to generate an XR image in which a stereoscopic XR object is mixed.

In addition, the terminal 100 may be configured to generate an avatar based on a photographed image of a user using an XR service platform.

In addition, the terminal 100 may be configured to execute stereoscopic XR contents included in the message after receiving a message including the XR contents received from the outside using the XR service platform.

In addition, the terminal 100 may be configured to collect or obtain the corresponding XR content when the stereoscopic XR content executed using the XR service platform is located within a preset motion or body region of the user.

In addition, the terminal 100 may be configured to output haptic information set in the stereoscopic XR content when the stereoscopic XR content executed using the XR service platform is located within a preset motion or body region of the user.

Meanwhile, the terminal 100 mentioned herein is a smart phone, a portable terminal, a mobile terminal, a foldable terminal, and a personal digital assistant (PDA). ), PMP (Portable Multimedia Player) terminal, telematics terminal, navigation terminal, personal computer, notebook computer, slate PC, tablet PC, ultrabook ), wearable device (including, for example, a watch type terminal (Smartwatch), a glass type terminal (Smart Glass), HMD (Head Mounted Display), etc.), Wibro terminal, IPTV (Internet Protocol Television) terminal It can be applied to various terminals such as smart TVs, digital broadcasting terminals, AVN (Audio Video Navigation) terminals, A/V (Audio/Video) systems, flexible terminals, digital signage devices, and the like.

More specifically, the terminal 100 includes an image capturing unit 110, a video object extracting unit 120, a stereoscopic XR object generating unit 130, an image mixing unit 140, an input unit 150, and an interaction unit 160. ), a haptic response unit 170 and an XR content detection unit 180.

The image capture unit 110 may be configured to capture an image of a user and a surrounding environment.

The image object extraction unit 120 may be configured to extract at least one or more image objects from a photographed image. Here, the image object may be objects of the body and the surrounding environment, such as hands, feet, and faces.

Meanwhile, the image object extractor 120 may select a desired area (size) in a captured image through the UI 125 and designate an image object to be extracted from among the selected images.

Next, the XR content generator 130 may be configured to generate or process an image object specified by a user among the extracted image objects as XR content.

For example, when an image object designated by a user is a body, XR contents may be generated or processed into XR contents using an avatar similar to the corresponding body. The avatar may be a 3D animation emphasizing personal characteristics.

In addition, if the video object designated by the user is a space, XR content for a space the user (message sender or message receiver) wants to visit or experience, or a space to deliver a gift to (e.g., a space to propose) can be created or processed into XR content.

In addition, the XR content generation unit 130 may store a large amount of 3D virtual images in advance, extract features from the body image, and match each pre-stored 3D virtual image with each of the pre-stored 3D virtual images to highly match the features of the 3D virtual images. can be used as an avatar.

In addition, when the image object designated by the user is a gift wrapping box, a 3D virtual object similar to the object may be created.

Here, the XR content generator 130 may not only generate a 3D virtual image, but also replace an image object designated by the user with one of various XR contents supported by the UI 125, or capture an image. The unit 110 may receive XR content to be included in the captured image from the UI 125 .

The rendering unit 140 may be configured to generate a stereoscopic XR image by rendering a captured image and a generated stereoscopic XR image object.

In addition, the rendering unit 140 may be configured to generate a corresponding stereoscopic XR image so as to include interaction information for an XR image object to be executed in a terminal by interworking with an interaction unit 170 to be described later.

Next, the message generating unit 150 may be a component that generates a message including the stereoscopic XR images mixed or matched by the rendering unit 140 .

Next, the input/output unit 160 transmits the message including the stereoscopic XR image generated by the message generating unit 150 to an external terminal designated by the user or receives the stereoscopic XR image generated by the external terminal. can be a composition.

Next, the interaction unit 170 may be configured to control the reaction or execution of the XR object in the stereoscopic XR image received from the input/output unit 160 under preset conditions. In addition, it may be configured to give an interaction condition value to the rendering unit 140 so that the three-dimensional XR object reacts or is executed under a preset condition.

Next, the XR content detecting unit 190 detects the XR content in the stereoscopic XR image exposed on the screen under a preset detection condition, for example, when the XR content is located within the user's body (hand), or the user's gesture or motion. There is a component that detects the presence or absence.

Next, the effect unit 180 may be configured to output an effect (vibration intensity, sound intensity, etc.) according to the interaction condition value based on the detection signal of the XR content detection unit 190 .

2 is a flowchart illustrating a method for providing a message-based XR mobile service using augmented reality according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a method for providing a message-based XR mobile service using augmented reality according to another embodiment of the present invention. Here is a flow chart explaining the method.

First, referring to FIG. 2, in the message-based XR mobile service providing method (S700) using augmented reality according to an embodiment of the present invention, after receiving (S710) a captured image taken by an image capturing unit, XR content is generated. The XR contents to be rendered in the photographed image captured in the unit 130 are received from the UI and generated, or XR contents similar to the image object designated by the user are generated (S720).

Here, the process S720 includes a process of receiving, from the interaction unit 170, an interaction condition value for reacting or executing the three-dimensional XR object under preset conditions.

Thereafter, the rendering unit 140 generates an XR image in which the XR contents in the captured image are rendered with the condition value designated by the user (S730) and provides it to the XR image message generation unit 150.

When the message generation unit 150 generates an XR video message containing XR content, the input/output unit sends or shares the XR video message to a number designated by the user or an SNS (S740).

On the other hand, referring to FIG. 3, in the message-based XR mobile service providing method (S800) using augmented reality according to another embodiment of the present invention, the input/output unit 160 receives an XR video message from an external terminal (S810) Then, the interaction unit 170 executes the XR content according to the interaction value set in the XR video message (S820). Thereafter, the XR content in the stereoscopic XR image exposed on the screen in the XR content detector 190 is detected under a preset detection condition, for example, when the XR content is located within the user's body (hand) or whether there is a user's gesture or motion. is detected, the effect unit 180 outputs an effect (vibration, sound, etc.) corresponding to the XR content.

Therefore, the present invention has the advantage of being able to solve the problem of simply selecting a gift and delivering it as a message, such as the problem of not richly expressing and delivering emotions as in the prior art.

In other words, there is an advantage in that it can satisfy the emotional delivery part in conveying the heart compared to directly seeing and delivering the gift offline.

Through the advantages described above, users can experience a space they want to visit or a space they want to deliver a gift to (for example, a space where they want to propose) through virtual content and avatars that are actually filmed. It has the advantage of providing pleasure.

In addition, you can present a different experience with realistic contents and gifts that are one level different from the feeling of receiving a mobile gift.

“~ unit” used in one embodiment of the present invention may be implemented as a hardware component, a software component, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The described hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The foregoing may be modified and modified by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: System supporting message-based XR mobile service using augmented reality
110: video recording unit
120: image object extraction unit
125 UI
130: XR content generation unit
140: rendering unit
150: XR video message generation unit
160: input / output unit
170: interaction unit
180: effect unit
190: XR content detection unit

Claims (10)

A terminal that creates and provides a message containing XR content, and executes the XR content included in the externally provided message in response to a predetermined condition; and
A system for supporting a message-based XR mobile service using augmented reality, including an XR service platform supporting an interface for producing and executing the XR content in the terminal.
According to claim 1,
The terminal
After extracting at least one object in the captured real image, processing the object designated by the user among the extracted objects into a 3D XR object, and then generating an XR image in which the 3D XR object processed in the real image is mixed. A system that supports message-based XR mobile service using
According to claim 2,
The terminal
A system for supporting a message-based XR mobile service using augmented reality, characterized in that an image object designated by a user among extracted image objects is created or processed as XR content.
According to claim 3,
The terminal
Augmented reality characterized by generating a three-dimensional virtual image, replacing an image object designated by the user with one of various XR contents supported by the UI, or receiving XR contents to be included in the captured image from the UI A system that supports message-based XR mobile service using
According to claim 4,
The terminal
A rendering unit generating an XR image obtained by rendering the captured image and the generated stereoscopic XR image object;
A system for supporting a message-based XR mobile service using augmented reality, characterized in that the rendering unit generates a stereoscopic XR image in conjunction with the interaction unit to include interaction information for the XR image object to be executed in the terminal. .
According to claim 5,
The terminal
A system for supporting a message-based XR mobile service using augmented reality, characterized in that for generating a message containing the XR image rendered by the rendering unit.
According to claim 6,
The terminal
An interaction unit for controlling an XR object in an XR video message received from the outside to react or execute under preset conditions;
The interaction part
A system for supporting a message-based XR mobile service using augmented reality, characterized in that for giving the rendering unit an interaction condition value for reacting or executing the XR object under a preset condition.
According to claim 7,
The terminal
Augmented reality characterized in that the XR content in the stereoscopic XR image exposed on the screen detects a preset sensing condition, for example, when the XR content is located in the user's body (hand) or the presence or absence of the user's gesture or motion A system that supports the message-based XR mobile service.
Receiving a photographed image captured by an image photographing unit;
generating XR contents to be rendered within the photographed image captured by the XR contents creation unit from a UI or generating XR contents similar to an image object specified by a user;
generating an XR image in which the XR content in the captured image is rendered by a user-specified condition value in a rendering unit;
generating an XR video message including an XR video in an XR video message generating unit; and
When the XR video message generating unit generates an XR video message containing XR contents, the input/output unit transmits or shares the XR video message to a number designated by the user or to an SNS based on a message using augmented reality. of XR mobile service delivery methods.
Receiving an XR video message from an external terminal in an input/output unit;
Executing XR content according to the interaction value set in the XR video message in an interaction unit;
detecting, by an XR content detector, the XR content in the stereoscopic XR image exposed on the screen under a preset detection condition, for example, when the XR content is located within the user's body (hand) or whether there is a user's gesture or motion; and
A method for providing a message-based XR mobile service using augmented reality, including outputting an effect (vibration, sound, etc.) corresponding to XR content in an effect unit.

Images