KR102434457B1 - Apparatus and method for prividing virtual experience service based on extended reality - Google Patents

Abstract

The present invention relates to an apparatus and method for providing an extended reality-based virtual experience service, and more particularly, to a communication unit configured to communicate with an external device or server; a camera unit for photographing the surrounding environment and the user; a sensor unit for detecting a user's movement or a change in the surrounding environment through at least one sensor; a display unit for displaying an image captured by the camera unit or at least one piece of information; an extraction unit for extracting and providing object information in real time from the image captured by the camera unit and the user's movement or changes in the surrounding environment sensed through the at least one sensor; a memory unit for storing at least one of images, contents, applications, programs, and data captured by the camera unit; After the user executes the experiential service through the web or application, a control unit for controlling to check an object selected by the user among at least one or more 3D modeled objects displayed and provide an experience service corresponding to the shape of the selected object; may include

Description

Apparatus and method for providing an extended reality-based virtual experience service

The present invention relates to an extended reality-based virtual experience service providing apparatus and method, and more particularly, to an extended reality-based virtual experience service that enables a user to virtually experience a product based on extended reality through a smart device. It relates to an apparatus and method for providing.

VR (Virtual Reality) technology provides only CG (Computer Graphic) images of objects or backgrounds in the real world, and AR (Augmented Reality) technology provides CG images created virtual on top of real objects, and MR (Mixed Reality) ) technology is a computer graphics technology that provides the real world by mixing and combining virtual objects. The aforementioned VR, AR, MR, and the like are all simply referred to as XR (extended reality) technology.

Virtual reality simulation technologies such as VR, AR, and MR are being applied to various industrial fields, and the cultural art industry is also paying attention to virtual reality technology.

In the global culture and arts industry, the demand for experiential culture and art products is increasing according to the recent changing consumption patterns, and the change in consumption value includes the value of the product (communication, expression, experience) from the product itself. is a trend that is changing to

Moreover, cultural art products are a field with strong three-dimensional properties compared to other fields, and there is a need to expand the range of choices of consumers by providing more realistic information and enabling interaction by converging virtual reality technology. .

Recently, as the demand for non-face-to-face un-tact consumption that minimizes contact with people is increasing, the Technology needs to be developed.

Therefore, the present invention has been proposed to solve the above problems, and the user can experience fittings based on extended reality using a smart device without the need to visit a store to experience or purchase a product. An object of the present invention is to provide an apparatus and method for providing an extended reality-based virtual experience service.

Objects of the present invention are not limited to those mentioned above, and other objects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

According to an aspect of the present invention for achieving the above object, an apparatus for providing an extended reality-based virtual experience service includes: a communication unit configured to communicate with an external device or server; a camera unit for photographing the surrounding environment and the user; a sensor unit for detecting a user's movement or a change in the surrounding environment through at least one sensor; a display unit for displaying an image captured by the camera unit or at least one piece of information; an extraction unit for extracting and providing object information in real time from the image captured by the camera unit and the user's movement or changes in the surrounding environment sensed through the at least one sensor; a memory unit for storing at least one of images, contents, applications, programs, and data captured by the camera unit; After the user executes the experiential service through the web or application, a control unit for controlling to check an object selected by the user among at least one or more 3D modeled objects displayed and provide an experience service corresponding to the shape of the selected object; may include

Meanwhile, the method for providing an extended reality-based virtual experience service according to the present invention includes: displaying at least one 3D modeled object through a display unit when a user executes the experience service through a web or an application; checking, by the controller, an object selected by a user from among at least one or more 3D modeled objects; and receiving, by the control unit, an experience service in the form of an experience corresponding to the selected object based on the image obtained from the camera unit and data obtained from the sensor unit.

According to the present invention, a user can experience fittings based on extended reality using a smart device without having to visit a store to experience or purchase a product.

Effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

1A and 1B are diagrams illustrating an example of receiving an experience service for a product through an apparatus for providing an experience service according to an embodiment of the present invention.
2 is a diagram illustrating an example of receiving an experience service for a product through an apparatus for providing an experience service according to another embodiment of the present invention.
3 is a diagram illustrating the configuration of an apparatus for providing an experience service according to an embodiment of the present invention.
4 is a flowchart illustrating a method for providing an experience service in an apparatus for providing an experience service according to an embodiment of the present invention.
5A and 5B are diagrams for explaining a series of processes for tracking a real-time object in an apparatus for providing an experience service according to an embodiment of the present invention.

In order to fully understand the configuration and effect of the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various forms and various modifications may be made. However, the description of the present embodiments is provided so that the disclosure of the present invention is complete, and to fully inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention. In the accompanying drawings, the components are enlarged in size from the actual ones for convenience of description, and the ratio of each component may be exaggerated or reduced.

When an element is described as being “on” or “adjacent to” another element, it should be understood that another element may be directly on or connected to the other element, but another element may exist in between. something to do. On the other hand, when it is described that a certain element is "on" or "directly" of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between elements, for example, “between” and “directly between”, etc. may be interpreted similarly.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms may be used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

The singular expression includes the plural expression unless the context clearly dictates otherwise. Terms such as "comprises" or "having" are intended to designate that a feature, number, step, action, component, part, or combination thereof described in the specification is present, and includes one or more other features or numbers, It may be construed that steps, operations, components, parts, or combinations thereof may be added.

Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those of ordinary skill in the art.

The present invention is to enable a user to experience a product in a virtual space through 3D modeling and digital archive of cultural and artistic products such as accessories, without the need for the user to visit a store to experience or purchase the product This is to provide an experience service like actual fitting without having to try on various products directly through fitting technology based on extended reality using smart devices.

Here, the smart device may be an extended reality (extended reality, hereinafter referred to as 'XR') device for providing an experience service. For example, the XR device may be any one of eye-glass, EMD (Eye Mounted Display), HMD (Head Mounted Display), HWD (Head Worn Display), and HUD (Head Up Display), It may be any one of a smart book, a mobile Internet device (MID), a netbook, a smart phone, a tablet PC, and a smart TV, as well as a general type of PC such as a laptop, a desktop, and the like.

XR integrates all of Virtual Reality, which is implemented exactly like the real world, Augmented Reality, which combines virtual objects on top of the real world, and Mixed Reality, which combines the real world with the virtual world. It is another third real world that was born.

That is, the present invention relates to an apparatus and method for providing a virtual experience service based on such extended reality.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1A and 1B are diagrams illustrating an example of receiving an experience service for a product through an apparatus for providing an experience service according to an embodiment of the present invention.

First, a user may receive an experience service by executing a web or an application using the experience service providing apparatus 100 .

Referring to FIGS. 1A and 1B , at least one product displayed on the screen of the experience service providing apparatus 100 may be checked, and a product to be experienced may be selected and the product may be experienced.

At this time, each product is a 3D modeled object, and the user can move, rotate, enlarge or reduce the object according to the 3D coordinate axis, and look at the detailed configuration.

Referring to FIG. 1A , the user can experience the product three-dimensionally by rotating the 3D modeled object through a 360° view.

In addition, referring to FIG. 1B , the user can experience the product three-dimensionally while placing the 3D modeled object at a desired place through simulation.

2 is a diagram illustrating an example of receiving an experience service for a product through an apparatus for providing an experience service according to another embodiment of the present invention.

Referring to FIG. 2 , the user can take a picture of his or her hand with the experience service providing device 100 , and do not have the actual product, but can experience the product three-dimensionally by wearing the product to be experienced on the photographed hand. have.

3 is a diagram illustrating a configuration of an apparatus for providing an experience service according to an embodiment of the present invention.

Referring to FIG. 3 , the service providing apparatus 100 according to an embodiment of the present invention includes a communication unit 101 , a camera unit 103 , a sensor unit 105 , a display unit 107 , an extraction unit 109 , and a memory. It includes a unit 111 and a control unit 113 . Of course, deleting, changing, or adding some modules according to the needs of those skilled in the art also falls within the scope of the present invention.

The communication unit 101 performs wired/wireless communication with an external device or server, for example, Wi-Fi, Bluetooth, etc. may be used for short-distance wireless communication, and for example, 3GPP communication standard may be used for long-distance wireless communication. have.

LTE refers to technology after 3GPP TS 36.xxx Release 8. In detail, LTE technology after 3GPP TS 36.xxx Release 10 is referred to as LTE-A, and LTE technology after 3GPP TS 36.xxx Release 13 is referred to as LTE-A pro. 3GPP 5G (5th generation) technology refers to technology after TS 36.xxx Release 15 and technology after TS 38.XXX Release 15, among which technology after TS 38.xxx Release 15 is referred to as 3GPP NR, and TS 36 The technology after .xxx Release 15 may be referred to as enhanced LTE. "xxx" stands for standard document detail number. LTE/NR may be collectively referred to as a 3GPP system.

The camera unit 103 captures the surrounding environment of the experience service providing apparatus 100 or captures a user and converts the image into an electrical signal. To this end, the camera unit 103 may include at least one camera to photograph the user as well as the surrounding environment of the experience service providing apparatus 100 .

The image captured by the camera unit 103 and converted into an electrical signal may be stored in the memory unit 111 and then displayed on the display unit 107 through the control unit 113 . In addition, the image captured by the camera unit 103 and converted into an electrical signal is not stored in the memory unit 111 , but may be directly displayed on the display unit 107 through the control unit 113 .

In this case, the camera unit 103 may have an angle of view. For example, the angle of view means an area capable of detecting a real object located around the camera unit 103 . The camera unit 103 may detect only real objects located within the angle of view. When the real object is located within the angle of view of the camera unit 103 , the experience service providing apparatus 100 may display a virtual reality object corresponding to the real object. Also, the camera unit 103 may detect an angle between the camera unit 103 and the real object.

The sensor unit 105 may include at least one or more sensors, for example, a gravity sensor, a geomagnetic sensor, a motion sensor, a gyro sensor, an acceleration sensor, an inclination sensor, a brightness sensor, an altitude sensor, Sensing means such as an olfactory sensor, a temperature sensor, a depth sensor, a pressure sensor, a bending sensor, an audio sensor, a video sensor, a global positioning system (GPS) sensor, and a touch sensor may be included.

The sensor unit 105 detects a user's movement and/or a change in the surrounding environment using at least one sensor.

Although the display 107 may be of a fixed type, it is implemented with a liquid crystal display (LCD), an organic light emitting diode (OLED), an electro uminescent display (ELD), and a micro LED (M-LED) to have high flexibility. It is possible. At this time, the sensor unit 105 is designed to detect the degree of bending and bending of the display unit 107 implemented with the above-described LCD, OLED, ELD, M-LED (micro LED), and the like.

The display unit 107 displays an image captured by the camera unit 103 and converted into an electrical signal, and at least one or more data, contents, programs, applications, etc. stored in the memory unit 111 .

The extraction unit 109 extracts and provides object information in real time from the image captured by the camera unit 103 and the user's movement or changes in the surrounding environment sensed through the at least one or more sensors. A process of extracting object information in real time by the extraction unit 109 will be described in detail below with reference to FIGS. 5A and 5B .

The memory unit 111 not only has a function of storing the image captured by the camera unit 103, but also has a function of storing all or part of the result value obtained by performing wired/wireless communication with an external device or server. Have. In particular, considering the trend of increasing communication data traffic (eg, in a 5G communication environment), efficient memory management is required.

The memory unit 111 stores data, content, programs, applications, etc. received from an external device or server.

The controller 111 checks an object selected by the user from among at least one or more 3D modeled objects displayed after the user executes the experience service through a web or an application, and provides an experience service corresponding to the shape of the selected object. control to provide.

For example, when looking at a 3D modeled object as a 360° view as in FIG. 1A , the controller 113 controls the 3D modeled object according to the user's manipulation based on the 360° view player. Controls movement, rotation, enlargement, or reduction to be executed.

On the other hand, as shown in FIG. 1B , when it is desired to examine the simulation of applying the 3D modeled object to the user's surrounding environment, the control unit 113 controls the 3D modeled object to be placed in a desired place through simulation according to the user's manipulation. do.

In addition, as shown in FIG. 2 , when a user wants to see a 3D modeled object through an experience of wearing it virtually, the control unit 113 takes a picture of a part that the user wants to wear through the camera unit 130 . It tracks and detects the part as an object in the image, and controls it so that it can be experienced three-dimensionally by wearing a 3D modeled object on the detected part.

4 is a flowchart illustrating a method for providing an experience service in an apparatus for providing an experience service according to an embodiment of the present invention.

Referring to FIG. 4 , when a user executes an experiential service through a web or an application (S201), at least one 3D modeled object is displayed on the display unit 107 (S203).

Thereafter, the controller 113 identifies an object selected by the user among at least one or more 3D modeled objects ( S205 ), and based on the image obtained from the camera unit 103 and data obtained from the sensor unit 105 . An experience is executed in an experience form corresponding to the selected object (S207).

First, as shown in FIG. 1A , when a 3D modeled object is viewed in a 360° view, the controller 113 controls the 360° view player to be executed ( S209 ).

On the other hand, as shown in FIG. 1B , when it is desired to examine the simulation of applying the 3D modeled object to the user's surrounding environment, the control unit 113 controls the 3D modeled object to be placed in a desired place through the simulation according to the user's manipulation. do (S211).

In addition, as shown in FIG. 2 , when a user wants to see a 3D modeled object through an experience of wearing it virtually, the control unit 113 takes a picture of a part that the user wants to wear through the camera unit 130 . The part is tracked and detected as an object in the obtained image, and the 3D modeled object is worn on the detected part and controlled so that it can be experienced three-dimensionally (S213).

After the steps S209 to S213, the experience service is performed by using a function of moving, rotating, enlarging or reducing the 3D modeled object according to a user's manipulation (S215).

5A and 5B are diagrams for explaining a series of processes for tracking a real-time object in the apparatus for providing an experience service according to an embodiment of the present invention. Step S213 will be described in detail with reference to FIGS. 5A and 5B. do.

For example, in order to extract real-time object information from a hand image, a user first acquires a hand image by photographing his or her hand using the camera unit 103 . Thereafter, the obtained hand image is converted from the object RGB color gamut to the HSV color gamut, and an H value that is a hue component that does not include a luminance component and an S value that is a chroma component are extracted.

After applying a double threshold to each of the H and S components, the hand object is separated from the background image by finding the intersection. Such an algorithm can be expressed as follows.

Here, Hmin(Smin) and Hmax(Smax) represent the minimum and maximum values of the threshold value of the hue (saturation) component, respectively, and Ixy represents the brightness value of the corresponding coordinate.

Meanwhile, for example, in the case of tracking a real-time object region, an object is tracked by applying a Continuously Adaptive Mean Shift (CAMShift) algorithm.

The back-projection image and the double-threshold processing object (hand) area are tracked, and the object area is detected through image processing within the search area. Thereafter, an object area to which a distance transform algorithm is applied to generate a minimum object area is detected. Such an algorithm can be expressed as follows.

As described above, the object registration position can be estimated through the minimum distance expression between pixels in the binary hand image.

As described above, time and economic efficiency can be improved by enabling customers (users) to virtually experience and purchase various products through smart devices in their own space without having to visit a store.

Meanwhile, in the above description, the object is referred to as a product for convenience of explanation, but various objects such as works of art and animals may be applied. That is, it is not only for the purpose of purchasing the product, but also for the purpose of observation, which may be changed according to circumstances.

In the present specification and drawings, preferred embodiments of the present invention have been disclosed, and although specific terms are used, these are only used in a general sense to easily explain the technical content of the present invention and help the understanding of the present invention. It is not intended to limit the scope. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

100: device for providing experience service 101: communication department
103: camera unit 105: sensor unit
107: display unit 109: extraction unit
111: memory unit 113: control unit

Claims (9)

a communication unit for communicating with an external device or server;
a camera unit for photographing the surrounding environment and the user;
a sensor unit for detecting a user's movement or a change in the surrounding environment through at least one sensor;
a display unit for displaying an image captured by the camera unit or at least one piece of information;
an extraction unit for extracting and providing object information in real time from the image captured by the camera unit and the user's movement or changes in the surrounding environment sensed through the at least one sensor;
a memory unit for storing at least one of images captured by the camera unit, contents, applications, programs, and data; and
After the user executes the experience service through the web or application, the control unit controls to check an object selected by the user from among at least one or more 3D modeled objects displayed and provide an experience service corresponding to the shape of the selected object; including,
The control unit, if the user wants to look at the 3D modeled object through the virtual wearing experience, when the user shoots a part desired to be worn through the camera unit, tracking and detecting the part as an object in the captured image and controlling the three-dimensional experience while wearing the selected 3D modeled object on the detected part.
According to claim 1,
The sensor unit,
Gravity sensor, geomagnetic sensor, motion sensor, gyro sensor, acceleration sensor, inclination sensor, brightness sensor, altitude sensor, olfactory sensor, temperature sensor, depth sensor, pressure sensor, bending sensor, audio sensor, video sensor , a global positioning system (GPS) sensor, an extended reality-based virtual experience service providing apparatus comprising at least one of a touch sensor.
According to claim 1,
The control unit is
If you want to look at the 3D modeled object in a 360° view, control to move, rotate, enlarge or reduce the selected 3D modeled object according to the user's manipulation based on the 360° view player. An extended reality-based virtual experience service providing device.
According to claim 1,
The control unit is
When it is desired to examine the 3D modeled object as a simulation of applying the 3D modeled object to the user's surrounding environment, an extended reality based virtual experience characterized in that the selected 3D modeled object is controlled to be placed in a desired place through the simulation according to the user's manipulation Service providing device.
delete displaying at least one 3D modeled object through a display unit when a user executes an experiential service through a web or an application;
checking, by the controller, an object selected by a user from among at least one or more 3D modeled objects; and
and receiving, by the control unit, an experience service in the form of an experience corresponding to the selected object based on the image obtained from the camera unit and data obtained from the sensor unit,
In the step of receiving the experience service, when a user wants to see a 3D modeled object through an experience of wearing it virtually, when the user takes a picture of a part that the user wants to wear through the camera unit, the part is selected from the photographed image. An extended reality-based virtual experience service providing method, characterized in that it is the step of receiving an experience service by tracking and detecting as an object, and enabling a three-dimensional experience while wearing a 3D modeled object on the detected part.
7. The method of claim 6,
The step of receiving the experience service is:
If you want to look at a 3D modeled object in a 360° view, you can use the experience service by moving, rotating, enlarging or reducing the selected 3D modeled object according to the user's operation based on the 360° view player. A method of providing an extended reality-based virtual experience service, characterized in that it is provided.
7. The method of claim 6,
The step of receiving the experience service is:
If you want to look at a simulation that applies a 3D modeled object to the user's surrounding environment, the 3D modeled object is placed in a desired place through simulation according to the user's manipulation, so that the experience service is provided. How to provide a virtual experience service.
delete

Images