KR102483781B1 - Operating method of augmented reality glass for providing docent service - Google Patents

Abstract

According to an embodiment of the present invention, an operating method of augmented reality glasses providing a user experience-based docent service includes receiving an image of a work from a hololens camera, generating an augmented reality layer image of the work, and a user It may include determining whether the user gazes at the work for a reference time period, and distinguishing the augmented reality layer image from surrounding images when the user gazes at the work for the reference time period.

Description

Operation method of augmented reality glasses providing docent service {OPERATING METHOD OF AUGMENTED REALITY GLASS FOR PROVIDING DOCENT SERVICE}

The present invention relates to a method of operating augmented reality glasses that provide a docent service.

Augmented reality (AR) is a field of 　virtual reality (VR), which is a 　computer graphics technique that synthesizes virtual objects or information in a real environment to make them look like objects existing in the original environment. In existing museum exhibitions, docent services are limited. It is a reality in which only information (demographic characteristics and situational characteristics are not considered, visual examples or various interpretations that can satisfy the individual emotions of users) is provided. In terms of upgrading the existing docent service method, it is the first to develop an AR (augmented reality) glass-based artificial intelligence guidance system that can easily convey art information to viewers and provide information that goes beyond spatial limitations. there is.

US Patent: US 9,599,821, Registration Date: March 21, 2017, Title: Virtual reality system immersion in virtual space to consist with actual movement in actual space.

Human docents, audio guides, and e-book guide services provided as art commentary services in existing museum exhibitions have the disadvantage of providing only general and limited information. If AR (augmented reality)-based user emotion recognition (AI) docent service that can overcome these disadvantages is provided, a large number of viewers can view the desired information about art works without time and space constraints. You can have a new experience with your work. In addition, it can open up a new niche market in the AR field and enhance the competitiveness of the culture and arts industry.

According to an embodiment of the present invention, a method of operating an augmented reality glasses providing a docent service based on user experience includes: receiving an image of a work from a hololens camera; generating an augmented reality layer image for the work; determining whether the user gazes at the work for a reference period of time; and distinguishing the augmented reality layer image from surrounding images when the user gazes at the work during the reference time.

In an embodiment, the method may further include recognizing the work being viewed using a work algorithm and a spatial recognition algorithm.

In an embodiment, the dividing of the surrounding image may include blurring the surrounding image.

In an embodiment, the blurring of the surroundings may include generating a slider for adjusting the blurring of the augmented reality layer image; and adjusting the blurring value by moving the slider up/down.

In an embodiment, the blurring of the surroundings may include adjusting a blurring density of the augmented reality image according to the blurring value; combining the adjusted augmented reality image with a video taken from a hololens camera for the work; and displaying the combined image to the user.

In an embodiment, a step of increasing immersion may be further included.

In an embodiment, the step of increasing the sense of immersion may include generating a slider for adjusting the lighting function of the augmented reality layer image; and setting an illumination value by moving the slider up/down.

In an embodiment, the step of increasing the sense of immersion may include adjusting lighting of the augmented reality image according to the lighting value; combining the adjusted augmented reality image with a video taken from a hololens camera for the work; and displaying the combined image to the user.

In an embodiment, the step of selecting and storing a part of the work may be further included.

In an embodiment, the step of selecting and storing a part of the work may include determining a part of the work to perform a cropping function by determining the start and end positions of a rectangle identified by a user's hand gesture function. choosing; and generating an augmented reality layer image to perform the cropping function.

In an embodiment, the selecting and storing a part of the work may include enlarging the augmented reality layer image; and storing the enlarged image.

In an embodiment, the method may further include displaying the enlarged image to a user.

According to an embodiment of the present invention, the method of operating AR glasses that provides a user experience-based docent service increases the user's sense of immersion in the work and maximizes the satisfaction of the user's appreciation of the work by selecting and storing a part of the work. can

The accompanying drawings are provided to aid understanding of the present embodiment, and provide examples along with detailed descriptions. However, the technical features of this embodiment are not limited to specific drawings, and features disclosed in each drawing may be combined with each other to form a new embodiment.
1 is a diagram showing an AR glasses 100 based on user experience according to an embodiment of the present invention by way of example.
FIG. 2 is a diagram exemplarily showing a user prior information consent process of the AR glasses 100 according to an embodiment of the present invention.
3 is a diagram exemplarily illustrating a process of processing an occurrence of an exceptional situation in the AR glasses 100 according to an embodiment of the present invention.
4 is a diagram showing an interactive viewing process of the AR glasses 100 according to an embodiment of the present invention by way of example.
5 is a diagram exemplarily showing a method of operating the AR glasses 100 according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining step S110 shown in FIG. 5 in more detail.
FIG. 7 is a diagram for explaining step S120 shown in FIG. 5 in more detail.
FIG. 8 is a diagram for explaining step S130 shown in FIG. 5 in more detail.

In the following, the content of the present invention will be described clearly and in detail to the extent that a person skilled in the art can easily implement the present invention using drawings.

Since the present invention can have various changes and various forms, specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific form disclosed, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms.

The terms may be used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Other expressions describing the relationship between elements, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly. Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, the terms "comprise" or "have" are intended to designate that there is an embodied feature, number, step, operation, component, part, or combination thereof, but one or more other features or numerals. However, it should be understood that it does not preclude the possibility of existence or addition of steps, operations, components, parts, or combinations thereof. Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, they are not interpreted in an ideal or excessively formal meaning. .

AR glass according to an embodiment of the present invention is technically superior to existing methods because there is no developed case of technology linking wearable glass-based exhibition content service, gesture recognition exclusively for viewing exhibits, and individual emotion recognition technology for viewing exhibits. excellent with

User experience-based AR glasses according to an embodiment of the present invention can be expected to be able to produce high value-added contents such as immersive sensory education, psychological/rehabilitation therapy, theme parks, and broadcasting in addition to museum docent service in the future. If various technologies are developed for use in indoor/outdoor exhibitions, it is expected to create new added value and preoccupy the global market. When developing contents in various fields such as museums, science museums, STEAM, and tourism industries by expanding the development of AR glass-based optimized guidance services, it is expected to create new added value and preoccupy the global market.

1 is a diagram showing an AR glasses 100 based on user experience according to an embodiment of the present invention by way of example. Referring to FIG. 1 , AR glasses 1000 include at least one processor 110, at least one memory 120, at least one camera 130, at least one sensor 140, and a display device 150. can include

At least one processor 110 may be implemented to control overall operations of the AR glasses 100. In an embodiment, the processor 110 may control wired/wireless communication with the outside. For example, the processor 110 may be an application processor (AP), integrated modem application processor (ModAP), or the like. As an integrated modem application processor, the processor 110 may include a broadcast reception module, a mobile communication module, a wireless Internet module, a short-distance communication module, and a location information module. However, it should be understood that the functionality of the processor 110 of the present invention is not limited thereto. The AR glasses 100 may further include a separate processor for wired/wireless communication.

At least one memory 120 may be implemented as a volatile memory or a non-volatile memory. In an embodiment, the memory 120 may include a buffer memory for temporarily storing data necessary for the processing operation of the AR glasses 100. For example, buffer memory includes random access memory (RAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), magnetoresistive random access memory (MRAM), ferroelectric RAM (FeRAM), phase change RAM (PRAM) may include at least one of them. Also, according to an embodiment, the memory 120 may include a storage device for storing a program (or algorithm) for providing a user experience-based docent service. For example, the storage device may include at least one of an embedded multimedia card (eMMC), a solid state drive (SSD), a universal flash storage (UFS), and a nonvolatile dual line memory module (NVDIMM).

At least one camera 130 may be implemented to detect images/videos. For example, camera 130 may

At least one sensor 140 may be implemented to detect user's biometric information.

The display device 150 may be implemented to display information related to the docent service. In an embodiment, the display device 150 may include a transparent display device for displaying images on the spectacle lenses of the AR glasses 100.

The AR glasses 100 according to an embodiment of the present invention can maximize the effect of displaying artworks by providing a docent service based on user experience.

2 is a diagram illustratively showing a user prior information consent process of the AR glasses 100 according to an embodiment of the present invention. Referring to FIG. 2 , when a user wears the AR glasses 100, a user experience-based selection operation on whether or not to use the docent service may be performed first. If user-based docent service provision is selected, a setting operation for the user's art knowledge level may be performed. Then, a docent service customized according to the user's art knowledge level may be provided. In addition, an interactive play activity of the user may be performed with respect to the artwork being viewed. In an embodiment, the AR glasses 100 analyzes a user's behavior pattern based on information collected through the camera 130 or the sensor 140, and generates a GUI (graphical user interface) character according to the analyzed behavior pattern. , and the GUI character can be touched or rotated by the user.

In addition, the image of the artwork being viewed by the user may be stored in the user's storage medium or directly uploaded to the user's social networking service (facebook, instragram, kakaotalk, etc.).

On the other hand, if provision of the user experience-based docent service is not selected, a limited docent service may be provided. For example, the limited docent service may include providing information about a work being viewed in voice/text/video. For example, the description of the work, the biography of the artist, etc. can be transformed into a text box the moment you touch a GUI (graphical user interface) character while viewing the work.

3 is a diagram exemplarily illustrating a process of processing an occurrence of an exceptional situation in the AR glasses 100 according to an embodiment of the present invention. As an exception, there may be times when the user takes off the AR glasses 100. When the worn AR glasses 100 are uncomfortable and the user takes them off, guidance may be received from the situation before taking them off. Then, when the user wears the AR glasses 100 again, the data is saved, the exhibition circulation is searched again, and the current location can be guided. In addition, in an exceptional situation, when a call is received while viewing a work, a call may be made through Bluetooth.

4 is a diagram showing an interactive viewing process of the AR glasses 100 according to an embodiment of the present invention by way of example. In an embodiment, the user may ask the GUI character questions through a 'tell me' gesture. In an embodiment, the user may ask questions using a text box.

In an embodiment, a user may be provided with information related to a work being watched through a 'tell me' gesture. Such information may include, for example, the life of the artist, the artist's work, the artist's art historical position, the reason for the high self-inflictedness, and the work sold at the highest price. In an embodiment, images and data videos may be viewed together in addition to information requested by the user.

5 is a diagram exemplarily showing a method of operating the AR glasses 100 according to an embodiment of the present invention. Referring to FIGS. 1 to 5 , the AR glasses 100 may perform an operation for providing a docent service based on user experience as follows.

An operating method of augmented reality glasses providing a user experience-based docent service according to an embodiment of the present invention includes receiving an image of a work from a hololens camera, generating an augmented reality layer image of the work, and providing a user with a reference time and determining whether the user gazes at the work for a reference period of time, and distinguishing between an augmented reality layer image and a surrounding image when the user gazes at the work for a reference time. An image processing operation may be performed to distinguish between the work and the surroundings of the work. For example, if a user gazes at a work for 5 seconds or more, the surrounding area may be blurred using pseudo haptic technology (S110). This allows the user to focus on the work. On the other hand, it should be understood that the peripheral blurring process does not necessarily need to be performed by pseudo haptic technology.

The user's sense of immersion in the selected work may be increased by using a lighting function based on pseudo haptic technology or multi-modal technology (S120).

Part of the work (eye or hand) is selected according to the pseudo-haptic technology or multi-modal technology, and the image of the selected work can be saved (S130). Selected works can be viewed and saved by manually dragging and expanding (Drag + Expand).

Steps and/or actions in accordance with the present invention may occur concurrently in different embodiments, in different orders, or in parallel, or for different epochs, etc., as would be understood by one skilled in the art. can

In some embodiments, some or all of the steps and/or actions may include instructions, programs, interactive data structures, clients and/or servers stored on one or more non-transitory computer-readable media. At least some of them may be implemented or performed using one or more processors that do. The one or more non-transitory computer-readable media may illustratively be software, firmware, hardware, and/or any combination thereof. Additionally, the functions of a “module” discussed herein may be implemented in software, firmware, hardware, and/or any combination thereof.

One or more non-transitory computer-readable media and/or means for implementing/performing one or more operations/steps/modules of the embodiments of the present invention may include application-specific integrated circuits (ASICs), standard integrated circuits, controllers that perform appropriate instructions, including microcontrollers, and/or embedded controllers, field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like, but are not limited to does not

FIG. 6 is a diagram for explaining step S110 shown in FIG. 5 in more detail. Referring to FIG. 6 , when one gazes at a work, a function of blurring the surroundings may be performed. The execution method may include a blur processing operation around the work when gaze at the work in front for 5 seconds or more. As a constraint condition, the distance between the work and the viewer is within 1.5m (determination of whether or not to stare at the work) is possible. In an embodiment, an operation of determining whether a user gazes at a work may be performed by a gaze tracking algorithm. Here, the gaze tracking algorithm may determine whether or not to gaze at the work by using information related to the focus of the user's eyes. Meanwhile, it should be understood that the gaze tracking algorithm of the present invention is not limited thereto.

In an embodiment, a blur processing operation may be performed by combining an image input from a hololens camera and an augmented reality image layer. In an embodiment, effect adjustment may include a step using a slide bar. Blurring operation for the surroundings may be performed through the following process. A user wears AR glasses and stands in front of an original work (2D painting, sculpture, etc.). The work you are currently viewing is recognized by the work recognition algorithm and space recognition algorithm in the AR glasses. Gaze at the work in front of you (2D painting, sculpture, etc.) for 3 seconds. At this time, it is judged whether the gaze is being performed for 3 seconds. An augmented reality layer image to perform the blur function is created. A slider that can adjust the blur is created on the right side of the screen in the created augmented reality layer image. The viewer sets the blur value by moving the slider floating in the air on the right side of the screen up and down. The set Blur value is transmitted to the system. The blur density is adjusted in the augmented reality layer image using the blur value transmitted to the system. The augmented reality layer image with adjusted blur density and the input camera video (original work) are combined and displayed to the viewer. Viewers who want to further perform the lighting effect function gaze at the work in front of them (2D painting, sculpture, etc.) for 3 seconds. At this time, the algorithm that determines whether you are gazed for 3 seconds works. Algorithms that perform lighting functions are executed. At this time, control is transferred to a lighting function to be described later. The Blur function only applies to works you are viewing (2D paintings, sculptures, etc.). To exit the Blur function, touch the Blur slide bar twice.

FIG. 7 is a diagram for explaining step S120 shown in FIG. 5 in more detail. Referring to FIG. 7 , the user can enjoy the selected work in an immersive sense by using a light effect function. The execution method may include an operation of executing a lighting function after blur processing. The constraint condition is that the distance between the work and the viewer is within 1.5m (determination of whether or not to stare at the work). In an embodiment, blur processing is performed by combining an image input from a hololens camera and an augmented reality image layer. In the algorithm described in FIG. 6, a viewer who wants to perform a light effect function on the work in front of him (two-dimensional painting, sculpture, etc.) gazes at the work in front of him for 3 seconds. At this time, it is judged whether the gaze is being performed for 3 seconds. An augmented reality layer image to perform the light effect function is created. In the created augmented reality layer image, a slider that can adjust the light effect function is created at the bottom of the screen. The viewer sets the light value by moving the slider floating in the air below the screen up and down. The set Light value is transmitted to the system. The light effect is adjusted on the augmented reality layer image using the light value passed to the system. The augmented reality layer image with the light effect applied and the input camera video (original work) are combined and displayed to the viewer. The light effect function only applies to the work you are currently viewing (2D paintings, sculptures, etc.). To exit the light effect function, touch the light effect slide bar twice.

FIG. 8 is a diagram for explaining step S130 shown in FIG. 5 in more detail. Referring to FIG. 8 , the user can select a part of the work and enjoy the Multi-Modal Expand by hand. The execution method may include designating the crop area using both hands and eyes. The constraint condition is that the distance between the work and the viewer is within 1.3m (determination of whether or not to stare at the work). In an embodiment, a combined image may be displayed by combining an image input from a hololens camera and a crop area image.

The user wears AR glasses and stands in front of the original work (two-dimensional painting). The work you are currently viewing is recognized by the work recognition algorithm and space recognition algorithm in the AR glasses. Specify the coordinates of the start of the rectangle of the upper left rectangle and the end of the rectangle of the lower right rectangle using both hands at the location of the work you want to crop. At this time, gaze within the square area for 3 seconds. The gesture recognizer judges the shape of the gesture of the hand to find out whether or not to perform Crop, and determines whether Gaze is being held within the shape of the gesture for 3 seconds. Determine the position to be cropped from the original work image by grasping the start and end positions of the rectangle identified by the hand gesture function. An augmented reality layer image to perform the Crop function is created. The enlarged image to be shown on the augmented reality layer image is stored as a digital image (4K level). An enlarged digital image that is cropped to the created augmented reality layer image is displayed. The crop function only applies to the work you are currently viewing (2D picture). To end the Crop function, touch the Crop image.

The AR glasses 100 providing a docent service based on user experience according to an embodiment of the present invention have a gesture learning function, a work information providing function, a tell me function, a knowledge level setting function, an exception handling function, a blurring function, and a lighting function. , interactive activity function, SND upload and storage function, etc. can be included.

According to an embodiment of the present invention, an operation method of augmented reality glasses for providing a docent service based on user experience includes: blurring the surroundings of the work when it is determined that the user gazes at the work for more than a predetermined period of time; Increasing a sense of immersion in the work by using a lighting function; and selecting and storing a part of the work.

In an embodiment, the method may further include recognizing the work being viewed using a work algorithm and a spatial recognition algorithm.

In an embodiment, the blurring of the surroundings may include generating an augmented reality layer image to perform a blurring function.

In an embodiment, the blurring of the surroundings may include generating a slider for adjusting the blurring of the augmented reality layer image; and adjusting the blurring value by moving the slider up/down.

In an embodiment, the blurring of the surroundings may include adjusting a blurring density of the augmented reality image according to the blurring value; combining the adjusted augmented reality image with a video taken from a hololens camera for the work; and displaying the combined image to the user.

In an embodiment, the step of increasing the sense of immersion may include generating an augmented reality layer image to perform the lighting function; generating a slider for adjusting the lighting function of the augmented reality layer image; and setting an illumination value by moving the slider up/down.

In an embodiment, the step of increasing the sense of immersion may include adjusting lighting of the augmented reality image according to the lighting value; combining the adjusted augmented reality image with a video taken from a hololens camera for the work; and displaying the combined image to the user.

In an embodiment, the step of selecting and storing a part of the work may include selecting a part of the work to perform a cropping function by determining a starting position and an ending position of a rectangle grasped by a user's hand gesture function. doing; and generating an augmented reality layer image to perform the cropping function.

In an embodiment, the selecting and storing a part of the work may include enlarging the augmented reality layer image; and storing the enlarged image.

In an embodiment, the method may further include displaying the enlarged image to a user.

The method of operating the AR glasses 100 according to an embodiment of the present invention may provide an artificial intelligence (AI) docent service for each user. For example, it is possible to enjoy a variety of information (providing the artist's life, affiliation art trend, production background, etc. according to the viewer's preference) on top of the work of art, allowing for a new, optimized experience that actively considers the characteristics of AR glasses. Do. The augmented reality market using AR glasses is growing as a next-generation core computing platform, and the demand for dedicated UX (user experience) technology development and augmented reality content and services related to it is rapidly expanding.

The commercial use of augmented reality services using characters is increasing due to the increase in the frequency of exposure, and in particular, the need for providing subdivided characters considering the demographic characteristics of viewers in art galleries has emerged in public exhibition facilities. Moreover, there is an advantage that no attempt has been made to spread the results of the task so far. In particular, research on existing exhibition-based interactions using AR glasses is insufficient, so it is necessary to implement AR glasses-based services that can replace existing services for the first time. Do.

On the other hand, the above-described content of the present invention is only specific embodiments for carrying out the invention. The present invention will include technical ideas, which are abstract and conceptual ideas that can be utilized as technology in the future, as well as concrete and practically usable means themselves.

100: AR glass
110: processor
120: memory
130: camera
140: sensor
150: display device

Claims (12)

In the operation method of augmented reality glasses providing docent service based on user experience:
Receiving an image of a work from a hololens camera;
generating an augmented reality layer image for the work;
determining whether the user gazes at the work for a reference period of time;
distinguishing the augmented reality layer image from surrounding images when the user gazes at the work during the reference time; and
Including; increasing the sense of immersion;
The step of distinguishing the surrounding image includes the step of blurring the surrounding image,
The step of blurring the surrounding image,
generating a slider for adjusting the blurring of the augmented reality layer image; adjusting a blurring value while moving a slider for adjusting the blurring; And adjusting the blurring density of the augmented reality layer image according to the blurring value;
The step of increasing the immersion is,
generating a slider for adjusting a lighting function of the augmented reality layer image; setting a lighting value while moving a slider for adjusting the lighting function; And adjusting the lighting of the augmented reality layer image according to the lighting value;
combining the augmented reality layer image of which the blurring density and the illumination are adjusted with a video taken from a hololens camera for the work;
and displaying the combined image to the user. According to claim 1,
The method further comprising the step of recognizing the work being viewed using a work algorithm and a spatial recognition algorithm. delete delete delete delete delete delete According to claim 1,
The method further comprising selecting and saving a portion of the work. According to claim 9,
The step of selecting and saving a part of the work,
selecting a part of the work to perform a cropping function by determining the start and end positions of the rectangle identified by the user's hand gesture function; and
And generating an augmented reality layer image to perform the cropping function. According to claim 10,
The step of selecting and saving a part of the work,
enlarging the augmented reality layer image; and
and saving the enlarged image. According to claim 11,
The method further comprising the step of displaying the enlarged image to a user.

Images