KR20230055779A - Method for providing automatic contents customzing service for virtual reality exposure therapy - Google Patents

Abstract

Provided is a method for providing an automated VR exposure treatment service, transmitting a diagnostic evaluation questionnaire to a user terminal, determining an exposure level based on user information and response data received from the user terminal, and VR content of the determined exposure level. Transmitting to a user terminal to be displayed, monitoring an event occurring in the user terminal, and when the monitored event coincides with a preset anxiety-increasing event, a pop-up UI (User Interface) that checks the user's status with the user terminal ) and controlling a 3D object previously mapped and stored to the reaction after confirming the reaction from the user terminal through a pop-up UI.

Description

VR exposure therapy automation service provision method {METHOD FOR PROVIDING AUTOMATIC CONTENTS CUSTOMZING SERVICE FOR VIRTUAL REALITY EXPOSURE THERAPY}

The present invention relates to a method for providing an automated VR exposure treatment service, and provides a platform capable of automatically controlling a 3D object by confirming a user's level of anxiety through a UI pop-up when VR content for VR exposure treatment is provided.

Virtual reality technology is one of the most noteworthy technologies that will have a great ripple effect on our lives in the future. Currently, virtual reality technology, which is used in various fields, provides a new paradigm, especially in the medical field, and phobia treatment using it is being implemented at home and abroad, and its effectiveness in terms of treatment effect is recognized and its position is gradually solidified. In particular, psychiatric treatment using virtual reality technology is attempted to treat phobias, and it has the longest history among treatments using virtual reality technology. In the virtual reality environment and situation, elements that stimulate the user's five senses are combined to provide a spatial and temporal experience similar to reality, allowing them to freely move in and out of the boundary between reality and imagination. It is adopting a method to help treatment by applying virtual reality technology to the general treatment method for patients suffering from mental disorders.

At this time, a method of providing finger rehabilitation treatment in a VR environment based on a sensorless system or performing training to respond to an accident or disaster in a VR environment has been researched and developed. Announced on April 03, 2020) and Korean Patent Publication No. 2020-0095174 (published on August 10, 2020), a finger rehabilitation treatment image is output to induce exercise of a rehabilitation subject, and a finger sensor or A configuration that can track rehabilitation exercises without sensing gloves, plant information is stored for operator training in emergency situations in the plant, safety routes that can be escaped in emergency situations are calculated, and safety routes are determined according to the operator's personal preference. Provides options for the operator to choose, and induces the operator to select an exit according to the route suitable for the operator by reflecting the individual operator's physical condition, motor nerve, and level of fear or fear after the exit calculation is completed, and presents a customized evacuation route Thus, configurations capable of increasing survivability are disclosed.

However, since the former only provides VR for the purpose of physical treatment, not VR for the purpose of mental treatment, the configuration for understanding the psychological state or instability of the current treatment subject is not disclosed. Even in the latter case, the optimal route for emergency evacuation is provided based on individual preference or physical condition, but the user's real-time anxiety or psychological state cannot be grasped. A number of sensor-based VR exposure treatments are being researched and developed to measure the psychological state, but cost problems arise because expensive sensors are essential. Human intervention is essential because you have to manually change or change the Therefore, research and development of a platform that can provide customized 3D objects by automatically reflecting the user's psychological state without applying VR exposure therapy-based sensors is required.

An embodiment of the present invention, when performing VR exposure treatment, detects the level of anxiety of the user as at least one event monitored from the user terminal without human intervention or mounting of a sensor, and then confirms the user's reaction through a UI pop-up By controlling at least one 3D object placed in the VR content, the exposure intensity and reality reflection are lowered when anxiety is increased, and the exposure intensity and reality reflection are increased when anxiety is decreased, so that medical staff and counselors can communicate with each other. It is possible to provide a VR exposure treatment automation service providing system capable of controlling and customizing a 3D object after automatically monitoring and confirming a user's reaction without human intervention. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention transmits a diagnostic evaluation questionnaire to a user terminal, and determines an exposure level based on user information and response data received from the user terminal. Step of transmitting the VR content at the determined exposure stage to the user terminal so that it is displayed, monitoring an event occurring in the user terminal, if the monitored event coincides with a preset anxiety-increasing event, the user terminal to the user terminal A step of providing a pop-up user interface (UI) for checking a state, and a step of controlling a stored 3D object mapped to the reaction after confirming a reaction from a user terminal through the pop-up UI.

According to any one of the above-described problem solving means of the present invention, when the VR exposure treatment is performed, the level of anxiety of the user is detected as at least one event monitored from the user terminal without human intervention or mounting of a sensor, and then the user's By controlling at least one 3D object placed in the VR content after confirming the reaction as a UI pop-up, if the anxiety increases, the exposure intensity and reality reflection are lowered, and if the anxiety is reduced, the exposure intensity and reality reflection are lowered. By raising the object, it is possible to control and customize the 3D object after automatically monitoring and confirming the user's reaction without human intervention such as a medical staff or counselor.

1 is a diagram for explaining a VR exposure treatment automation service providing system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating an automated service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which an automated VR exposure treatment service according to an embodiment of the present invention is implemented.
5 is an operational flowchart illustrating a method for providing an automated VR exposure treatment service according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a VR exposure treatment automation service providing system according to an embodiment of the present invention. Referring to FIG. 1 , the VR exposure therapy automated service providing system 1 may include at least one user terminal 100, an automated service providing server 300, and at least one HMD 400. However, since the VR exposure treatment automation service providing system 1 of FIG. 1 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to an automation service providing server 300 through a network 200 . In addition, the automation service providing server 300 may be connected to at least one user terminal 100 and at least one HMD 400 through the network 200 . In addition, at least one HMD 400 may be connected to the user terminal 100 and the automation service providing server 300 through the network 200 .

Here, the network refers to a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include a local area network (LAN) and a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communications networks, telephone networks, and wired and wireless television communications networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi , Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( A Near-Field Communication (Near-Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

At least one user terminal 100 may be a terminal such as a client or a patient who receives VR exposure therapy using a web page, app page, program, or application related to VR exposure therapy automation service. The user terminal 100 may be a terminal that receives a diagnostic evaluation questionnaire from the automation service providing server 300 and transmits user information when transmitting a response thereto. Also, the user terminal 100 may be a terminal that is connected to the HMD 400 and outputs VR contents received from the automation service providing server 300 . In addition, the user terminal 100 receives the user's response and transmits it to the automation service providing server 300, and the 3D object controlled in real time by the automation service providing server 300 is deleted, added or modified VR content. It may be a terminal that receives and outputs streaming in real time.

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

The automation service providing server 300 may be a server that provides a VR exposure treatment automation service web page, app page, program or application. In addition, the automated service providing server 300 may be a server that stores diagnostic evaluation questionnaires and pre-maps and stores exposure steps based on response data and user information. In addition, the automation service providing server 300 may be a server that maps and stores exposure steps and VR content, and stores event data and types of events to be sensed from the user terminal 100 . In addition, the automation service providing server 300 maps a pop-up UI to provide a pop-up UI (User Interface) for checking the user's condition to the user terminal 100 when the monitored event coincides with a preset anxiety-increasing event. It may be a server that stores In addition, after checking the reaction through the pop-up UI, the automation service providing server 300 controls the stored 3D object mapped to the reaction, and provides VR content including the changed, deleted, and added 3D object to the user. It may be a server that streams to the terminal 100 in real time.

Here, the automation service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser.

At least one HMD 400 outputs VR content from the automation service providing server 300 via the user terminal 100 with or without using a web page, app page, program or application related to VR exposure treatment automation service. It may be a device that

Here, at least one HMD 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one HMD 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one HMD 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

2 is a block diagram illustrating an automated service providing server included in the system of FIG. 1, and FIGS. 3 and 4 describe an embodiment in which an automated VR exposure treatment service is implemented according to an embodiment of the present invention. It is a drawing for

Referring to FIG. 2 , the automated service providing server 300 may include a determining unit 310 , a display unit 320 , a monitoring unit 330 , a pop-up unit 340 and a control unit 350 .

An automated service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 and at least one HMD 400 is a VR exposure treatment automation service application. , When transmitting a program, app page, web page, etc., at least one user terminal 100 and at least one HMD 400 install a VR exposure treatment automation service application, program, app page, web page, etc. can be opened In addition, a service program may be driven in at least one user terminal 100 and at least one HMD 400 by using a script executed in a web browser. Here, the web browser is a program that allows users to use the web (WWW: World Wide Web) service, and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language). For example, Netscape , Explorer, Chrome, and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smart phone).

Referring to FIG. 2 , the determination unit 310 may transmit a diagnostic evaluation questionnaire to the user terminal 100 and determine an exposure level based on user information and response data received from the user terminal 100. there is. At this time, in the automated service providing server 300, the [diagnostic evaluation questionnaire-response data-exposure step] is pre-mapped and stored, the exposure step and VR content are pre-mapped and stored, and finally [diagnostic evaluation questionnaire-response data- exposure stage-VR content].

The display unit 320 may transmit the VR content of the determined exposure stage to the user terminal 100 to be displayed. At this time, the exposure step is not fixed because it is an exposure step for the initial setting that is set for the first one-time diagnosis, and can be continuously increased or decreased according to the user's response to be described later. The user terminal 100 may be connected to a Head Mounted Display (HMD) 400 that outputs VR content.

The monitoring unit 330 may monitor events occurring in the user terminal 100 . In this case, the event occurring in the user terminal 100 may mean all types of data input to the user terminal 100 or detected by the user terminal 100 . When monitoring an event occurring in the user terminal 100, the monitoring unit 330 analyzes the position and pupil size of the user's pupil, the user's voice and breathing sound, and a photograph of at least one body part. can do. The user terminal 100 is connected to an HMD 400 (Head Mounted Display), and the HMD 400 includes an eye tracking module that tracks the position of the pupil and monitors the size of the pupil, and a microphone that receives the user's voice and breathing sound. may further include. The user terminal 100 includes a camera that photographs the user, and the camera may photograph at least one body part. In this case, even if a separate heart rate sensor or the like is not used, the user's excitement state or anxiety state can be confirmed based on an image or sound.

When a person is excited or anxious, they may tremble their legs, change the size of their pupils, bite their nails, or show restlessness. Bodily changes occur, such as not being able to do it properly. After analyzing and checking based on sound and image, the user is confirmed once again through the pop-up UI, so that each individual's reaction situation and anxiety level can be mapped and stored, and personalized 3D objects can be provided accordingly. .

<pupil>

The pupil is a circular empty space located in the center of the iris, and functions to optimize the amount of light reaching the retina. When the pupillary sphincter is stimulated by the parasympathetic nerve, the pupil constricts, and when the dilated muscle is stimulated by the sympathetic nerve, the pupil dilates. Pupil size refers to the transverse diameter of the pupil and expands as mental effort or cognitive load increases, and is also changed by illumination, age, fatigue, and drug intake. Pupil size can be measured non-intrusively and continuously, and responds immediately to psychological states, so that changes in states such as anxiety or excitability can be sensitively captured.

Here are some things to consider to accurately measure pupil size: An eye tracker can be used that precisely calculates the sphericity of the pupil and captures the instantaneous eye movement and pupil size. At this time, when calculating the pupil size, the baseline for each user should be considered. The baseline response is the physiological and psychological data collected before the user is exposed to the stimulus, which is a basic evaluation of the physical and mental state of the individual research subject. It serves as a standard for physiological and psychological response analysis before playing VR content and can control individual differences in pupils. Since the pupil reacts sensitively to contrast and brightness, the effects of these physical stimuli should also be minimized. Accordingly, it is assumed that anxiety increased if pupil size increased more than the individual's baseline response. Here, a value obtained by subtracting the user's basal response from raw measures of the user's pupil size measured by the eye tracker in an environment in which illumination is controlled is regarded as the pupil size, and the level of anxiety may be determined according to the pupil size.

<breathing>

Respiration is divided into two types: involuntary respiration and voluntary respiration. Involuntary breathing is associated with activity in the respiratory center of the brainstem or medulla oblongata. Veterinary breathing is related to cerebral cortical activity in connection with language activity, so this activity is possible only in conscious breathing, but in normal conditions, these two types can be seen as a mixture. In a state of physiological relaxation or tension, breathing affects the depth and number of breaths and the relationship between inhalation and exhalation. Even in the case of hyperventilation syndrome, in which breathing becomes rapid under stress, excitement or anxiety can be measured by measuring the cycle, pattern, volume, and depth of inhalation and exhalation even if there is no equipment to measure blood pressure or oxygen saturation. there is.

In order to understand the breathing pattern, the user's basal response is checked like the pupil, raw measures are prepared, and the measurement starts by considering the cycle, pattern, size, etc. of breathing. At this time, unlike the pupil, respiration takes the form of periodic waves with exhalation if there is inhalation, so the collected sound data can be converted into a spectrogram image using Short-Time Fourier Transform (STFT). At this time, a problem arises in which an error occurs in the frequency analysis technique for an abnormal signal of Fourier Transform, and a method for performing frequency analysis even at the moment when such an error occurs is required.

Accordingly, STFT is a method of dividing a target signal into frame units and performing Fourier transform while moving a window of a certain size in consideration of signal stability. After applying a window function to the signal to be analyzed, the Fourier transform is performed. The formula for STFT in the time domain is STFT(t,w)=∫x(τ-t)e- ^jwτ dτ, and the integral range is from -infinity to +infinity. Here, x(τ) is the signal to be analyzed, and w(τ-t) is the window function. In addition, a spectrogram is a tool for visualizing and grasping sound or waves, and the characteristics of a waveform and a spectrum are combined. In the waveform, you can see the change in the time axis, that is, the change in the amplitude axis in the time domain. In the spectrum, you can see the change in the amplitude axis according to the change in the frequency axis. On the other hand, in the spectrogram, you can see the difference in amplitude according to the change in the time axis and the frequency axis. It is expressed as a difference in density or displayed color.

Feature data may be extracted from the converted spectrogram image using a Convolutional Neural Network (CNN). Since the spectrogram is output as an image, as a result, when comparing sound data with pre-stored sound data through image analysis, it is necessary to compare using an image. At this time, when trying to calculate the degree of similarity between different images, it is difficult to determine the degree of similarity between images through comparison in pixel units. There is a method of extracting and comparing images, which is deep learning technology, CNN. CNN learns the features of images by itself and can classify images and recognize patterns of images based on the learned features, and is suitable for image processing with a network structure using convolution layers, Since the image can be classified based on the characteristics of the image, it is used in one embodiment of the present invention. However, it is not limited to CNN, and any method that can classify images and extract their features will be possible.

The CNN can extract the features of the spectrogram in the form of a vector and measure the similarity between images using the extracted features. In order to extract the features of the spectrogram, in one embodiment of the present invention, a model using the basic structure of a convolutional neural network consisting of a convolution layer, an activation function layer, and a max pooling layer is used. Before measuring similarity, a feature vector of a spectrogram can be extracted and used from a layer before a Softmax layer used to classify an image. Basically, it can be used to measure similarity by extracting features from a CNN with a Fully-Connected Layer and a model with a GAP layer (Global Average Layer) that brings an average value on a feature map. .

As another model for measuring the similarity of images, a CNN-based autoencoder model can be used. The encoder is composed of a convolutional neural network structure, and a decoder for reconstructing compressed data as a result of the encoder ( decoder). The spectrogram features are extracted from a specific layer of the learned autoencoder model, and the feature vectors obtained through the GAP layer can be used for image similarity measurement. The Euclidean Distance and Cosine Similarity can be measured from the image feature vectors extracted through the three models above, and can be sorted according to the degree of similarity by spectrogram, that is, the similarity. , it is possible to check not only the most similar data but also the most dissimilar data for each spectrogram using the sorted order. The feature data extracted from the sound data in this way is compared with the feature data of the raw sound value, which is the base response.

<Body parts>

It is difficult to catch changes in facial muscles or the appearance of lips closed using jaw muscles when the user terminal 100 is a general smartphone, but cannot keep their hands still, tremble their legs, or bring their nails near their lips to bite. Anxiety symptoms or reactions, such as biting off or touching the hair frequently, can be easily measured with a standard camera. Although the user terminal 100 is connected to the HMD 400, since the camera can still be driven through multi-functions, the camera is positioned facing the user wearing the HMD 400 and then the image received from the camera is displayed. The video can be analyzed in real time by being streamed in real time, and real time can be maintained by using, for example, You Only Look Once (YOLO). One thing to consider here is that since the user terminal 100 is currently receiving VR content, it may be already maximally using available computing and networking resources. Accordingly, the workload processing the data photographed by the user may select and distribute nodes within the distributed cloud environment, which will be described later.

The pop-up unit 340 may provide a pop-up UI (User Interface) for checking the user's status to the user terminal 100 when the monitored event coincides with a preset anxiety-increasing event. At this time, the preset anxiety increase event may include any one or a combination of at least one of a movement stop event for a preset time or longer, an abnormal use speed of VR content (abnormal) event, and a pause event generated from the user terminal 100. can In this case, the movement stop event may be, for example, when the user needs to go forward or enter a building, but does not do so and stops for about N seconds, or when the user needs to proceed with the next action or process, but does not show any reaction. The usage speed or higher may mean, for example, a speed set by default between A and B, but a speed outside this range, for example, too high or too low. The pause event may mean that the user does not show any action or reaction, or may mean that the user suddenly presses the Pause button or the Panic button. The pop-up UI may be output on the VR content for each branch of the story of the VR content. In this case, the branching of the story may be between A-B and B-C, for example, in the case of VR content composed of [A-B-C-D-E] storylines.

The controller 350 may control a 3D object previously mapped and stored to the reaction after confirming the reaction from the user terminal 100 through the pop-up UI. VR content may be content in which at least one pre-mapped 3D object is synthesized in real time according to an exposure step or a user's reaction. If a firefighter is suffering from fire trauma and PTSD exposure treatment is provided, the 3D object may be, for example, a flame or a building on fire. If the anxiety of firefighters (users) is increasing, the number, size, or volume of 3D objects corresponding to the flame may be reduced, or the frequency or cycle may be reduced.

After confirming the reaction from the user terminal 100 through the pop-up UI, the control unit 350 controls a stored 3D object mapped to the reaction, when the reaction confirmed through the pop-up UI is an anxiety-increasing event, Content with lower exposure intensity and reality reflection can be applied in real time, and if the reaction confirmed through the pop-up UI is an anxiety-reducing event, content with increased exposure intensity and reality reflection can be applied in real time. VR content may be content for treating posttraumatic stress disorder (PTSD), phobia, depression, and dementia, but is not limited to those listed and is not excluded for reasons not listed.

<Cloud VR>

Cloud VR can be used when VR content is received in real time from the automation service providing server 300 in the user terminal 100 (client). Cloud VR is a technology that allocates and loads tasks within the edge cloud by considering the amount of resources, bandwidth, and computing resources when multiple users access. In a typical cloud VR operation process in a single user environment, the user's input value and motion sensing data are detected in the client and transmitted to the edge server. The edge server processes VR content based on this input and then renders the user's viewport screen. The rendered result is encoded and transmitted to the client, and the client implements the content by playing it. An edge server is a server instance in the network that processes input signals, renders graphics, and encodes images on behalf of clients. Network delay must be minimized, so it operates physically or geographically close to the user.

At this time, the physical server in the network where the edge server operates is called an edge node. An edge node is a network equipment itself or a physical machine directly connected thereto, and can be regarded as the same as a network node. In this situation, the points to be considered from the network point of view are as follows. For a particular application, information such as compute and network resource requirements, network latency tolerance, compute and networking load variations, etc. must be considered, as well as the edge node's available computing resources. The edge server of a specific application is configured separately for each user, and the edge node where the server operates can be determined to improve content and network performance indicators.

In contrast, in VR content in which 3D objects are changed in real time, each user transmits each input and operation to the automation service providing server 300, and these inputs affect the arrangement of 3D objects. In existing online content, only input signals are shared and processed individually by each client, but in cloud VR, input from all adjacent users needs to be processed by a centralized server. That is, each user's input is transmitted to the central server, data is collected, and engine processing is performed, and the result is transmitted to the edge server associated with each user, rendered for each user's 3D object, and then transmitted to the client. In addition, delay-sensitive operations such as HMD motion processing may be simultaneously sent to and processed by the central server and the edge server. Motion data sent to the central server is used to compose VR contents in which 3D objects are reflected through engine processing, and at the same time, motion data sent to the edge server can be used for rapid VR content rendering according to HMD operation.

In this case, an expected traffic load for each candidate may be obtained using the traffic factor for each link. The traffic factor for each link means the traffic load required for each link on the network path between the edge server and the client. For each node of all node sets in the network, the expected traffic load of the candidate node is obtained by multiplying the traffic factor by the sum of the shortest paths between the corresponding node and each client. In other words, it means the traffic load expected to occur when an edge server is deployed on a node. Candidate nodes obtained in this way are sorted in ascending order based on expected traffic load to form a candidate list. Create each candidate list for every client in the network in the same way.

When selecting candidates considering computing resources, it is best to place edge servers on the best candidate nodes of all clients, but it may not be possible to place all edge servers on a particular node due to insufficient computing resources. If the client's candidate node selection situation is a deployment plan, the best candidate node of all clients is initially initialized as selected. However, if there are overresource nodes, some of the edge servers that were scheduled to be deployed on those nodes must be replaced. A replacement node selection method may use Weighted Vector Bin Packing (W-VBP). The VBP-based replacement technique replaces the server with the greatest difference between the node's resource composition ratio and the edge server's resource composition ratio. Since this is vector-based, there is no limit on the number of resources, and even if only some resources are exhausted, side effects of not being able to use other types of resources are minimized. If all candidates for replacement clients are exhausted, the algorithm is finally considered to have failed. Therefore, the remaining candidate ratio for each client may be considered as a weight in order to prevent a situation in which only candidates for a specific client are continuously exhausted. The edge server with the largest value obtained by multiplying the vector angle, which is the result of VBP, by the weight, becomes the replacement target.

The client holding the edge server to be replaced removes the currently selected candidate node and selects the next candidate node from its own candidate list. Then, based on the current state, the existence of excess nodes is checked again. Algorithm failure may occur when the computing resource demand of VR contents in the network approaches or exceeds the total available resources. This situation can be overcome by adding hardware or applying countermeasures that reduce the level of offloading of contents to edge computing.

Hereinafter, an operation process according to the configuration of the above-described automated service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto.

Referring to FIG. 3, the upper flow is a process of monitoring anxiety by analyzing input sounds and images using an eye tracker, a microphone, and a camera, and then changing a 3D object to proceed with VR content, and the lower flow is First, the user's anxiety is monitored using events occurring in the user terminal 100 without sound and video analysis, and the user's anxiety is checked by secondarily confirming through the pop-up UI, and then the 3D object is changed to VR It is the process of creating content. In the automated service according to an embodiment of the present invention, VR content is configured according to the user's response without the need for counselors or medical staff to manually control or adjust 3D objects, so counselors or medical staff can focus on face-to-face counseling or treatment. there is.

The service according to an embodiment of the present invention can be used for VR exposure therapy as shown in FIG. 4a, can provide personalized content as shown in FIG. 4b, and can be utilized in the field as shown in FIG. 4c, and FIG. It is possible to provide a solution for occupational groups that are frequently exposed to traumatic situations, such as firefighters, police officers, and soldiers who have mental trauma at firefighting sites, and provide personalized VR content by changing the degree of exposure as shown in FIG. 4e. 4f, so that the user can practice in a situation where various fears can be induced, such as public speaking phobia, by being "exposed" to a fear situation, the user can virtually experience the fear situation and eliminate or reduce the user's fear can give As shown in FIG. 4g, it can be used for dementia, and as shown in FIG. 4h, anyone can easily use and use personalized content with high-quality 3D modeling, standard compliance, and user-friendly UI/UX.

Matters not described for the VR exposure treatment automation service provision method of FIGS. 2 to 4 are the same as or easily inferred from the description of the VR exposure treatment automation service provision method through FIG. 1 above. Therefore, the following description is omitted.

5 is a diagram illustrating a process of transmitting and receiving data between each component included in the VR exposure treatment automation service providing system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present application is not limited to such an embodiment, and according to various embodiments described above, It is obvious to those skilled in the art that a process of transmitting and receiving data may be changed.

Referring to Figure 5, the automation service providing server,

The order between the above-described steps (S5100 to S5400) is only an example, and is not limited thereto. That is, the order of the above-described steps (S5100 to S5400) may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters not explained about the VR exposure treatment automation service provision method of FIG. 5 are the same as or easily inferred from the description of the VR exposure treatment automation service provision method through FIGS. 1 to 4 above. Therefore, the following description is omitted.

The VR exposure treatment automation service providing method according to an embodiment described with reference to FIG. 5 may be implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. . Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above-described method for providing an automated VR exposure treatment service according to an embodiment of the present invention may be executed by an application basically installed in the terminal (this may include a program included in a platform or operating system basically loaded in the terminal). and may be executed by an application (that is, a program) directly installed in the master terminal by a user through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the VR exposure treatment automation service providing method according to an embodiment of the present invention described above is implemented as an application (ie, a program) that is basically installed in a terminal or directly installed by a user and can be read by a computer such as a terminal. can be recorded on a recording medium.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (10)

In the automation service providing method executed in the automation service providing server,
Transmitting a diagnostic evaluation questionnaire to a user terminal, and determining an exposure level based on user information and response data received from the user terminal;
transmitting the determined VR content of the exposure step to the user terminal and displaying the VR content;
monitoring events occurring in the user terminal;
If the monitored event coincides with a preset anxiety-increasing event, providing a pop-up UI (User Interface) for checking the user's status to the user terminal; and
After confirming a reaction from the user terminal through the pop-up UI, controlling a 3D object previously mapped and stored to the reaction;
VR exposure treatment automation service providing method comprising a.
According to claim 1,
The method of providing VR exposure treatment automation service, characterized in that the user terminal is connected to a head mounted display (HMD) that outputs the VR content.
According to claim 1,
The preset anxiety rising event,
A motion stop event for a predetermined period of time or longer;
Abnormal event using the VR contents; and
a pause event generated from the user terminal;
A VR exposure treatment automation service providing method comprising any one or a combination of at least one of them.
According to claim 1,
The VR content,
The VR exposure treatment automation service providing method, characterized in that at least one 3D object pre-mapped according to the exposure step or the user's reaction is synthesized content in real time.
According to claim 1,
The pop-up UI,
VR exposure therapy automation service providing method, characterized in that output on the VR content for each story quarter of the VR content.
According to claim 1,
The VR content,
A method for providing VR exposure therapy automation service, characterized in that content for treating PTSD (PostTraumatic Stress Disorder), Phobia, Depression and Dementia.
According to claim 1,
After confirming a reaction from the user terminal through the pop-up UI, the step of controlling a 3D object previously mapped and stored to the reaction,
If the reaction confirmed through the pop-up UI is an anxiety-increasing event, content with lower exposure intensity and reality reflection is applied in real time, and if the reaction confirmed through the pop-up UI is an anxiety-reducing event, exposure intensity and reality reflection are increased Applying content in real time;
A VR exposure treatment automation service providing method characterized in that it is executed while performing.
According to claim 1,
The user terminal is connected to a Head Mounted Display (HMD),
The HMD includes an eye tracking module that tracks the position of the pupil and monitors the size of the pupil;
a microphone for receiving the user's voice and breathing;
VR exposure treatment automation service providing method further comprising.
According to claim 8,
The user terminal includes a camera for photographing the user,
The method of providing an automated VR exposure treatment service, characterized in that the camera photographs at least one body part.
According to claim 9,
The step of monitoring an event occurring in the user terminal,
analyzing the position and pupil size of the user's pupil, the user's voice and breathing sound, and a photograph of the at least one body part;
VR exposure treatment automation service providing method comprising a.

Images