KR101795345B1 - Virtual reality survey system - Google Patents

Abstract

A virtual reality survey system is disclosed. The virtual reality survey system includes a provider device for generating and providing a virtual reality survey, a user device for performing a generated virtual reality survey, and a server for connecting the provider device and the user device. It is possible to provide an efficient learning system using a virtual reality space.

Description

Virtual reality survey system {VIRTUAL REALITY SURVEY SYSTEM}

The present invention relates to a virtual reality survey system.

Virtual Reality (VR) is a human-computer interface that makes a certain environment or situation computerized and makes the person who uses it behave as if they are actually interacting with the surrounding environment.

Augmented Reality (AR) is a technology that superimposes a virtual object on the real world seen by the user. It is called Mixed Reality (MR) because it combines real world and virtual world with additional information in real time and displays it as one image.

Open Patent Publication No. 10-2012-0095325, Aug. 28, 2012

A problem to be solved by the present invention is to provide a virtual reality questionnaire system.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a virtual reality survey system including a first HMD (Head Mount Display), a first user operation including at least one sensor for recognizing an operation of a first user, The method of claim 1, further comprising displaying a list of one or more question items and a questionnaire on the first HMD, selecting at least one of the one or more questionnaire items using the first user action recognition unit, A questionnaire generating unit for recognizing the action and moving the selected at least one questionnaire item to the questionnaire and displaying the questionnaire; a first image acquiring unit including at least one camera; a first image acquiring unit for acquiring, from the image acquired by the first image acquiring unit, A first object recognition unit for recognizing an object and displaying a first indicator for the first object on the first HMD; Wherein the first user action recognition unit recognizes an action of the first user to select a first questionnaire which is one of the one or more questionnaire items and move the first questionnaire to the first indicator by using the first user action recognition unit An object matching unit for moving the first questionnaire to a position of the first indicator and displaying the first questionnaire to match the first object, a first position information acquiring unit for acquiring position information of the first HMD, And displaying the list of at least one questionnaire generated by the questionnaire generating unit in the first HMD, displaying the position information of the first HMD, and using one of the one or more questionnaires Recognizing an operation of the first user to select a second questionnaire and moving the second questionnaire to positional information displayed on the first HMD, And a position information matching unit for matching the second questionnaire with the position information, and matching the first questionnaire, the first questionnaire matched with the first object, the position information, and the position information, A second HMD, a second user motion recognition unit including at least one sensor for recognizing an operation of a second user, and a second user motion recognition unit including a second user motion recognition unit, A second position information acquisition unit for acquiring position information of the second HMD, a second position information acquisition unit for acquiring position information of the second HMD, Displaying a third questionnaire on the second HMD, acquiring a response of the second user to the third questionnaire using the second user action recognition unit, receiving a response to the obtained third questionnaire And the third questionnaire includes a questionnaire item about the indoor environment of the store corresponding to the position information of the second HMD and a questionnaire item about the purpose of visiting the store of the second user, A surveying unit for receiving the location information in the store corresponding to the response of the second user for the purpose of visiting the store from the server and transmitting the guide image for guiding the second user to the location in the store, A second image acquiring unit including at least one camera, a second image acquiring unit for recognizing a second object from the image acquired by the second image acquiring unit, and a second indicator for the second object, A second object recognition unit for displaying a second image on the second HMD, an image of the recognized second object to the server, and a fourth questionnaire corresponding to the second object from the server Displays the fourth questionnaire on the second HMD, acquires a response of the second user to the fourth questionnaire using the second user action recognition unit, and transmits a response to the obtained fourth questionnaire An object-based survey unit for transmitting the second indicator to the server; a selection input for the second user to select the second indicator using the second user action recognition unit; A second HMD for inputting an input window for inputting an opinion of the second user and receiving at least a part of the second object, A user opinion examination unit for capturing at least a part of the images and transmitting the captured image and the opinions of the second user to the server; Acquiring a response of the second user to the fifth questionnaire using the second user action recognition unit, receiving a response of the second user to the fifth questionnaire, receiving a response to the fifth questionnaire, displaying a fifth questionnaire corresponding to the feedback on the second HMD, A feedback processor for transmitting a response to the obtained questionnaire to the server, and a reward according to a response to the fourth questionnaire or the fifth questionnaire from the server, wherein the reward includes a predetermined commodity Acquiring a coupon which can be purchased, displaying the received reward on the second HMD, acquiring a response of the second user as to whether or not the coupon is used by using the second user motion recognition unit, A reward processing unit for transmitting the use status to the server; And a response for delivering to the provider terminal a response from the second user regarding whether to use the third questionnaire, the fourth questionnaire, the fifth questionnaire, and the coupon received from the user terminal, Wherein the indoor environment of the store is controlled according to a response of the second user to the indoor environment of the store, wherein the indoor environment of the store is related to the temperature, brightness And at least one of background music, and an indoor environment control unit.

The second object is a logo displayed on a product purchased by the second user at the store or a product purchased by the second user at the store, and the second object recognizing unit may recognize the second object, The method of claim 1, further comprising: receiving settlement information for purchasing an object; acquiring a kind of goods corresponding to the payment information; determining whether the recognized second object corresponds to the acquired product type; In the case where it corresponds to the brand logo, the second indicator can be displayed.

The second object recognizer may display a marker indicating that a questionnaire matched with the second object is present together with the second indicator when there is a questionnaire matched to the second object, Information about a reward to be paid may be displayed together with the second indicator when responding to a questionnaire matched with the second indicator.

The second user action recognizing unit recognizes that the second user selects the second object or the second indicator with a finger while the second user recognizes that the second user selects the second object or the second indicator, Based survey section to display a questionnaire matched to the second object when the shape of the finger of the second user is a first form, If the shape is of the second type, the user can inquire the user's opinion to input the opinion on the second object of the second user.

If the answer to the fourth questionnaire is negative, the object-based surveying unit displays an input window for inputting a cause of the negative response, receives the cause of the negative response, and receives at least one Recognizes a third object corresponding to the extracted keyword in the image acquired by the second image acquisition unit, captures an image of the third object, and acquires the image of the captured third object And transmit the cause of the input to the server.

The user terminal may further include an emotion state determining unit that can determine the emotion state of the second user by acquiring body information of the second user, and the object- And transmits the negative response and the emotional state of the second user judged by the emotional state judging unit to the server when the response is negative, and the user opinion examination unit judges whether the negative keyword And transmits the emotional state of the second user determined by the emotional state determiner to the server if the emotional state is included, and the body information may include at least one of breath, pulse, and body temperature have.

In addition, when the content of the product defect corresponding to the second object is included in the opinion of the second user received from the user opinion examination unit of the user terminal, And a feedback management unit for transmitting the fifth questionnaire including a questionnaire item about exchange or refund to the user terminal, wherein the response transfer unit receives from the feedback processing unit of the user terminal, When the response requesting exchange or refund is received, information requesting exchange or refund of the goods corresponding to the second object is transmitted to the order system of the store, and the use of the coupon from the reward processing unit of the user terminal When a requesting response is received, information on the use request of the coupon is transmitted to the ordering system of the store can do.

The server may store and manage at least one of a position of seats disposed in the store, whether or not each of the seats is used, a position of goods displayed on the store, and an inventory of goods displayed on the store, And a store manager for transmitting the information to the user terminal to guide the second user to a product displayed on the store or to an empty seat arranged in the store according to a visit purpose of the second user received from the user terminal can do.

In addition, the indoor environment control unit adjusts an indoor environment of the store according to one or more users' responses to the indoor environment of the store, and when the responses of the one or more users are different from each other, When the second user is determined to leave the store by using the location information of the second HMD obtained by the second location information obtaining unit, The indoor environment control unit transmits information indicating that the second user leaves the store to the server, and when the indoor environment control unit receives from the user terminal information that the second user leaves the store, In response to the at least one user's response to the indoor environment of the store It can be adjusted using the average of merging the indoor environment of the store.

The object-based surveying unit may also receive a sixth questionnaire corresponding to the recognized second object from the server, wherein the sixth questionnaire includes at least one learning question associated with the second object, Wherein the sixth questionnaire is displayed on the second HMD and the second questionnaire is displayed on the sixth questionnaire using the second user action recognition unit. The feedback processing unit receives from the server a seventh questionnaire for explaining the learning problem and further learning, if the response of the second user is an incorrect answer, and displays it on the second HMD And the reward processing unit receives the reward according to the correct answer from the server and displays it on the second HMD when the answer of the second user is correct have.

Other specific details of the invention are included in the detailed description and drawings.

According to the disclosed embodiment, there is an effect that a user can provide a convenient interface for generating surveys in a virtual reality space, and can provide various services by matching with the real space.

In addition, when the virtual reality questionnaire system according to the disclosed embodiment is utilized in education, it is possible to provide an efficient learning system using the virtual reality space.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating a virtual reality survey system in accordance with one embodiment.
2 is a diagram illustrating a method by which a provider terminal generates a survey in accordance with one embodiment.
3 is a diagram illustrating a method by which a provider terminal generates a survey that interacts with an object, in accordance with one embodiment.
4 is a diagram illustrating a method of inputting a survey at a provider terminal according to an embodiment.
5 is a diagram illustrating a method of conducting a survey using a mobile terminal according to an embodiment.
6 is a diagram illustrating a method by which a user terminal performs a survey according to an embodiment.
7 is a diagram illustrating a method by which a user terminal transmits a user's opinion according to an embodiment.
8 is a diagram illustrating a method by which a user terminal conveys an opinion on a product according to an embodiment.
9 is a diagram illustrating a training method using a virtual reality questionnaire according to an exemplary embodiment of the present invention.
10 is a configuration diagram illustrating a provider terminal according to an embodiment.
11 is a configuration diagram illustrating a user terminal according to an embodiment.
12 is a configuration diagram showing a server according to an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, Is provided to fully convey the scope of the present invention to a technician, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element. Like reference numerals refer to like elements throughout the specification and "and / or" include each and every combination of one or more of the elements mentioned. Although "first "," second "and the like are used to describe various components, it is needless to say that these components are not limited by these terms. These terms are used only to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense that is commonly understood by one of ordinary skill in the art to which this invention belongs. In addition, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

As used herein, the term "part" or "module" refers to a hardware component, such as a software, FPGA, or ASIC, and a "component" or "module" performs certain roles. However, "part" or " module " is not meant to be limited to software or hardware. A "module " or " module " may be configured to reside on an addressable storage medium and configured to play back one or more processors. Thus, by way of example, "a" or " module " is intended to encompass all types of elements, such as software components, object oriented software components, class components and task components, Microcode, circuitry, data, databases, data structures, tables, arrays, and variables, as used herein. Or " modules " may be combined with a smaller number of components and "parts " or " modules " Can be further separated.

The terms spatially relative, "below", "beneath", "lower", "above", "upper" And can be used to easily describe a correlation between an element and other elements. Spatially relative terms should be understood in terms of the directions shown in the drawings, including the different directions of components at the time of use or operation. For example, when inverting an element shown in the figures, an element described as "below" or "beneath" of another element may be placed "above" another element . Thus, the exemplary term "below" can include both downward and upward directions. The components can also be oriented in different directions, so that spatially relative terms can be interpreted according to orientation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In this specification, the virtual reality and the virtual reality image are not limited to the VR (Virtual Reality) and the VR image but may be a virtual reality (VR), a virtual reality image, an augmented reality (AR) , Mixed reality (MR), mixed reality images, and general images, and is used to include all types of images including real, virtual, real, and virtual images.

1 is a diagram illustrating a virtual reality survey system in accordance with one embodiment.

Referring to FIG. 1, a virtual reality survey system includes a server 10, a provider terminal 100, and a user terminal 200.

In the embodiment shown in FIG. 1, the provider terminal 100 includes a display 110 and a client 120. In addition, the user terminal 200 includes a display 210 and a client 220.

The displays 110 and 210 shown in FIG. 1 may be applied to various displays such as a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display ), A three-dimensional display (3D display), a head mount display (HMD), a smart glass, and other virtual reality image viewing apparatuses, and also includes a display Is not limited to this, and is used as a meaning including any type of device capable of displaying an image.

In addition, the clients 120 and 220 shown in FIG. 2 refer to a client included in a computing device including at least one processor, or the computing device itself. Computing devices may include, but are not limited to, desktops, laptops, smart phones, tablet PCs, and servers.

In the disclosed embodiment, the provider terminal 100 generates at least one survey and sends it to the server 10.

The server 10 transmits the questionnaire generated by the provider terminal 100 to the user terminal 200.

The server 10 receives feedback including a response to the survey from the user terminal 200 and transmits the received feedback to the provider terminal 100. [

In the disclosed embodiment, the questionnaire refers to any form of content that can ask questions and answers to questions in a multiple choice or open-ended format. For example, the questionnaire may be a content including a question for asking the user's thoughts, intentions or evaluations, or a learning content in the form of presenting a problem and requesting an answer thereto.

The provider terminal 100 generates a survey in a virtual reality environment, and the user terminal 200 also responds to a survey in a virtual reality environment.

As described above, the virtual reality of the present specification is understood as a concept including both Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

Accordingly, the provider terminal 100 generates a questionnaire that can interact with a virtual object or an existing object included in the virtual reality, and the user terminal 200 interacts with a predetermined virtual object or an existing object, You can respond to inquiries.

In one embodiment, the user terminal 200 interacts with a virtual object or an existing object whose survey is not predefined, thereby generating a comment or response to the virtual object or the existing object, Lt; / RTI >

The server 10 may transmit the feedback or the response received from the user terminal 200 to the provider terminal 100.

2 is a diagram illustrating a method by which a provider terminal generates a survey in accordance with one embodiment.

Referring to FIG. 2, an example of a virtual reality screen 300 displayed on a provider terminal is shown.

In one embodiment, the virtual reality screen 300 displays a questionnaire 310 and one or more questionnaire items 320. Also, a virtual keyboard 330 is displayed on the virtual reality screen 300 for directly inputting a questionnaire or for modifying inputted questionnaire items.

The provider terminal 100 receives input from a provider according to the movement of the provider. For example, if the display 110 included in the provider terminal 100 is in the form of an HMD, the provider terminal 100 may receive an input from the provider according to the head movement of the provider using the marker displayed on the HMD . As another example, the provider terminal 100 may recognize the hand movements of the provider and receive inputs that are manually indicated by the provider. In one embodiment, the provider terminal 100 may further comprise a module for recognizing the position of the user's hand and the shape of the user's hand.

In one embodiment, the provider terminal 100 receives an action input by which a provider selects at least one question item 322 and moves it to the questionnaire 310. For example, the provider terminal 100 may receive a drag-and-drop input by which the provider grabs the survey item 322 by hand and moves it to the questionnaire 310.

The provider terminal 100 adds the item 322 to the questionnaire 310 and displays the questionnaire 310 to which the item 322 is added.

3 is a diagram illustrating a method by which a provider terminal generates a survey that interacts with an object, in accordance with one embodiment.

Referring to FIG. 3, a virtual reality image 400 displayed on the provider terminal 100 is shown.

In the virtual reality image 400 of FIG. 3, one or more surveys 410 and objects 420 are displayed.

In one embodiment, the object 420 may be a virtual object or an existing object.

The provider terminal 100 recognizes the object 420 from the virtual reality image 400 using the image processing. The provider terminal 100 may indicate an indication that an object 420 has been recognized and an indicator 425 for indicating a range of the recognized object 420. [ The indicator 425 may be in the form of an outline indicating the range of the object 420 as shown in FIG. 3, or may be displayed in a form of changing the color of the object 420.

In addition, the provider terminal 100 recognizes the marker 430 from the virtual reality image 400 using image processing. In one embodiment, marker 430 may be a logo displayed on object 420, or may be a two-dimensional code such as a QR code.

For example, the provider terminal 100 recognizes a coffee cup, which is an existing object 420, as shown in FIG. 3, and displays an indicator 425 for the object 420. Likewise, the provider terminal 100 can recognize the marker 430, i.e., the logo, displayed on the coffee cup, as shown in FIG. 3, and display an indicator 435 indicating that the logo has been recognized.

In one embodiment, the provider terminal 100 searches the database for images recognized in the coffee cup or images stored on an external server to determine whether the logo is indicative of a particular name, and if the recognized image is a logo It is determined that the recognized image is the marker 430 and the indicator 435 is displayed.

The provider terminal 100 determines the current location of the provider terminal 100 and displays location information 440 including information on the store or facility corresponding to the current location or the current location of the provider terminal 100 do. The provider terminal 100 may display an indicator 445 indicating the location information 440. [

In one embodiment, the provider terminal 100 receives an action input that the provider selects and matches at least one survey to an object displayed in the virtual reality image 400. For example, the provider terminal 100 may receive a drag-and-drop input by which the provider grabs the survey 412 by hand and moves it to the object 420 or the indicator 425 of the object.

The provider terminal 100 matches at least one survey with an object according to an input from the user and stores the matching object and the information about the survey to the server 10.

Likewise, the provider terminal 100 may match the questionnaire 414 and the marker 430 and match the questionnaire 416 and the location information 440 according to the user's input. The provider terminal 100 transmits information on each survey and information on objects matched with each survey to the server 10.

When the user terminal 200 is connected to the server 10 and the user terminal 200 enters a location corresponding to the location information 440, the user terminal 200 transmits a survey 416 matched with the location information 440 to the user terminal 200 or an indicator indicating that there is a questionnaire 416 matched with the position of the user terminal 200 on the user terminal 200.

For example, the location corresponding to the location information 440 is a cafe, the questionnaire 416 asks whether the mood of the cafe is, or if the internal environment of the cafe (e.g., the current cafe temperature (cooling / And what brightness is, etc.). In response to the user's response to the temperature of the cafe, the server 10 can automatically adjust the temperature in conjunction with the cooling control system of the cafe. Similarly, in response to the user's response to the brightness of the cafe, the server 10 can automatically adjust the brightness in conjunction with the cafe's lighting control system. As another example, the server 10 may transmit the responses of the users to the provider terminal 100 so that the provider directly adjusts the internal environment of the cafe.

In addition, the questionnaire 416 may include a question about the purpose of visiting a cafe of a user and the number of parties. When the user visits the cafe for conversation with the acquaintance, the user terminal 200 guides the user through a multi-lingual conversation, and when the user visits the cafe for reading alone, the user terminal 200 displays the user in a quiet place Can guide.

For example, the server 10 manages information about a seating state of a cafe, determines a position of a seat suitable for a user's purpose among empty seats, displays an indicator indicating the position of an empty seat in the user terminal 200 Or a direction indicator or walking navigation that can guide the user to an empty seat.

The user terminal 200 transmits information about the user's visit and information about the place where the user is seated to the server 10, and the server 10 can deliver the information to the provider terminal 100.

In addition, the user terminal 200 may be connected to the server 10, and may receive information on the fact that the user can receive compensation according to the survey 416 matched with the location of the user terminal 200, May be displayed together with an indicator indicating the presence of the survey (416), thereby inducing the user to participate in the survey.

In addition, a survey of the object 420 and the marker 430 may be stored in the server 10 together with the location information 440. For example, the server 10 may correspond to the object 420 and the marker 430 only when the user terminal 200 recognizes the object 420 and the marker 430 at a position corresponding to the positional information 440 To display a questionnaire.

4 is a diagram illustrating a method of inputting a survey at a provider terminal according to an embodiment.

Referring to FIG. 4, a virtual reality image 500 displayed on a provider terminal 100 is shown.

The provider terminal 100 displays the recognized object 520 and its indicator 525, the recognized marker 530 and its indicator 535.

The provider terminal 100 recognizes the operation of the provider and receives a selection input that the provider selects at least one of the object 520 and its indicator 525, the recognized marker 530 and its indicator 535, And displays an interface for generating a survey 510 to match the object.

The provider terminal 100 recognizes the operation of the provider and inputs the contents of the survey 510 using the virtual keyboard or inputs the contents of the survey 510 in the same manner as the embodiment shown in FIG. have.

5 is a diagram illustrating a method of conducting a survey using a mobile terminal according to an embodiment.

5, a mobile terminal 600 is used to capture an object 610 or a marker 620 and a mobile terminal 600 displays a survey corresponding to an object 610 or a marker 620 An embodiment is shown.

6 is a diagram illustrating a method by which a user terminal performs a survey according to an embodiment.

Referring to FIG. 6, a virtual reality screen 700 displayed on the user terminal 200 is shown.

In one embodiment, the virtual reality screen 700 displays an object 720.

The object 720 may be a virtual object or an existing object.

The user terminal 200 recognizes the object 720 from the virtual reality image 700 using image processing. The user terminal 200 may indicate that the object 720 has been recognized and may indicate an indicator 725 to inform the recognized object 720 that the survey is matched. The indicator 725 may be in the form of an outline indicating the range of the object 720 as shown in FIG. 6, or may be displayed in a form of changing the color of the object 720.

Also, the user terminal 200 recognizes the marker 730 from the virtual reality image 700 using image processing. In one embodiment, the marker 730 may be a logo displayed on the object 720, or may be a two-dimensional code such as a QR code.

For example, the user terminal 200 may recognize a coffee cup, which is an existing object 720, as shown in FIG. 6, and a matching survey for the object 720 at the location of the user terminal 200 Or not. If there is a matching survey for object 720 at the location of user terminal 200, indicator 725 is displayed. Similarly, the user terminal 200 recognizes the marker 730, i.e., the logo displayed in the coffee cup, as shown in FIG. 6, and if the survey is matched to the recognized logo at the location of the user terminal 200, And an indicator 735 indicating that there is a matching survey.

In one embodiment, the user terminal 200 may retrieve images recognized in a coffee cup with images stored in a database or an external server to determine whether the logo is indicative of a particular name, and if the recognized image is a logo It is determined that the recognized image is the marker 730. In addition, the marker 730 is searched for whether the survey is matched, and if there is a matching survey, the indicator 735 is displayed.

The user terminal 200 receives a user input that selects an object 720 or indicator 725 based on the user's actions and displays a survey 710 that is matched to the object 720. For example, the questionnaire 710 may include questions about the taste of coffee.

The user terminal 200 determines the response of the survey 710 according to the operation input of the user, and transmits the determined response to the server 10.

The server 10 transmits the user's response to the provider terminal 100.

The user terminal 200 also receives a user input for selecting the marker 730 or indicator 735 based on the user's actions and displays a survey 740 that matches the marker 730. For example, the survey 740 may include a question about an employee's service.

The user terminal 200 determines the response of the survey 740 according to the operation input of the user and transmits the determined response to the server 10.

The server 10 transmits the user's response to the provider terminal 100.

Also, the user terminal 200 may receive information on the user's opinion, such as an improvement, after the responses to the surveys 710 and 740, and may transmit the information to the server 10. For example, the user terminal 200 may display a window 750 for receiving user feedback after the responses to the surveys 710 and 740 are completed.

When an input is received from the user, the user terminal 200 may transmit the user's opinion to the server 10. [

7 is a diagram illustrating a method by which a user terminal transmits a user's opinion according to an embodiment.

Referring to FIG. 7, a virtual reality screen 800 displayed on the user terminal 200 is shown.

In one embodiment, a virtual reality screen 800 displays a window 810 for receiving a user's opinion.

The user terminal 200 receives the input from the user and displays the inputted input in the window 810.

In one embodiment, the user terminal 200 matches the user's opinion with the object 820 corresponding to the opinion in accordance with the user's action input. For example, the user terminal 200 receives a gesture input that links the object 820 with the user's feedback displayed in the window 810.

The user terminal 200 displays a survey 830 for the object 820. For example, the user terminal 200 determines a keyword included in a user's opinion, recognizes that the user's opinion is related to the drink, and sends a survey 830 querying the type of beverage corresponding to the object 820 Display.

The user terminal 200 determines the type of beverage according to the user's input and transmits the determined type of beverage to the server 10 along with the user's opinion.

When the user's opinion is received, the server 10 provides compensation for this. The user terminal 200 displays the compensation 840 provided from the server 10. For example, compensation 840 may be a coupon that can be used in a cafe where user terminal 200 is located.

In this case, the user terminal 200 can store the coupon according to the input of the user, or can immediately order the goods corresponding to the coupon.

When the user requests to order the goods corresponding to the coupon immediately, the user terminal 200 transmits order information about the goods corresponding to the coupon to the server 10, and the server 10 transmits the received order information to the provider And transmits it to the terminal 100.

When the order information is received through the provider terminal 100, the provider can directly provide the product corresponding to the coupon to the user.

8 is a diagram illustrating a method by which a user terminal conveys an opinion on a product according to an embodiment.

Referring to FIG. 8, a virtual reality screen 900 displayed on the user terminal 200 is shown.

In one embodiment, an object 920 may be displayed in the virtual reality screen 900. [

The user terminal 200 may recognize the object 920 and receive a selection input for the recognized object 920 from the user.

The user terminal 200 displays a window 910 where an input for the object 920 can be entered. The user terminal 200 acquires the user's opinion about the object 920 according to the operation input of the user and displays the acquired user's opinion on the window 910.

The user terminal 200 matches at least a portion of the object 920 or the object 920 with the user's opinion displayed in the window 910 according to the user's gesture input.

For example, the user terminal 200 recognizes the finger movements of the user moving from the user's opinion displayed in the window 910 to the foreign matter adhering to the object 920 and the finger movement of the user pointing around the foreign object in a circle , It is possible to match the user's opinion with the position indicated by the user with the finger. Also, the user terminal 200 may analyze the keyword included in the user's opinion to determine that there is foreign substance at the position indicated by the user.

The user terminal 200 enlarges and photographs the position pointed by the user, and displays the photographed image. The user terminal 200 can capture (store) the photographed image or transmit it to the server 10 according to a user's input.

The user terminal 200 transmits the user's opinion to the server 10 together with the photographed image, and the server 10 transmits the transmitted opinions and images to the provider terminal 100. [

The server 10 delivers the message to the user terminal 200 automatically or based on the information received from the provider terminal 100. The user terminal 200 displays a survey 940 corresponding to the delivered message.

The user terminal 200 transmits a response corresponding to the survey 940 to the server 10.

The server 10 forwards the response to the provider terminal 100 and forwards the compensation to the user terminal 200.

The user terminal 200 displays the received compensation 950.

The compensation 950 shown in FIG. 8 corresponds to the compensation 840 shown in FIG.

Thus, the method described in connection with the compensation 840 shown in FIG. 8 applies to the method of providing the user's action and hence the compensation for the compensation 950.

9 is a diagram illustrating a training method using a virtual reality questionnaire according to an exemplary embodiment of the present invention.

Referring to FIG. 9, a virtual reality screen 1000 displayed on the user terminal 200 is shown.

The virtual reality screen 1000 includes at least one object 1010. The object 1010 may be a virtual object or an existing object.

In one embodiment, the provider terminal 100 shown in FIG. 1 is used to produce educational content, which can be used to create a learning questionnaire (i.e., a question or problem) in accordance with the embodiment shown in FIG. 3, Match.

The learner performs learning using the virtual reality image 1000 using the user terminal 200 and selects the object 1010 including the indicator 1020. [ The user terminal 200 displays a questionnaire 1030 matched with the object 1010 according to the learner's movement.

The learner responds to the question included in the questionnaire 1030, and the user terminal 200 transmits a response according to the learner's movement to the server 10. [

The server 10 transmits feedback based on the learner's response to the user terminal 200. [ For example, the server 10 transmits to the user terminal 200 whether or not the learner's response is correct and the learner's learning content to be re-learned if the learner's response is wrong, Display content.

In addition, after the learning ends, the user terminal 200 can display a questionnaire on the learning method using the virtual reality image on the virtual reality image 1000. [

For example, the user terminal 200 can display a questionnaire about what is better or worse than a learning method using a virtual image or a learning method using a textbook. The user terminal 200 can transmit the feedback (response) according to the learner's operation to the server 10.

10 is a configuration diagram illustrating a provider terminal according to an embodiment.

10, the provider terminal 100 includes a first HMD 110, a first image acquiring unit 122, a questionnaire generating unit 124, a first object recognizing unit 126, an object matching unit 128, A first location information obtaining unit 130, a location information matching unit 132, a communication unit 134, and a first user action recognizing unit 136. [

10 is shown for convenience of explanation, and the connection relationship between the respective components shown in FIG. 10 is not limited to this.

10, the provider terminal 100 includes a first HMD (Head Mount Display) 110, a first user motion recognition unit 136 including at least one sensor for recognizing the operation of the first user ), Displays at least one list of questionnaire items and a questionnaire on the first HMD 110, selects at least one of the one or more questionnaire items using the first user action recognizer 136, and moves to the questionnaire A questionnaire generating unit 124 for recognizing the operation of the first user and moving the selected at least one questionnaire item to the questionnaire, a first image acquiring unit 122 including at least one camera, A first object recognition unit 126 for recognizing a first object from the image acquired by the first image acquisition unit 122 and displaying a first indicator for the first object on the first HMD 110, The first HMD 110 is provided with a questionnaire The method of claim 1, further comprising: displaying a list of one or more questionnaires generated by the verifying unit (124), selecting a first questionnaire, one of the one or more questionnaires, using the first user action recognizing unit An object matching unit 128 for recognizing the operation of the first user, moving the first questionnaire to the position of the first indicator and displaying the first questionnaire, matching the first questionnaire with the first object, A first position information obtaining unit 130 for obtaining position information of the first HMD 110 and a second position information obtaining unit 130 for obtaining position information of the first HMD 110, And the first user action recognizing unit 136 selects the second questionnaire, which is one of the one or more questionnaires, to move to the position information displayed on the first HMD 110, Recognize the behavior of A position information matching unit 132 for displaying the second questionnaire at a position where the position information is displayed and matching the second questionnaire with the position information, and a position information matching unit 132 for matching the first questionnaire with the first questionnaire, And a communication unit 134 for transmitting the first questionnaire, the location information, and the second questionnaire matched with the location information to the server 10.

11 is a configuration diagram illustrating a user terminal according to an embodiment.

11 is shown for convenience of explanation, and the connection relationship between the respective components shown in FIG. 11 is not limited to this.

11, the user terminal 200 includes a second HMD 210, a second user motion recognition unit 242 including at least one sensor for recognizing the operation of the second user, A second position information obtaining unit 234 for obtaining position information of the HMD 210 and a second position information obtaining unit 234 for obtaining position information of the second HMD 210 from the server 10, The third questionnaire is displayed on the second HMD 210 and the third questionnaire is displayed on the third questionnaire using the second user action recognition unit 242. [ And transmits the obtained response to the server 10 to the server 10, and the third questionnaire includes a response to the location information of the store corresponding to the location information of the second HMD 210 A questionnaire item for the indoor environment and a questionnaire item for the purpose of visiting the store of the second user, Based surveying unit 236 for receiving the in-store location information corresponding to the response of the second user to the purpose of visiting the store from the server 10, and for notifying the second user to the in- A second image acquiring unit 222 including at least one camera, a second acquiring unit 222 for acquiring images acquired by the second image acquiring unit 222, A second object recognition unit 224 for recognizing a second object from the first HMD 210 and displaying a second indicator for the second object on the second HMD 210, Receives the fourth questionnaire corresponding to the second object from the server 10, displays the fourth questionnaire on the second HMD 210, and the second user action recognition unit 242 ) To obtain the response of the second user to the fourth questionnaire An object-based questionnaire section 226 for transmitting a response to the obtained questionnaire to the server 10, and a second user action recognizing section 242 for receiving the second indicator Receives an input of a selection from the second HMD 210, receives an input of the second user, displays an input window for inputting a second user's opinion on the second object on the second HMD 210, Capturing an image of at least a portion of the second object indicated according to an operation of the second user indicating at least a portion of the second object, and transmitting the captured image and the opinion of the second user to the server A feedback inquiry unit 232 for receiving feedback on the feedback of the second user from the server 10, displaying a fifth questionnaire corresponding to the feedback on the second HMD 210, 2 User motion recognition (228) for obtaining a response of the second user to the fifth questionnaire using the second questionnaire (242) and transmitting a response to the obtained questionnaire to the server (10), and a server ) To the fourth questionnaire or the fifth questionnaire, wherein the reward includes a coupon to purchase a predetermined commodity at the store, and the received reward is transmitted to the second HMD 210 , The second user action recognition unit 242 acquires the response of the second user as to whether or not the coupon is used, and displays a reward for transmitting the use of the coupon to the server 10 And a processing unit 230.

In an embodiment, the second object is a logo displayed on a product purchased by the second user at the store or a product purchased by the second user at the store, and the second object recognizing unit 224 The method comprising: receiving settlement information in which a second user has purchased the second object, acquiring a kind of goods corresponding to the settlement information, determining whether the recognized second object corresponds to the acquired product type, And displays the second indicator when the second object corresponds to the brand logo of the store.

If there is a questionnaire matched with respect to the second object, the second object recognizer 224 recognizes a marker indicating that a questionnaire matched to the second object is present together with the second indicator And displays information on a reward to be paid when the second responds to the questionnaire matched with the second object together with the second indicator.

In one embodiment, the second user motion recognizing unit 242 recognizes that the second user selects the second object or the second indicator with a finger, and the second user or the second indicator And if the shape of the finger of the second user is the first shape, the object-based survey unit 226 is called to display a questionnaire matched to the second object If the shape of the finger of the second user is the second type, the user inquiry checking unit 232 is called to receive the input of the second user's opinion on the second object. For example, the first form may be a finger shape, and the second form may be a finger shape.

In one embodiment, when the response to the fourth questionnaire is negative, the object-based survey unit 226 displays an input window for inputting the cause of the negative response, receives the cause of the negative response, Extracts at least one keyword from the input cause, recognizes a third object corresponding to the extracted keyword in the image acquired by the second image acquiring unit 222, captures an image of the third object, And transmits the image of the captured third object and the cause of the input to the server (10).

In one embodiment, the user terminal 200 further includes an emotion state determination unit 240 that can determine the emotion state of the second user by acquiring the body information of the second user, 226) transmits the negative response and the emotional state of the second user determined by the emotional state determination unit to the server when the response to the fourth questionnaire is negative, and the user opinion investigation unit (232) The second user's opinion and the emotional state of the second user determined by the emotional state determiner 240 are transmitted to the server when a negative keyword is included in the inputted second user's opinion, The body information includes at least one of breathing, pulse and body temperature.

In one embodiment, the object-based survey unit 226 receives a sixth questionnaire corresponding to the recognized second object from the server 10, wherein the sixth questionnaire includes at least one Receives a correct answer of at least one learning question included in the sixth questionnaire from the server and displays the received sixth questionnaire on the second HMD 210, The feedback processing unit 228 obtains a response of the second questionnaire to the sixth questionnaire using the motion recognition unit 242. If the answer of the second user is an incorrect answer, A seventh questionnaire for additional learning is received from the server and displayed on the second HMD 210. When the response of the second user is correct, the reward processor 230 transmits a reward according to the correct answer, Lobu It received, and displayed on the HMD of claim 2 (210).

12 is a configuration diagram showing a server according to an embodiment.

12 is shown for convenience of explanation, and the connection relationship between the respective components shown in FIG. 12 is not limited to this.

Referring to FIG. 12, the server 10 receives a response of the second user regarding whether the third questionnaire, the fourth questionnaire, the fifth questionnaire, and the coupon, which are received from the user terminal 200, A response transmission unit (12) for transmitting the response to the provider terminal and the indoor environment control system of the store to adjust the indoor environment of the store according to the response of the second user to the indoor environment of the store, The indoor environment includes at least one of temperature, brightness, and background music of the store.

In one embodiment, the server 10 includes information on the defect of the goods corresponding to the second object in the opinion of the second user received from the user opinion inquiry unit 232 of the user terminal 200 And a feedback management unit (16) for transmitting the fifth questionnaire including a questionnaire item regarding exchange or refund of a commodity corresponding to the second object to the user terminal, wherein the response management unit ) Receives a request to exchange or refund a commodity corresponding to the second object from the feedback processing unit (228) of the user terminal (200), exchanges or refunds the commodity corresponding to the second object When a response requesting use of the coupon is received from the reward processing unit (230) of the user terminal (200), the request for use of the coupon is transmitted to the order system of the store And transmits the information to the order system of the store.

In one embodiment, the server stores and manages at least one of the position of the seats disposed in the store, the availability of each of the seats, the location of the commodities displayed on the store, and the inventory of commodities displayed on the store And transmits to the user terminal information for guiding the second user to a vacant seat arranged in the commodity or in the store according to a visit purpose of the second user received from the user terminal, (18).

In one embodiment, the indoor environment adjustment unit 14 adjusts the indoor environment of the store according to one or more user responses to the indoor environment of the store, and when the responses of the one or more users are different, The user terminal 200 adjusts the indoor environment of the store using the average of the user's responses and the location information of the second HMD 210 obtained by the second location information obtaining unit 234 To the server (10), information indicating that the second user leaves the store when it is determined that the second user leaves the store, and the indoor environment control unit (14) (200), when the second user receives information indicating that the second user leaves the store, a response to the indoor environment of the store Using the average of all but the response of the second user group to control the interior environment of the store.

In one embodiment, the indoor environment control unit 14 assigns the same weight to each of the responses of one or more users, and uses the average to adjust the indoor environment of the store.

For example, if half of the store's users want to raise the temperature and the other half wants to lower the temperature, the current temperature can be maintained as the middle.

According to another embodiment, the indoor environment control unit 14 may assign different weights to each of one or more users. For example, the indoor environment control unit 14 may assign a corresponding weight value to the indoor environment control unit 14 in consideration of the number of parties when the number of the parties is plural, even though the number of the respondents is one. In addition, the indoor environment control unit 14 may assign different weights in the case of a regular customer or a VIP customer in consideration of the visit record and cumulative points in the meantime. In addition, the indoor environment control unit 14 can sell a product that can change the indoor environment of the store through a separate billing system provided in the user terminal 200.

For example, the second user can purchase a commodity that pays a predetermined fee and can lower the room temperature of the store by one degree. In addition, the second user can pay a predetermined fee, adjust the indoor brightness of the store, or purchase a product that can change the background music of the store.

As described above, the second user can obtain a right to change the background music of the store when paying a predetermined fee or frequently using a specific store. Accordingly, when the second user enters the store and selects the desired music using the location-based survey section, the store can play the music selected by the second user in accordance with the position of the second user.

Likewise, you can apply a gay messaging method that allows you to place an auction or point competition between different users with a store's background music option.

Also, the user terminal 200 may recognize the user terminal of another third user or a third user in the store using the second image acquisition unit 222, and transmit the third user or the third user to the second HMD 210 An indicator corresponding to the user terminal of the user terminal can be displayed. The user terminal 200 can recognize the operation of the second user selecting the user terminal of the third user or the third user and can display a window for inputting or selecting the request of the second user.

The user terminal 200 sends a message to the server 10 to request the server 10 to send a message to the third user via the server 10 or to ask the third user to reduce the conversation sound through the clerk, . In addition, the user terminal 200 may transmit a message to the server 10 to transfer the beverage to the third user at the request of the second user, and settle the cost. The server 10 may deliver a message to the store's order system to deliver the beverage to the third user at the request of the user terminal 200. [

The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in software modules executed in hardware, or in a combination of both. The software module may be a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD- May reside in any form of computer readable recording medium known in the art to which the invention pertains.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

10: Server
100: Provider terminal
200: user terminal

Claims (10)

A first HMD (Head Mount Display);
A first user motion recognition unit including at least one sensor for recognizing an operation of a first user;
Displaying a list of one or more questionnaire items and a questionnaire on the first HMD and selecting at least one of the one or more questionnaire items using the first user action recognition unit to recognize the action of the first user to move the questionnaire to the questionnaire A questionnaire generating unit for displaying at least one selected question item on the questionnaire;
A first image acquiring unit including at least one camera;
A first object recognition unit for recognizing a first object from the image acquired by the first image acquisition unit and displaying a first indicator for the first object on the first HMD;
A list of one or more questionnaires generated by the questionnaire generating unit is displayed on the first HMD and a first questionnaire which is one of the one or more questionnaires is selected using the first user motion recognizing unit and moved to the first indicator An object matching unit for recognizing the operation of the first user, moving the first questionnaire to a position of the first indicator and displaying the first questionnaire, and matching the first questionnaire with the first object;
A first position information acquiring unit for acquiring position information of the first HMD;
And displaying the list of one or more questionnaires generated by the questionnaire generating unit in the first HMD, displaying the position information of the first HMD, and using the first user motion recognizing unit, The second questionnaire is moved to a position where the position information is displayed and the second questionnaire is displayed on the position information of the first questionnaire, A position information matching unit for matching the position information with the position information; And
A communication unit for transmitting information on the first object, the first questionnaire matched with the first object, the location information, and the second questionnaire matched with the location information to the server; A provider terminal;
A second HMD;
A second user motion recognition unit including at least one sensor for recognizing an operation of a second user;
A second position information acquiring unit for acquiring position information of the second HMD;
Transmitting the position information of the second HMD to the server, receiving a third questionnaire corresponding to the position information of the second HMD from the server, displaying the third questionnaire on the second HMD, Acquires a response of the second user to the third questionnaire using the user's action recognition unit, transmits a response to the obtained questionnaire to the server, and the third questionnaire includes position information of the second HMD Based surveying unit including a questionnaire item on the indoor environment of the store corresponding to the second user and a questionnaire item on the purpose of visiting the store of the second user;
Store location information corresponding to a response of the second user to the store visit destination from the server, and displays a guide image guiding the second user to the in-store location on the second HMD A user guide section for performing a user operation;
A second image acquiring unit including at least one camera;
A second object recognition unit for recognizing a second object from the image acquired by the second image acquisition unit and displaying a second indicator for the second object on the second HMD;
Transmitting the image of the recognized second object to the server, receiving a fourth questionnaire corresponding to the second object from the server, displaying the fourth questionnaire on the second HMD, An object-based survey unit for acquiring a response of the second user to the fourth questionnaire using a recognition unit and transmitting a response to the obtained questionnaire to the server;
An input window for receiving the selection input by the second user to select the second indicator using the second user action recognition unit and for inputting the opinion of the second user to the second object, Capturing an image of at least a part of the indicated second object in accordance with an operation of the second user indicating at least a part of the second object, A user opinion examination unit for transmitting the user's opinion and the opinion of the second user to the server;
Receiving feedback from the server on the opinion of the second user, displaying a fifth questionnaire corresponding to the feedback on the second HMD, and using the second user action recognition unit to receive the feedback for the fifth questionnaire A feedback processor for acquiring a response of the second user and transmitting a response to the obtained questionnaire to the server; And
Receiving a reward according to a response to the fourth questionnaire or the fifth questionnaire from the server, wherein the reward includes a coupon capable of purchasing a predetermined commodity at the store, and transmitting the received reward to the second HMD A reward processing unit for acquiring a response of the second user as to whether or not the coupon is used by using the second user action recognition unit and transmitting the use of the coupon to the server; A user terminal; And
A response delivery unit for delivering, to the provider terminal, a response of the second user to the use of the third questionnaire, the fourth questionnaire, the fifth questionnaire, and the coupon received from the user terminal; And
The indoor environment of the store is controlled according to the response of the second user to the indoor environment of the store, and the indoor environment of the store is controlled by the temperature, brightness and background music of the store An indoor environment control unit including at least one of: The server; A virtual reality survey system. The method according to claim 1,
The second object is a logo displayed on a product purchased by the second user at the store or a product purchased by the second user at the store,
Wherein the second object recognizer comprises:
The second object receiving the payment information in which the second user has purchased the second object, acquiring the kind of goods corresponding to the payment information, and if the recognized second object corresponds to the kind of the obtained goods, And if the second object corresponds to the brand logo of the store, displays the second indicator. 3. The method of claim 2,
Wherein the second object recognizer comprises:
If there is a questionnaire matched with respect to the second object, a marker indicating that a questionnaire matched to the second object exists is displayed together with the second indicator, and when a questionnaire matched to the second object is answered And displays information about the awarded reward with the second indicator. The method of claim 3,
Wherein the second user action recognition unit comprises:
Recognizing the operation of the second user to select the second object or the second indicator with a finger, recognizing the shape of the finger of the second user selecting the second object or the second indicator, If the shape of the finger of the second user is the first form, the object-based questionnaire is called to display the questionnaire matched to the second object, and if the shape of the finger of the second user is the second form, Invite a reviewer to receive input of a comment on the second object of the second user. 5. The method of claim 4,
The object-
If the answer to the fourth questionnaire is negative, an input window for inputting a cause of the negative response is displayed to receive the cause of the negative response, extract at least one keyword from the input cause, A third object corresponding to the extracted keyword is recognized from the image acquired by the image acquisition unit, an image of the third object is captured, and an image of the captured third object and the cause of the input are transmitted to the server , A virtual reality survey system. 6. The method of claim 5,
The user terminal comprises:
An emotion state determiner that can determine the emotion state of the second user by acquiring body information of the second user; Further comprising:
The object-
If the response to the fourth questionnaire is negative, transmits the negative response and the emotional state of the second user determined by the emotional state determiner to the server,
The user-
And transmitting the second user's opinion and the emotional state of the second user determined by the emotional state determiner to the server when a negative keyword is included in the input of the second user,
Wherein the body information comprises at least one of breathing, pulse and body temperature. The method according to claim 6,
The server comprises:
When the contents of the defect of the goods corresponding to the second object are included in the opinion of the second user received from the user opinion examination unit of the user terminal, A feedback management unit for transmitting the fifth questionnaire including the questionnaire to the user terminal; Further comprising:
Wherein the response transfer unit comprises:
When a response requesting exchange or refund of a commodity corresponding to the second object is received from the feedback processing unit of the user terminal, information requesting exchange or refund of the commodity corresponding to the second object, And transmits information on the use request of the coupon to the ordering system of the store when a response requesting use of the coupon is received from the reward processing unit of the user terminal. 8. The method of claim 7,
The server comprises:
Storing and managing at least one of a position of seats disposed in the store, use of each of the seats, a location of goods displayed on the store, and an inventory of products displayed on the store, A store manager for transmitting to the user terminal the information that guides the second user to a product displayed on the store or to an empty seat arranged in the store according to a visit purpose of the second user; Further comprising a virtual reality survey system. 9. The method of claim 8,
The indoor-
The indoor environment of the store is adjusted according to a response of one or more users to the indoor environment of the store, and when the responses of the one or more users are different, ≪ / RTI &
The user terminal comprises:
When the second user leaves the store by using the position information of the second HMD obtained by the second position information obtaining unit, transmits information to the server that the second user leaves the store and,
The indoor-
Wherein when the second user receives from the user terminal information indicating that the second user leaves the store, a response from the one or more users to the indoor environment of the store, excluding the response of the second user to the indoor environment of the store, Wherein the indoor environment of the store is adjusted using an average of the virtual reality survey system. 10. The method of claim 9,
The object-
Receiving a sixth questionnaire corresponding to the recognized second object from the server, wherein the sixth questionnaire includes at least one learning question associated with the second object, and wherein the at least one learning question associated with the second questionnaire Receiving the answer of one learning question, displaying the received sixth questionnaire on the second HMD, acquiring the response of the second user to the sixth questionnaire using the second user action recognition unit,
Wherein the feedback processing unit comprises:
If the answer of the second user is an incorrect answer, receiving from the server a seventh questionnaire for explaining the learning question and further learning, displaying the questionnaire on the second HMD,
The reward processing unit,
And if the answer of the second user is correct, receives a reward according to the correct answer from the server and displays it on the second HMD.

Images