KR20230012196A - Master device, slave device and control method for connecting virtual space and real space - Google Patents

Abstract

A server is disclosed. A server for providing content to a user terminal device for providing a virtual reality service includes: a communication part that communicates with at least one source instrument and the user terminal device; and a processor that receives content captured in real time from the source instrument at a preset location based on location information received from the at least one source instrument and providing the same to the user terminal device when a content transmission request for the preset location is received from the user terminal device.

Description

Virtual space and real space connection service server, user terminal device and its control method {MASTER DEVICE, SLAVE DEVICE AND CONTROL METHOD FOR CONNECTING VIRTUAL SPACE AND REAL SPACE}

The present invention relates to a virtual space and real space connection service server, a user terminal device, and a control method thereof, and more particularly, to a server providing a VR service, a user terminal device, and a control method thereof.

In general, VR (Virtual Reality) technology provides all senses (sight, hearing, smell, taste, touch) of the human body in a three-dimensional virtual environment similar to the real environment created through computer graphics (CG) technology. It refers to technology that ultimately aims to maximize the utilization of information by reproducing the virtual space in three dimensions to the participant and immersing the human in the virtual space by allowing them to be immersed in the virtual world created by this interaction.

For example, if you put on a VR headset, a ski slope covered with white snow appears in front of you. Even if you turn your head from side to side, up and down, you are in the middle of the ski resort. The snowboard on my feet starts to slide, and suddenly a cliff appears. For a moment, the air beneath your feet is empty. My head is aware that this is virtual reality, but the sense of presence is so great that the virtual ski field in the headset feels like reality.

As such, VR services provide new experiences to users, but there is a problem in that they are not commercially available in relation to content purchase and consumption.

The present invention is in accordance with the above-described necessity, and an object of the present invention is to provide a virtual space and real space connection service server, a user terminal device, and a control method that can provide a virtual experience service by linking a plurality of devices. there is.

A server providing content to a user terminal device providing a virtual reality service according to an embodiment of the present invention for achieving the above object communicates with at least one source device and the user terminal device. and, when a content transmission request for a preset location is received from the user terminal device, content captured in real time from the source device at the preset location is received based on the location information received from the at least one source device. and a processor for transmitting to the user terminal device.

In addition, the processor may sequentially transmit a plurality of captured contents received from a plurality of source devices to the user terminal device based on location information received from the plurality of source devices, or may transmit a plurality of captured contents received from the plurality of source devices. Content may be synthesized based on location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, the processor selects content corresponding to the permission information from among the content received from the source device based on permission information for setting a content sharing level that the at least one source device can share with the user terminal device. It can be transmitted to the user terminal device.

In addition, the processor may process the content received from the source device according to a preset condition set in the user terminal device and transmit the content to the user terminal device.

In addition, when the content received from the source device is 3D content, the processor projects the left eye image and the right eye image included in the 3D content to a virtual stereoscopic space, respectively, to generate a left eye stereoscopic space image and a right eye stereoscopic space image and when information corresponding to the motion state of the user terminal device is received from the user terminal device, viewpoint regions corresponding to the motion state of the user terminal device are extracted from the generated left and right eye 3D space images, and the user terminal device can be sent to the device.

In addition, a user terminal device providing a virtual reality service transmits information related to the corresponding location to the server when a user command specifying a predetermined location is input to a communication unit that communicates with a display and a server. and a processor receiving real-time photographed content related to the location from the server and displaying the content through the display.

Also, the processor may receive only content selected from the server based on authority information for setting a content sharing level capable of being shared by at least one source device providing the real-time captured content with the user terminal device.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor transmits information corresponding to a preset path connecting the first point and the second point to the server, , It is possible to receive and display a plurality of contents related to the preset path from the server.

In addition, when a user command for setting a first point and a second point is input on the UI screen, the processor transmits information corresponding to a preset path connecting the first point and the second point to the server, , It is possible to receive and display synthesized content synthesized from a plurality of contents related to the preset path from the server.

In addition, when a user command for setting a preset region is input, the processor transmits information related to the corresponding region to the server, and receives location information about at least one source device located within the preset region from the server. It can be received and controlled to display a GUI indicating the location of the at least one source device.

In addition, the processor controls to receive, from the server, permission information on a content sharing level that the at least one source device can share with the user terminal device, and display the information in an area adjacent to a GUI indicating the location of the source device. can

In addition, the processor transmits information related to the location to the server when a user command specifying the preset location is input on the UI screen providing the virtual experience service, and the virtual experience captured at the location is transmitted from the server. Content can be received and displayed.

Also, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

Meanwhile, a control method of a server providing content to a user terminal device providing a virtual reality service according to an embodiment of the present invention includes the steps of receiving a content transmission request for a preset location from the user terminal device. And, based on the location information received from at least one source device, receiving the content captured in real time from the source device at the preset location and providing it to the user terminal device in a streaming form.

In addition, the providing in the form of streaming to the user terminal device may include sequentially transmitting a plurality of captured contents received from a plurality of source devices to the user terminal device based on location information received from the plurality of source devices; The captured content received from the plurality of source devices may be synthesized based on location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, the providing of the content to the user terminal device in a streaming form may include the content received from the source device based on authority information for setting a content sharing level capable of being shared with the user terminal device by the at least one source device. Content corresponding to authority information may be selected and transmitted to the user terminal device.

In the step of providing the content to the user terminal device in a streaming form, the content received from the source device may be processed according to a preset condition set in the user terminal device and transmitted to the user terminal device.

In addition, in a control method of a user terminal device providing a virtual reality service according to an embodiment of the present invention, when a user command specifying a preset location is input, information related to the location is transmitted to the server. and receiving, from the server, real-time content related to the location and displaying the content through the display.

In addition, the displaying may include receiving and displaying only content selected from the server based on authority information in which at least one source device providing the real-time captured content sets a content sharing level capable of being shared with the user terminal device. can do.

In addition, the transmitting to the server may include, when a user command for setting the first point and the second point is input on the UI screen, information corresponding to a preset path connecting the first point and the second point is sent to the server. The transmitting to the server and the displaying may include receiving and displaying a plurality of contents related to the predetermined path from the server, or receiving and displaying synthesized contents obtained by synthesizing a plurality of contents related to the predetermined path from the server. can do.

As described above, according to various embodiments of the present invention, it is possible to virtually experience a sense of reality in a remote place using information obtained through another person's wearable device, so that various experiences can be provided to the user.

1 is a diagram illustrating a display device providing a virtual reality service according to an embodiment of the present invention.
2 is a diagram for explaining a content providing system according to an embodiment of the present invention.
3A and 3B are block diagrams showing the configuration of a server according to an embodiment of the present invention.
4A and 4B are block diagrams showing the configuration of a user terminal device according to an embodiment of the present invention.
5 is a diagram for explaining a method of generating and outputting panoramic VR content according to an embodiment of the present invention.
6A and 6B are diagrams for explaining a method of streaming content in a server according to an embodiment of the present invention.
7 is a flowchart for explaining the operation of a source device according to an embodiment of the present invention.
8 is a sequence diagram for explaining the operation of a server according to an embodiment of the present invention.
9 is a sequence diagram for explaining the operation of a user terminal device according to an embodiment of the present invention.
10 to 12 are diagrams for explaining a virtual experience service providing method according to various embodiments of the present disclosure.
11 is a diagram for explaining a method of providing a virtual tour service according to an embodiment of the present invention.
12 is a diagram for explaining a method of providing a virtual event experience service according to an embodiment of the present invention.
13 is a flowchart illustrating a method of controlling a server according to an embodiment of the present invention.
14 and 15 are flowcharts for explaining a control method of a user terminal device according to an embodiment of the present invention.
16 to 18 are diagrams illustrating a usage action between a user of a first space and a user of a second space, which is an offline space, according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a diagram illustrating a display device providing a virtual reality service according to an embodiment of the present invention.

According to FIG. 1 , a user terminal device 100 according to an embodiment of the present invention may be worn on a user's head and function as a head mounted display (HMD). For example, the user terminal device 100 is implemented as a smart phone or a tablet and is attached to the front of a main body (eg, housing) having a shape of glasses, a headset, or a helmet to provide an image to both eyes of the user. there is.

For example, when the user terminal device 100 implemented as a smart phone is attached to the front of the headset body, the display provided on the smart phone is located close to both eyes of the user. Accordingly, the user terminal device 100 can display an image through the display and provide the image in front of the user's eyes. In this case, on the rear surface of the headset body, temples or bends for glasses to be worn on the user's head may be formed. In addition, a track pad for operation, a return button, a volume control key, and the like may be mounted on the headset body.

Alternatively, the user terminal device 100 may be implemented as a device having a display itself and having a shape that can be worn on the user's head. In this case, the display may be fixedly installed rather than detachable.

Meanwhile, when the user wears the user terminal device 100 on the head, the area viewed by the user's left eye and the area viewed by the right eye on the display may be spatially separated. Accordingly, the user terminal device 100 may display different images in an area viewed by the user's left eye and an area viewed by the user's right eye on the display, so that different images are incident on the left and right eyes.

Meanwhile, the user terminal device 100 is implemented to immediately update a visual field image by tracking a user's head movement, and may be implemented to provide a 3D image as well as a 2D image. For example, when the user wears the user terminal device 100 on the head, it completely dominates the user's field of view and provides 360-degree stereoscopic images and sounds, and the gyro sensor and acceleration sensor provided in the user terminal device 100 It can be implemented to detect when the user moves his or her head up and down or left and right, and produce visual effects suitable for the direction.

Accordingly, the user can be provided with a 3D image that exists in the direction in which the user's gaze is directed among the panoramic 3D image, so that the user can experience virtual reality (VR).

2 is a diagram for explaining a content providing system according to an embodiment of the present invention.

As shown in FIG. 2, the content providing system according to an embodiment of the present invention includes a user terminal device 100, a server 200, and at least one source device 300-1, 300-2, 300-3 includes

The user terminal device 100 communicates with the server 200 to receive content transmitted to the server 200 from various types of source devices 200-1, 200-2, and 200-3 through real-time streaming. there is. In this case, the source devices 300-1, 300-2, and 300-3 may be implemented as wearable devices such as smart glasses and smart watches capable of taking pictures, but smart phones, camcorders, cameras, etc. Any device having a photographing function is not limited thereto and is applicable.

Accordingly, the user terminal device 100 can receive an image captured in real time from a remote location through the server 200 and provide a virtual experience service for the corresponding place to the user. Hereinafter, the present invention will be described with reference to the drawings. Various embodiments of will be described in detail.

3A and 3B are block diagrams showing the configuration of a server according to an embodiment of the present invention.

According to FIG. 3A , the server 200 according to an embodiment of the present invention includes a communication unit 210 and a processor 220.

Here, the server 200 may be implemented as a central server (or integration server) responsible for interactions between various operating systems and applications in all network systems, or a cloud server using cloud computing technology. Here, cloud computing refers to Internet-based computing technology, and is a web-based software service in which a program is placed in a utility data server on the Internet and then called and used on a computer or mobile phone.

The communication unit 210 communicates with the user terminal device 100 and the source devices 300-1, 300-2, and 300-3.

The communication unit 210 may be implemented as a wireless communication module that is connected to an external network and performs communication according to a wireless communication protocol such as Wi-Fi or IEEE. In addition, the wireless communication module is a user terminal device 100 and source devices 300-1 and 300-2 through various communication schemes such as 3rd generation (3G), 3rd generation partnership project (3GPP), and long term evolution (LTE). , 300-3) and communication can be performed. Here, the communication unit 210 may communicate with the user terminal device 100 and the source devices 300-1, 300-2, and 300-3 through one communication module, but communicate through a separate communication module. It may be implemented to perform. Here, the separate communication module may be implemented in a form supporting the same communication method, but may be implemented in a form supporting different communication methods.

For example, when a preset event occurs, the communication unit 110 communicates with the display device 100 and the source devices 300-1, 300-2, and 300-3 according to a predefined communication method to interlock. state can be Here, interworking is an operation of initializing communication between the server 200 and the display device 100 and between the server 200 and the source devices 300-1, 300-2, and 300-3, an operation of forming a network, and pairing. This may mean any state in which communication is possible, such as an operation being performed. For example, communication between the server 200 and the display device 100 may be performed according to an event of driving an application providing a virtual experience service according to the present invention on the display device 100 .

The communication unit 210 receives images captured in real time from the source devices 300-1, 300-2, and 300-3 under the control of the processor 220 described below, and transmits the received images to the display device 100. can stream

The processor 220 controls the overall operation of the server 200.

In particular, when a content transmission request for a preset location is received from the user terminal device 100, the processor 220 transmits location information received from at least one source device 300-1, 300-2, or 300-3. Based on this, content photographed from a source device at a preset location may be received in real time and provided to the user terminal device 100 in a streaming form. Here, at least one of the source devices 300-1, 300-2, and 300-3 is implemented as a wearable device such as a smart glass or a smart watch, as described above, to provide a place for the corresponding source device. can be filmed Accordingly, the user of the user terminal device 100 experiences the current environment of the corresponding place in real time through the images captured by the source devices 300-1, 300-2, and 300-3, even without directly visiting the corresponding place. You can do it.

Meanwhile, the content received from the source devices 300-1, 300-2, and 300-3 may be general 2D video, but is not limited thereto, and includes 3D video including left and right eye video for providing VR service. Images, panoramic VR content, general 2D images, and the like may also be included therein.

In addition, the processor 220, when receiving a content transmission request for the changed location from the user terminal device 100, takes pictures in real time from the source devices 300-1, 300-2, and 300-3 corresponding to the changed location information. The content may be received in real time and streamed to the user terminal device 100 . For example, when a user command for selecting another location is input from the user terminal device 100, a content transmission request for the corresponding location may be transmitted to the server 200.

In this case, when a request for a virtual experience service for a specific place is received from the user terminal device 100, the processor 220 operates the source devices 300-1, 300-2, and 300-3 based on map information of the corresponding place. Information indicating the location of is transmitted to the user terminal device 100, and the user of the user terminal device 100 selects a specific source device based on the information displayed on the screen through a command to select a source device to which the captured image is to be shared. A content transmission request for a location may be transmitted to the server 200 . However, in some cases, it is also possible to display the location of a user of a source device capable of sharing an image within the current gaze range by applying AR technology to the currently displayed image. In this case, the source device Distance information to the user may be additionally displayed.

In addition, the processor 220 transmits a plurality of captured contents received from the plurality of source devices 300-1, 300-2, and 300-3 from the plurality of source devices 300-1, 300-2, and 300-3. Based on the received location information, it is sequentially transmitted to the user terminal device 100 or a plurality of captured contents received from the plurality of source devices 300-1, 300-2, and 300-3 are transmitted to the plurality of source devices 300. -1, 300-2, 300-3) may be synthesized based on the location information received, and the synthesized content may be transmitted to the user terminal device 100. For example, when the captured content is transmitted to the user terminal device 100 on a path via location a → location b → location c, a plurality of source devices 300- 1, 300-2, 300-3) are sequentially provided to the user terminal device 100, or a plurality of source devices 300-1, 300 corresponding to positions a, b, and c, respectively -2, 300-3) may configure a synthesized image in which the captured images are sequentially connected and transmit it to the user terminal device 100.

In this case, the processor 220 is based on content information that varies depending on the resolution information, display arrangement state, screen size, etc. of the plurality of source devices 300-1, 300-2, and 300-3 and the user terminal device 100. The content received from the plurality of source devices 300-1, 300-2, and 300-3 may be processed to be suitable for the user terminal device 100 and transmitted to the user terminal device 100. However, in some cases, it may be possible for the user terminal device 100 to process and display the received content to conform to the display format.

In addition, the processor 220 processes the plurality of captured contents received from the plurality of source devices 300-1, 300-2, and 300-3 to conform to the display format of the user terminal device 100, and processes the processed contents. may be synthesized and transmitted to the user terminal device 100 .

Further, the processor 220 transmits only at least some contents corresponding to the authority information to the user terminal device 100 based on authority information for at least one source device 300-1, 300-2, and 300-3. can Here, the authority information refers to information in which a user of at least one source device 300-1, 300-2, or 300-3 sets a sharing level of content that can be shared with other terminal devices. Also, when a plurality of contents are synthesized and provided, the processor 220 may synthesize at least some contents corresponding to authority information and transmit the synthesized contents to the user terminal device 100 .

That is, the user of at least one source device (300-1, 300-2, 300-3) can set a content sharing level such as audio data, video data, location information, message reception or not, and the processor 220 Among the contents received from at least one of the source devices 300-1, 300-2, and 300-3, only content that can be shared according to authority information may be selected and transmitted to the user terminal device 100.

For example, when the user of at least one source device (300-1, 300-2, 300-3) sets permission information to share only video data and not share audio data, only video data is shared with the user terminal device. (100). As another example, when a user of at least one source device 300-1, 300-2, or 300-3 sets authority information to allow message reception from another terminal device, the user of the user terminal device 100 at least It is also possible to transmit a message to one source device (300-1, 300-2, 300-3).

On the other hand, in the above-described embodiment, the server 200 determines authority information and selects content that can be shared based on authority information among contents received from at least one source device 300-1, 300-2, or 300-3. and transmitted to the user terminal device 100, but according to another embodiment, at least one source device (300-1, 300-2, 300-3) transmits only the contents corresponding to the authority information to the server 200. It is also possible to transmit. For example, when the user of at least one source device (300-1, 300-2, 300-3) sets permission information to share only video data and not share audio data, the server 200 Only video data may be received from at least one source device (300-1, 300-2, 300-3).

In addition, the processor 220 may process the content received from at least one source device 300-1, 300-2, or 300-3 and transmit it to the user terminal device 100.

In this case, the processor 220 processes the content received from at least one of the source devices 300-1, 300-2, and 300-3 according to preset conditions set in the user terminal device 100, and the user terminal device ( 100) can be transmitted.

For example, when the user of the user terminal device 100 sets to receive a virtual experience service in the form, the processor 220 receives information from at least one source device 300-1, 300-2, and 300-3. The processed content may be processed into a panoramic content form and transmitted to the user terminal device 100 .

Specifically, the processor 220 projects the left eye image and the right eye image included in the 3D content received from the source devices 300-1, 300-2, and 300-3 onto a virtual 3D space, respectively, to obtain a left eye 3D space image and A right eye stereoscopic image may be generated. Then, when information corresponding to the motion state of the user terminal device 100 is received from the user terminal device 100, the processor 120 determines the motion state of the user terminal device 100 in the generated left and right eye stereoscopic space images. A viewpoint area corresponding to may be extracted and transmitted to the user terminal device 100 .

In this case, the processor 220 may determine a viewpoint (or direction) area corresponding to the motion state of the user terminal device 100 from among the input 3D images based on the motion information received from the user terminal device 100 . Thereafter, the processor 220 may transmit the image of the determined viewpoint area to the user terminal device 100 . Here, the motion information may include real-time direction information and angle information for all directions of the user terminal device 100, but is not limited thereto, and may include movement speed information and rotation information. In addition, after initial direction and angle information is received from the user terminal device 100, direction and angle information moved based on previously received direction and angle information may be received, but is not limited thereto.

In this case, the processor 220 may transmit an image corresponding to the user terminal device 100 to the corresponding user terminal device 100 based on device information received from the user terminal device 100, for example, a device ID. . In addition, the processor 220 secures a region of the image buffer as a transmission buffer based on the device display information received from the user terminal device 100, and the image corresponding to the motion information of the user terminal device 100 is stored in the transmission buffer. may be buffered and transmitted to the corresponding user terminal device 100 . In this case, the processor 220 may separately secure a transmission buffer corresponding to each user terminal device 100 .

In addition, the processor 220 decompresses (or decodes) the input image, stores it in an image buffer, extracts a region corresponding to the motion information of the user terminal device 100 from the image stored in the image buffer, and converts the extracted image to It may be compressed (or encoded) and transmitted to the user terminal device 100 .

Alternatively, the processor 220 may extract and decode a region corresponding to the motion information of the user terminal device 100 and transmit the decoded region to the user terminal device 100. Among the regions corresponding to the motion information, a margin other than the previously transmitted video may be used. It is also possible to decode and transmit only the region.

Alternatively, the processor 220 determines based on the motion information of the user terminal device 100 together with the main video decoded based on the content received from the source devices 300-1, 300-2, and 300-3. The margin image for the region may be transmitted to the user terminal device 100 as a low-quality image or an encoded image using a codec. Here, the margin image is an image excluding the decoded main image among regions determined based on the motion information of the user terminal device 100.

It can be. In this case, the user terminal device 100 can minimize delay due to video transmission by decoding the received margin video and synthesizing the received main video and outputting the result.

In addition, the processor 220 predicts future motion information of the slave devices 200-1, 200-2, and 200-3 based on the currently received motion information of the user terminal device 100, and returns information to the predicted motion information. Based on the input image, an image of a viewpoint area corresponding to a future motion state of the user terminal device 100 may be transmitted to the user terminal device 100 . For example, the processor 220 may track and predict a future motion state of the user terminal device 100 using a filtering method such as a conventional Kalman filter.

In addition, the processor 220 groups the plurality of user terminal devices into one group when the motion information received from the plurality of user terminal devices 100 falls within a preset threshold range, and the user terminal devices belonging to the same group ( 100), an image of the same viewpoint area can be transmitted. For example, the processor 220 groups the first and second user terminal devices having motion information between 10° and 15° in the right direction into one group, and outputs an image of the same view area corresponding to the corresponding motion information. It can be transmitted to the first and second user terminal devices.

Figure 3b is a block diagram showing a specific configuration of the server shown in Figure 3a. According to FIG. 3B, the server 200' includes a communication unit 210, a processor 220, and a storage unit 230. Among the components shown in FIG. 3B, detailed descriptions of components overlapping with those shown in FIG. 3A will be omitted.

The storage unit 230 stores various data such as an O/S (Operating System) software module for driving the server 200 and various information required to perform functions of the server 200 . For example, the storage unit 230 may store authority information for each of the source devices 300-1, 300-2, and 300-3.

Meanwhile, the above-described operation of the processor 220 may be performed by a program stored in the storage unit 230 .

Specifically, the processor 220 includes a RAM 221, a ROM 222, a main CPU 223, a graphic processing unit 224, first to n interfaces 225-1 to 225-n, and a bus 226. include

The RAM 221 , the ROM 222 , the main CPU 223 , the graphics processing unit 224 , the first to n interfaces 225 - 1 to 225 - n may be connected to each other through a bus 226 .

The first to n interfaces 225-1 to 225-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device through a network.

The main CPU 223 accesses the storage unit 230 and performs booting using the O/S stored in the storage unit 230 . In addition, various operations are performed using various programs, contents, data, etc. stored in the storage unit 230 . For example, the main CPU 223 determines the source device at the desired location of the user terminal device 100 using various modules stored in the storage unit 230, and transmits the captured image received from the corresponding source device to the user terminal device. (100) can be provided.

The ROM 222 stores a command set for system booting and the like. When a turn-on command is input and power is supplied, the main CPU 223 copies the O/S stored in the storage unit 230 to the RAM 221 according to the command stored in the ROM 222 and executes the O/S. boot up the system When booting is completed, the main CPU 223 copies various application programs stored in the storage unit 230 to the RAM 221 and executes the copied application programs in the RAM 221 to perform various operations.

The graphic processing unit 224 uses a calculation unit (not shown) and a rendering unit (not shown) to create a screen including various objects such as icons, images, and text. The calculation unit (not shown) calculates attribute values such as coordinate values, shape, size, color, and the like of each object to be displayed according to the layout of the screen based on the received control command. The rendering unit (not shown) creates screens of various layouts including objects based on the attribute values calculated by the calculation unit (not shown).

Meanwhile, the processor 220 may provide content to the user terminal device 100 by using pre-stored data in some cases.

In detail, the processor 220 may select and provide an image according to the request of the user terminal device 100 from a database in which captured images are previously stored, or may synthesize and provide the selected images. For example, when a user of the user terminal device 100 sets a start point and an end point in a virtual space, an optimal route is created, and image data from the database to the start and end are selected according to the created route. , Landscape images may be acquired one by one from the starting point and transmitted to the user terminal device 100 . In addition, the virtual space is set by mapping the image obtained from the database to the virtual space, and a virtual camera is created in the virtual space through the VR engine, so that the image of the virtual space projected by the virtual camera is transferred to the user terminal device 100. can provide In this case, the position of the camera in the virtual space is determined according to the generated path, and the direction of the camera may be determined from the direction of the HMD.

In addition, the processor 220 may provide various modes for each virtual experience service. For example, in the case of a virtual wayfinding experience service, a car mode, a walking mode, a running mode, etc. may be provided, and audio feedback, haptic feedback, etc. may be provided according to user selection. The car mode is a mode that provides a virtual road experience while moving in the direction of the route within the landscape data obtained from the database. In this movement mode, the short distance moves from the current 3D space image, and if long distance movement is required, the current 3D space image and the 3D space image can be combined and provided when moving if there is a connected input image or a connected 3D space image. there is. In this case, corresponding audio/haptic information may be obtained from a database based on meta data generated or stored through image analysis and provided to the user terminal device 100 . Here, examples of meta data include natural sounds such as birdsong, cicada sound, and rain sound, artificial sound data such as car sound, train sound, and airplane sound, haptic data having a profile of various cycles and intensities, and video screen brightness control. data, etc.

Meanwhile, the walking mode and the running mode are basically similar modes, but there may be a difference in moving speed within the same landscape and a difference in the position of the virtual camera in the virtual space. In this case, when a path is selected by a user input and a walking mode and a running mode are selected, the corresponding path is created and a pulse wave (Ripple) is generated in a vertical direction of the generated path. Here, the period and amplitude of the pulse wave vary according to the walking mode and the running mode, and may have a predetermined size. If the image obtained from the database is a full 3D image, the image may be provided through a change in camera position/direction according to the pulse wave and a virtual VR engine. Alternatively, if the image obtained from the database is a binocular parallax image, depth information may be extracted to generate a virtual viewpoint image, and information of a region exposed may be virtually generated based on the extracted depth information. Alternatively, if the image obtained from the database is a monocular image, the image may be provided based only on a change in the position/direction of the virtual camera. Meanwhile, a zoom function may be provided according to user selection while moving according to each mode. Through the zoom function, the user can view the video by enlarging the landscape, and when the head motion is changed in the zoom state, for example, the amount of change in the head motion is greater than a certain value compared to the zoom function start standard, or the user's zoom mode release input If it comes in, you can end the zoom function.

4A and 4B are block diagrams showing the configuration of a user terminal device according to an embodiment of the present invention.

According to FIG. 4A , the user terminal device 100 according to an embodiment of the present invention includes a display 110, a communication unit 120, and a processor 130.

The display 110 displays various information.

Here, the display 110 may be implemented as a Liquid Crystal Display Panel (LCD), Organic Light Emitting Diodes (OLED), or the like, but is not limited thereto. Also, the display 110 may be implemented as a flexible display, a transparent display, or the like, depending on circumstances. On the other hand, as mentioned in FIG. 1, when a smart phone is connected and used as a monitor, the display 110 can be implemented as a display of the smart phone, of course.

The communication unit 120 communicates with the server 200 . The communication unit 210 may be implemented as a wireless communication module that is connected to an external network and performs communication according to a wireless communication protocol such as Wi-Fi or IEEE. In addition, the wireless communication module may perform communication with the server 200 through various communication schemes such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE).

The processor 130 controls overall operations of the user terminal device 100 .

In particular, when a user command specifying a preset location is input, the processor 130 transmits information related to the location to the server 200, receives real-time content related to the location from the server 200, and displays ( 110) to display.

Here, the user command for specifying a preset location may be a user command for selecting a specific location, a user command for selecting an event taking place at a specific location, and the like.

In addition, the processor 130 receives only content selected from the server 200 based on permission information for setting a content sharing level that allows at least one source device providing real-time content to be shared with the user terminal device 100. can do.

In addition, when a user command for setting the first and second points is input on the UI screen, the processor 130 transmits information corresponding to a preset path connecting the first and second points to the server 200. In addition, a plurality of contents related to a predetermined path may be received from the server 200 and displayed on the screen.

In addition, when a user command for setting the first and second points is input on the UI screen, the processor 130 transmits information corresponding to a preset path connecting the first and second points to the server 200. In addition, synthesized content obtained by synthesizing a plurality of contents related to a preset path may be received from the server 200 and displayed on the screen.

In addition, when a user command for setting a preset region is input, the processor 130 transmits information related to the region to the server 200, and transmits information related to the region to the server 200 to at least one source device located within the preset region. A GUI indicating the location of at least one source device may be displayed on the screen by receiving location information about the source device.

In addition, the processor 130 receives permission information on a content sharing level that at least one source device can share with the user terminal device 100 from the server 200, and displays it in an adjacent area of the GUI indicating the location of the source device. can do.

In addition, the processor 130 transmits information related to the location to the server 200 when a user command specifying a preset location is input on the UI screen providing the virtual experience service, and the server 200 provides the location. The captured virtual experience content may be received and displayed. Here, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

Meanwhile, the processor 130 may zoom-in or zoom-out the displayed content according to the user's zoom-in/out command for enlarging or reducing the image.

In detail, when the user terminal device 100 is implemented as an HMD, the display 110 displays a left eye image in a display area viewed by the user's left eye and a right eye image in a display area viewed by the user's right eye to provide information to the user. 3D images can be provided. Here, the left eye image and the right eye image may be implemented as a panoramic 3D image.

In this case, the processor 130 may generate a left eye 3D space image and a right eye 3D space image by projecting each of the left eye image and the right eye image constituting the 3D image onto a virtual 3D space. Here, the generated 3D space image may be a panoramic 3D image. That is, the processor 130 projects each of the left eye image and the right eye image onto a virtual spherical space, so that the left eye image and the right eye image respectively exist in a 360° direction with respect to the center of the virtual spherical space, and the left eye stereoscopic space image and the right eye image exist in a direction of 360°. A three-dimensional space image can be created.

Also, the processor 130 may extract an image located in a direction in which the user wearing the HMD looks from among the stereoscopic space images, and display the extracted image through the display 110 . Specifically, the processor 130 determines the direction of the user's gaze using a sensor (not shown) provided in the display device 100, and uses a virtual camera located at the center of each of the left eye stereoscopic space image and the right eye stereoscopic space image. Accordingly, an image existing in an area toward which the user's gaze is directed can be extracted and displayed from each of the left eye stereoscopic space image and the right eye stereoscopic space image. To this end, the processor 130 detects the moving direction of the user's head wearing the display device 100 using at least one of a gyro sensor, an acceleration sensor, and a geomagnetic sensor provided in the display device 100 so that the user can You can judge the direction you are looking at.

Further, the processor 130 positions a virtual camera having a constant field of view (FOV) at the center of each of the left eye stereoscopic space image and the right eye stereoscopic space image, and rotates the virtual camera in the direction of the user's line of sight to rotate the left eye stereoscopic image. An image existing in the direction of the user's gaze may be extracted from each of the space image and the right eye stereoscopic space image and displayed. For example, if it is determined that the user is looking in the 60° direction to the left from the front, the processor 130 sets the virtual camera located at the center of each of the left eye stereoscopic space image and the right eye stereoscopic space image to the left 60° direction. By rotating to , an image existing in the left 60° direction can be extracted from each of the left eye stereoscopic space image and the right eye stereoscopic space image.

In this case, the processor 130 displays the image extracted from the left-eye stereoscopic space image on the area viewed by the user's left eye on the display 110 wearing the display device 100, and displays the image extracted from the right-eye stereoscopic space image. The right eye of the user wearing the display device 100 may display the display 110 in an area viewed from the display 110 . Accordingly, the user can be provided with a 3D image of a direction in which the user's gaze is directed among the panoramic 3D image.

Meanwhile, when a user command for enlarging or reducing a 3D image is input, the processor 130 adjusts the 3D effect of the 3D image to correspond to the user command, and displays the enlarged or reduced image with the adjusted 3D effect. can control. Here, the enlargement command includes a full enlargement command for enlarging the entire 3D image and a partial enlargement command for enlarging a part of the 3D image, and the reduction command includes a full reduction command for reducing the entire 3D image and a part of the 3D image. Contains a partial reduction command for reducing .

The processor 130 moves each of the left eye image and the right eye image included in the stereoscopic image, and projects the moved left eye image and right eye image on a virtual stereoscopic space image to generate a left eye stereoscopic space image and a right eye stereoscopic space image, An image corresponding to a user command may be extracted from each of the left eye stereoscopic space image and the right eye stereoscopic space image and displayed. That is, the processor 130 moves each of the left eye image and the right eye image to correspond to a user command before projecting the left eye image and the right eye image into the virtual 3D space, and then displays each of the moved left eye image and right eye image in the virtual 3D space. A left-eye stereoscopic space image and a right-eye stereoscopic space image may be generated by projecting onto the image.

Specifically, when an overall enlargement command for enlarging the entire 3D image is input, the processor 130 moves the left eye image to the left and the right eye image to the right, and moves the left eye image moved to the left and the right eye moved to the right. A left-eye 3D space image and a right-eye 3D space image may be generated using each of the images, and an image corresponding to an overall enlargement command may be extracted from each of the left eye 3D space image and the right eye 3D space image and displayed. That is, the processor 130 generates a left-eye stereoscopic space image and a right-eye stereoscopic space image by moving each of the left eye image and the right eye image to the right when an entire enlargement command is input, and magnifies the left eye stereoscopic space image and the right eye stereoscopic space image, respectively. Images can be extracted and displayed.

Specifically, the processor 130 moves the left eye image to the left, and generates a left eye stereoscopic space image using the left eye image moved to the left. Further, the processor 130 reduces the amount of the angle of view of the virtual camera existing in the left-eye stereoscopic space image by an enlargement ratio for the entire zoom-in command, and enlarges the entire image in the left-eye stereoscopic space image by using the virtual camera having the reduced amount of the field-of-view. An enlarged image corresponding to the command may be extracted. In this case, the processor 130 only adjusts the angle of view of the virtual camera and does not change the focal position of the virtual camera. That is, the processor 130 maintains the focal position of the virtual camera the same as before the full zoom command was input, and outputs an enlarged image corresponding to the full zoom command from the left eye stereoscopic space image generated based on the left eye image moved to the left. can be extracted. In this way, when a left eye stereoscopic space image is generated by the left eye image moved to the left, the left eye stereoscopic space image may have a counterclockwise rotation compared to before being moved to the left.

Therefore, when extracting an enlarged image from a left-eye stereoscopic space image rotated in a counterclockwise direction while maintaining the focal position of the virtual camera as it is, it is located on the right side more than when extracting an enlarged image from a left-eye stereoscopic space image that is not moved to the left. An existing magnified image can be obtained. That is, when an enlarged image is extracted by moving the left eye image to the left, an enlarged image existing on the right side may be extracted compared to the case of extracting the enlarged image from the left eye image before the movement.

Meanwhile, the processor 130 may display the obtained enlarged image through the display 110 . Specifically, the processor 130 may display the enlarged image obtained from the left eye 3D space image on the display 110 in an area viewed by the user's left eye. In this case, the processor 130 may adjust the size of the enlarged image to have the same size as the image displayed on the display 110 before the entire enlargement command is input and display the enlarged image.

However, the above-described image processing, panoramic 3D image generation, image extraction according to motion direction, image extraction according to zoom in/out command, etc. may be performed by the server 200, and the user terminal device 100 may be performed by the server 200. Of course, it can be implemented to display only the image received from ). For example, when a zoom-in command is input from the user terminal device 100, the user terminal device 100 transmits the corresponding command to the server 200, and the server 200 performs image processing according to the zoom-in command. The obtained image may be provided to the user terminal device 100 .

FIG. 4B is a block diagram showing a specific configuration of the user terminal device shown in FIG. 4A. According to FIG. 4B , the user terminal device 100' includes a display 110, a communication unit 120, a processor 130, a storage unit 140, an image processing unit 150, and a sensor unit 160. Among the components shown in FIG. 4B, detailed descriptions of components overlapping with those shown in FIG. 4A will be omitted.

The storage unit 140 stores various data such as an O/S (Operating System) software module for driving the user terminal device 100' and various multimedia contents.

The user interface unit 150 receives various user commands. Specifically, the user interface unit 150 may receive user commands for various UI screens for controlling functions of the user terminal device 100 ′ provided through the display 110 . In this case, the user interface unit 150 may be implemented as a keypad or touchpad provided on one side of the user terminal device 100, but in some cases, it may be provided with a camera for motion recognition, a microphone for voice recognition, or the like. It can be.

The sensor unit 160 senses the motion state of the user terminal device 100'.

Specifically, the sensor unit 160 includes a geomagnetic sensor, a gyro sensor, an acceleration sensor, and the like for sensing a motion state. The geomagnetic sensor is a sensor for detecting a rotation state and a moving direction of the user terminal device 100'. The gyro sensor is a sensor for detecting a rotation angle of the user terminal device 100'. Both the geomagnetic sensor and the gyro sensor may be provided, but even if only one of them is provided, the user terminal device 100' can detect the rotation state. The acceleration sensor is a sensor for detecting an inclination degree of the user terminal device 100'.

In addition, the sensor unit 160 may further include a touch sensor, a proximity sensor, or the like that detects a touch or hovering manipulation through a touch pad attached to the device.

In this case, the processor 130 transmits information corresponding to the motion state sensed by the sensor unit 160 to the server 200, receives content corresponding to the motion state from the server 200, and displays the display 110. can be displayed through

The image processor 170 renders the received image and provides it to the display 110 .

Meanwhile, since the detailed configuration of the processor 130 shown in FIG. 4B is similar to the detailed configuration of the processor 220 of the server 200 shown in FIG. 3B, a detailed description thereof will be omitted.

5 is a diagram for explaining a method of generating and outputting panoramic VR content according to an embodiment of the present invention.

As shown in FIG. 5, when a 3D image is input from at least one source device (300-1, 300-2, 300-3), the server 200 separates left and right images (510), and then converts each image into a virtual image. A left-eye stereoscopic image and a right-eye stereoscopic image are generated by projecting on the stereoscopic circular space of (520). In this case, the generated stereoscopic image can be a panoramic VR image, and the existing 3D-based Avatar, Object, Space, Panorama VR and Photo Object A panoramic VR image can be created using various VR techniques such as Object) VR.

However, in the case of receiving a stereo image in which the left and right eye images are separated, the left and right images may be used as they are, or a 3D image may be generated and used based on a 2D image according to a 2D-3D conversion technique. should be omitted.

Thereafter, a desired viewpoint image is selected through the virtual camera engine and transmitted to the user terminal device (eg, HMD) 100, and the user terminal device 100 provides the received images as left and right images, respectively, to create 3D VR video can be provided.

Meanwhile, when the changed motion information is received from the user terminal device 100, the server 200 analyzes the received motion information (540) and adjusts the direction of the virtual camera based on the analyzed motion information (550). Through the virtual camera engine 530 , left and right images of the viewpoint area corresponding to the corresponding motion information may be extracted and transmitted to the user terminal device 100 . That is, when it is assumed that there is a virtual camera in a circular three-dimensional space, the direction of the virtual camera is adjusted in a direction corresponding to the motion of the user terminal device 100, and the virtual camera whose direction is adjusted among images mapped in the circular three-dimensional space The captured image may be extracted and transmitted to the user terminal device 100 .

6A and 6B are diagrams for explaining a method of streaming content in a server according to an embodiment of the present invention.

According to FIG. 6A , when a plurality of images 610 to 640 photographed from a plurality of source devices are received, the server 200 provides them to the output buffer 650 and individually transmits them to the user terminal device 100. there is. In this case, the output buffer 650 may sequentially or simultaneously transmit the plurality of images 610 to 640 to the user terminal device 100 . Here, the plurality of images 610 to 640 may be general 2D videos, but are not limited thereto, and 3D videos including left and right eye videos for providing VR services, panoramic VR contents, and general 2D videos may also be included therein. can

According to FIG. 6B, when a plurality of images 610 to 640 photographed from a plurality of source devices are received, the server 200 synthesizes them (660) and provides them to the output buffer 650 so that the user terminal device 100 can be sent to In this case, the plurality of received images 610 to 640 may be synthesized based on location information of a plurality of source devices that provided them.

On the other hand, the server 200 receives images based on content information that varies depending on the resolution information of the plurality of source devices that provided the plurality of images 610 to 640 and the user terminal device 100, display arrangement state, screen size, and the like. may be processed to be suitable for the user terminal device 100 and transmitted to the user terminal device 100, or processed content may be synthesized and transmitted to the user terminal device 100.

7 is a flowchart for explaining the operation of a source device according to an embodiment of the present invention.

According to the flowchart shown in FIG. 7, the source devices 300-1, 300-2, and 300-3 start providing information according to a preset event (S711). Here, the predetermined event may be various events that can be regarded as starting to provide information to a third party, such as an event of installing/running an application to provide a corresponding service or an event of subscribing to a corresponding service.

Meanwhile, when the users of the source devices 300-1, 300-2, and 300-3 start to provide information, they request access to the server 200 and set an information provision level (S720). For example, information sharing levels with third parties, such as audio data, video data, location information, and message reception or not, may be set.

Subsequently, when the information provision request is received from the server 200 (S730:Y), the corresponding information is transmitted to the server 200 according to the information provision level set in step S720 (S740).

8 is a sequence diagram for explaining the operation of a server according to an embodiment of the present invention.

According to the flowchart shown in FIG. 8, the server 200 registers/manages information providers according to preset events (S810). For example, when users of the source devices 300-1, 300-2, and 300-3 subscribe to the corresponding service, the users of the subscribed source devices 300-1, 300-2, and 300-3 are registered. and can be managed. In this case, the server 200 may also manage information provision levels set by users of the source devices 300-1, 300-2, and 300-3 for each user.

Next, when there is an access request from the user terminal device 100 (S820:Y), the user of the user terminal device 100 requesting the access is checked (S830). For example, the server 200 may check whether the corresponding user is a user subscribed to a service provided by the server 200 . In this case, whether the user has subscribed to the service provided by the server 200 through device information (eg, device identification information of the user terminal device) as well as user authentication information (eg, ID, password, etc.) You can check. However, since one device can be used by multiple users, it may be desirable to verify the user through user authentication information.

Meanwhile, when the user of the user terminal device 100 is confirmed in step S830, the information provider, that is, the location of the source devices 300-1, 300-2, and 300-3 is provided to the user terminal device 100 (S840). ). In this case, the location of the information provider for the place set by the user terminal device 100 may be provided. For example, when the user of the user terminal device 100 requests information about the vicinity of Gangnam Station, the location of the source devices 300-1, 300-2, and 300-3 owned by the information provider for the vicinity of Gangnam Station is sent to the user. It can be provided to the terminal device 100. In this case, the server 200 may also provide an image (for example, a map image) in which the location of the information provider is displayed, but it is also possible to provide only location information of the information provider.

Subsequently, when a request for information on at least one specific location is received from the user terminal device 100 (S850), the video received from the source device of the corresponding location is transmitted to the user terminal device 100 (S860). In this case, the server 200 may buffer and synthesize the received image and provide it, or stream it in real time.

9 is a sequence diagram for explaining the operation of a user terminal device according to an embodiment of the present invention.

According to the flowchart shown in FIG. 9 , the user terminal device 100 accesses the server 200 providing the virtual experience service (S910) and sets a region to be accessed (S920).

Subsequently, the user terminal device 100 sets an information request authority level (S930). For example, information request authority levels such as whether to receive audio data and video data, whether to receive only video data, or whether to send a message to an information provider may be set.

Thereafter, the user terminal device 100 receives and renders information of the requested level, for example, an image, from the server 200 (S940). In this case, when the level of information requested by the user terminal device 100 is higher than the information provision level of the information provider, that is, the source device 300-1, 300-2, or 300-3, the corresponding source device 300-1, Information may be provided within the information provision level of 300-2 and 300-3). For example, when the user terminal device 100 requests video and audio from the first source device 300-1, but the information provision level of the first source device 300-1 is set to video only, the server 200 ) may transmit only video to the user terminal device 100.

Then, the user terminal device 100 displays the rendered image (S950). That is, when the user terminal device 100 is implemented as an HMD, the user of the user terminal device 100 is provided with a real-time image of a specific place through the displayed image and can experience a virtual experience.

10 to 12 are diagrams for explaining a virtual experience service providing method according to various embodiments of the present disclosure.

As shown in FIGS. 10 to 12, by receiving an image of a specific place from an external user using a source device, that is, a wearable device, the user terminal device 100, that is, the HMD user, is provided to the field of view of other users. It is possible to provide a service in the form of sharing. In this case, the user of the wearable device may set the content sharing level through the authority setting of the user's device. For example, a level of sharing only video through a camera, a level of sharing only sound information through a microphone, a level of sharing both video and sound, a level of not sharing both video and sound, and text messages from accessors separately from each sharing level. In this case, the content sharing level can be set to a level that provides permission to allow voice information to be input.

10 is a diagram for explaining a method of providing a virtual road experience service according to an embodiment of the present invention.

As shown in FIG. 10, when the HMD user selects an area to receive the virtual road experience service, the locations of a plurality of users (1011, 1012, 1013..) with wearable devices in the selected area are displayed (1021, A UI screen 1020 displaying 1022, 1023...) may be provided. In this case, content sharing levels set by wearable device users may also be displayed. In this case, the HMD user can share the wearable device user's images (A, B, C...) through the server 200 by selecting a user corresponding to a desired content sharing level and location.

Alternatively, by searching for the user's location close to the destination route in conjunction with the route finding function, the wearable device users (1011, 1012, 1013..) located on the route are provided with images and sounds in sequence, so that the virtual It can also predict real-time route conditions. In this case, when the distance between wearable device users located on the route exceeds a predetermined range, data such as a previously stored street view may be provided to the HMD user instead of sharing the wearable device user's field of view.

11 is a diagram for explaining a method of providing a virtual tour service according to an embodiment of the present invention.

As shown in FIG. 11, when the HMD user selects an area to receive virtual tour service, the locations of a plurality of users (1111, 1112, 1113..) with wearable devices in the selected area are displayed (1121, 1122). , 1123..) may be provided. In this case, content sharing levels set by wearable device users may also be displayed. In this case, the HMD user can share an image of the wearable device user through the server 200 by selecting a user corresponding to a desired content sharing level and location.

12 is a diagram for explaining a method of providing a virtual event experience service according to an embodiment of the present invention.

As shown in FIG. 12, when the HMD user selects a performance for receiving a virtual performance experience service through the corresponding UI screen 1220, a performance video received from a plurality of users with wearable devices in the selected performance hall can be shared. be able to In some cases, a UI screen displaying locations of a plurality of users capable of sharing content within the performance hall may be provided. In this case, content sharing levels set by wearable device users may also be displayed. Accordingly, the HMD user can selectively share an image captured at a seat of his or her choice even within the concert hall.

Alternatively, audio information is obtained from a user closest to the performance location among a plurality of wearable device users in the performance hall, and image information is collected/combined with image information possessed by each user and changes in viewpoint between various users according to user input. You can also provide a video.

13A and 13B are diagrams illustrating an implementation example of a UI screen provided by a user terminal device according to an embodiment of the present invention.

As shown in FIG. 13A, when a user command for setting a preset region is input, the user terminal device 100 transmits information related to the corresponding region to the server 200, and is located within the preset region from the server 200. By receiving location information about at least one source device, a UI screen indicating locations 1311 to 1314 of the at least one source device may be displayed. In this case, as shown, at least one source device may receive permission information on a content sharing level that can be shared with the user terminal device 100 from the server 200 and display the same.

As shown in FIG. 13B, it is also possible to display the locations (1321 to 1324) of source device users who can share images within the current gaze range by applying AR technology to images displayed at the current virtual experience location. In this case, as shown, distance information to the corresponding source device user may be additionally displayed.

On the other hand, the user of the user terminal device 100 uses a command to select a source device to share the captured image based on the information displayed on the UI screen as shown in FIGS. 13A and 13B to select content for a specific location. A transmission request may be transmitted to the server 200 .

14 is a flowchart illustrating a method for controlling a server according to an embodiment of the present invention.

According to the method for controlling a server shown in FIG. 14 , when a content transmission request for a preset location is received from a user terminal device (S1410:Y), the server determines the location information received from at least one source device. Content captured in real time is received from a source device at a set location (S1420).

Subsequently, the received content is provided to the user terminal device in a streaming form (S1430).

Meanwhile, in step S1330 of providing the received contents to the user terminal device in a streaming form, the plurality of captured contents received from the plurality of source devices are sequentially transmitted to the user terminal device based on the location information received from the plurality of source devices. Alternatively, captured content received from a plurality of source devices may be synthesized based on location information received from the plurality of source devices, and the synthesized content may be transmitted to the user terminal device.

In addition, in step S1430 of providing the received content to the user terminal device in a streaming form, among the content received from the source device, based on the authority information in which at least one source device sets a content sharing level capable of being shared with the user terminal device. Content corresponding to the corresponding authority information may be selected and transmitted to the user terminal device.

In addition, in step S1430 of providing the received content to the user terminal device in a streaming form, the content received from the source device may be processed according to a preset condition set in the user terminal device and transmitted to the user terminal device.

In addition, the control method includes, when the content received from the source device is 3D content, projecting a left eye image and a right eye image included in the 3D content to a virtual stereoscopic space, respectively, to generate a left eye stereoscopic space image and a right eye stereoscopic space image. and, when information corresponding to the motion state of the user terminal device is received from the user terminal device, extracting a viewpoint area corresponding to the motion state of the user terminal device from the generated left and right eye 3D space images and transmitting the extracted view area to the user terminal device. may further include.

15 is a flowchart illustrating a control method of a user terminal device according to an embodiment of the present invention.

According to the control method of the user terminal device illustrated in FIG. 15 , when a user command specifying a preset location is input (S1510:Y), information related to the location is transmitted to the server (S1520).

Thereafter, real-time photographing content related to the corresponding location is received from the server and displayed on the display (S1530).

In step S1530 of receiving and displaying content, at least one source device providing real-time content can receive only selected content from the server based on authority information setting a content sharing level that can be shared with the user terminal device. . However, in some cases, it is also possible for the source device to transmit only content selected based on authority information to the server.

In addition, in steps S1510, S1520, and S1530, when a user command for setting the first and second points is input on the UI screen, information corresponding to a preset path connecting the first and second points is transmitted to the server. In addition, a plurality of contents related to the predetermined path may be received from the server and displayed, or synthesized contents obtained by synthesizing a plurality of contents related to the predetermined path may be received and displayed from the server.

In addition, the control method, when a user command for setting a preset region is input, transmits information related to the corresponding region to the server, receives location information about at least one source device located within the preset region from the server, and at least The method may further include displaying a GUI indicating a location of one source device.

In this case, authority information on a content sharing level that can be shared by at least one source device with a user terminal device may be received from the server and displayed in an area adjacent to a GUI indicating a location of the source device.

In addition, in steps S1510, S1520, and S1530, when a user command specifying a predetermined location is input on the UI screen providing the virtual experience service, information related to the location is transmitted to the server, and the server transmits the virtual image captured at the location. Experience content may be received and displayed. Here, the virtual experience service may include at least one of a virtual road experience service, a virtual tour service, and a virtual event experience service.

As described above, according to various embodiments of the present disclosure, it is possible to experience a sense of reality in a remote place using information acquired through another person's wearable device. In addition, a service for directly or indirectly paying a price may be provided to a user of a wearable device located in a remote location.

16 to 18 are diagrams illustrating an example of performing a use action between a user of a first space and a user of a second space, which is an offline space, according to an embodiment of the present invention.

The first space is a space where a first user holding a VR device is located or a virtual reality space in which the first user is expressed. The second space is a real space where one or more users are located. The display device on the second space outputs an image for the first user. Through this, people located in the second space can feel as if they are living in the second space (eg, viewing an exhibition hall corresponding to the second space) together with the first user. As the video obtained by the display device in the second space is provided to the VR device of the first user, the first user can feel as if he is with other users in the second space.

Meanwhile, the methods according to these various embodiments may be programmed and stored in various storage media. Accordingly, methods according to various embodiments described above may be implemented in various types of electronic devices that execute storage media.

Specifically, according to an embodiment of the present invention, the steps of receiving a content transmission request for a preset location from a user terminal device, and a source device at a preset location based on location information received from at least one source device. A non-transitory computer readable medium storing a program that sequentially performs the steps of receiving content captured in real time from and providing it to a user terminal device in a streaming form may be provided.

A non-transitory readable medium is not a medium that stores data for a short moment, such as a register, cache, or memory, but a medium that stores data semi-permanently and can be read by a device. Specifically, the various applications or programs described above may be stored and provided in non-transitory readable media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: user terminal device 110: display
120: communication unit 130: processor
200: user terminal device 210: communication unit
220: processor

Claims (10)

In a server providing content to a user terminal device providing a virtual reality service,
a communication unit that communicates with at least one source device and a user terminal device; and
When a content transmission request for a preset location is received from the user terminal device, based on the location information received from the at least one source device, content captured in real time from the source device at the preset location is received, and the user terminal A server that includes; a processor that transmits to a device. According to claim 1,
the processor,
A plurality of captured contents received from a plurality of source devices are sequentially transmitted to the user terminal device based on location information received from the plurality of source devices, or the captured contents received from the plurality of source devices are transmitted to the plurality of sources. A server characterized in that for synthesizing content based on location information received from a device and transmitting the synthesized content to the user terminal device. According to claim 1,
the processor,
The at least one source device selects the content corresponding to the permission information from among the content received from the source device based on the permission information setting the content sharing level that can be shared with the user terminal device, and transmits the selected content to the user terminal device. A server characterized by doing. According to claim 1,
the processor,
The server characterized in that the content received from the source device is processed according to a predetermined condition set in the user terminal device and transmitted to the user terminal device. According to claim 4,
the processor,
When the content received from the source device is 3D content, a left eye image and a right eye image included in the 3D content are projected onto a virtual 3D space to generate a left eye 3D space image and a right eye 3D space image, and the user terminal device When information corresponding to the motion state of the user terminal device is received from, a viewpoint area corresponding to the motion state of the user terminal device is extracted from the generated left and right eye stereoscopic space images and transmitted to the user terminal device. server with . In a user terminal device providing a virtual reality service,
display;
a communication unit that communicates with the server; and
A user terminal including a processor configured to transmit information related to the location to the server, receive real-time content related to the location from the server, and display the content through the display when a user command specifying a preset location is input. Device. According to claim 5,
the processor,
The user terminal device according to claim 1 , wherein at least one source device providing the real-time captured content receives only selected content from the server based on authority information setting a content sharing level that can be shared with the user terminal device. According to claim 5,
the processor,
When a user command for setting the first point and the second point is input on the UI screen, information corresponding to a preset path connecting the first point and the second point is transmitted to the server, and the server transmits the A user terminal device characterized in that it controls to receive and display a plurality of contents related to a set path. According to claim 5,
the processor,
When a user command for setting the first point and the second point is input on the UI screen, information corresponding to a preset path connecting the first point and the second point is transmitted to the server, and the server transmits the A user terminal device characterized in that it controls to receive and display synthesized content synthesized from a plurality of contents related to a set path. According to claim 5,
the processor,
When a user command for setting a preset region is input, information related to the corresponding region is transmitted to the server, location information on at least one source device located in the preset region is received from the server, and the at least one source device is received. A user terminal device characterized by controlling to display a GUI indicating a location of a source device.

Images