KR101917359B1 - Realistic seeing-through method and system using adaptive registration of inside and outside images - Google Patents

Abstract

A realistic seeing-through method using adaptive matching of internal and external images of a specific space comprises: a step of capturing an omnidirectional internal image of a first space by using a first camera of a remote device located inside the first space; a step of uploading omnidirectional internal image data of the first space to a real time streaming server through a wired or wireless network; a step of capturing an external obstacle in the first space by using a second camera of a user device located outside the first space; a step of downloading the omnidirectional internal image data of the first space from the real time streaming server; a step of generating a seeing-through image by adaptively matching the omnidirectional internal image of the first space and an external obstacle image of the first space; and a step of displaying the seeing-through image on the user device. According to the realistic seeing-through method, the seeing-through image generated by adaptively matching an image of a destination which is streamed in real time through the server and an image of an obstacle which is captured from the outside by the user device is displayed to provide a user with a feeling of immersion as if he or she is looking directly at the inside by seeing through the obstacle.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and system for real-

More particularly, the present invention relates to a method and system for real-time sensing through adaptive matching of internal and external images, and more particularly, And more particularly, to a real-time perspective method and system for allowing a user to view a perspective image by adaptively matching the real-perspective view.

[Explanation of R & D for National Support]

This study was supported by the Korea Institute of Science and Technology under the support of the Global Frontier Project of the Creation Science Department of the Future (Involuntary VST Glasses based Coexistence Reality UI / UX and Human Factor Research and Development Project Assigned No. 1711041262).

A mixed reality environment is a virtual reality in which a user is completely immersed in a virtual space created by a computer and an augmented reality in which a virtual object created by a processor is enhanced in a real world viewed by a user Augmented Reality) is an environment in which a user can interact with these virtual objects. Mixed reality environment can be utilized in various fields such as game, multimedia, SNS, telepresence, education and design, and it is becoming important.

Recently, a variety of contents utilizing hybrid reality have been developed as a head mounted display (HMD) device supporting the immersive virtual environment and the augmented reality has been popularized. HMD devices such as Samsung Gear VR, Microsoft Hololens, and AMD Sulon can be combined with mobile phones to provide content, allowing users to use mixed real-world content as well as specified space.

In addition, recently, a diminished reality technique has been developed which treats objects in real life in the augmented reality field invisibly according to the purpose of the user. In the reduction reality, when the user selects an object desired to be removed from the image information acquired through the camera, the object image and the background area of the object are recognized, and the object image is removed from the image, In reality, visualization is often used as the object is removed.

For example, in an X-ray game using Microsoft's Hololens, combining such reduced reality with virtual reality content, if a user attacks a wall with a laser in the image of the actual wall, the wall disappears The virtual reality world was visualized in that position.

Another example of using a mobile mixed reality device is a location based guiding service provided when a user moves from a building to a specific destination. For example, when a user visits a hospital, since a place to receive a service is different according to a schedule of a user, such as receptionist, doctor's diagnosis, close-up examination, prescription, Should guide the next destination correctly considering the situation of the user.

The user can connect the image of the destination space to the display to find a route in the building, and can receive a guidance service such as interacting with employees at a destination, receiving a guidance message, and the like. However, there is a need to provide a more intuitive visualization method for interacting with the object of the guidance service or destination when using the guidance service in a more complex building structure, or for those who are not familiar with the operation of the device.

Korean Patent No. 10-1229078 B1

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a mixed reality environment for a user, which is capable of providing a user with an internal image of a destination (for example, (For example, a building wall image) is adaptively matched to provide an immersion feeling as if the user views the obstacle directly through the obstacle and further provides a technique for interacting with the object in the destination .

A sensory perspective method according to one aspect of the present invention includes: capturing an omnidirectional internal image of a first space using a first camera of a remote device located inside a first space; Uploading forward direction internal image data of the first space to a real time streaming server through a wired or wireless network; Capturing an external obstacle in a first space using a second camera of a user device located outside the first space; Downloading the forward direction internal image data of the first space from the real time streaming server; Generating a perspective image by adaptively matching an image in the forward direction of the first space with an image of the obstacle in the first space; And displaying the perspective image on the user device.

In one embodiment, the step of generating the perspective image by adaptively matching the forward and backward images of the first space with the outer obstacle image of the first space may include: ; Determining a shooting direction of the second camera based on motion of the user device based on an external obstacle image of the first space or using an inertial measurement unit (IMU); Extracting a one-directional internal image from the forward direction internal image of the first space based on the perspective area and the photographing direction of the second camera; And generating a perspective image by replacing the area to be viewed in the external obstacle image with the one-directional internal image.

In one embodiment, the perspective image may include a visual effect in which the one-directional internal image appears as a part of the external obstacle is broken down.

In one embodiment, the sensory inspection method includes: photographing an object located in the first space using the first camera; Uploading information about the object to the real-time streaming server;

Selecting the object through an interface of the user device; Downloading information about the object from the real-time streaming server; And displaying the information on the user device.

In one embodiment, the user device may be a head mounted display device.

In one embodiment, a computer program for executing the realistic perspective method may be provided.

According to another aspect of the present invention, there is provided a sensory perspective system comprising: a remote device for capturing and uploading an internal image of a first space; A real time streaming server for streaming video data in real time; And a user device located outside the first space, the remote device comprising: a first camera capable of capturing 360 degrees of the interior of the first space; And a data upload module for uploading the forward direction internal image data of the first space to the real time streaming server via a wired or wireless network, the user device comprising: 2 camera; A data download module for downloading the forward direction internal image data of the first space from the real time streaming server; A perspective image generation module for generating a perspective image by adaptively matching the forward direction internal image of the first space and the external obstacle image of the first space; And a display for displaying the perspective image.

In one embodiment, the perspective image generation module includes: a perspective area designation unit for designating an area to be viewed from an outer obstacle image of the first space; A photographing direction judging unit for judging a photographing direction of the second camera based on an image of an external obstacle in the first space or based on a motion of the user device tracked using an inertial measurement unit (IMU); An internal image extraction unit for extracting a one-directional internal image from the forward direction internal image of the first space based on the shooting direction of the perspective region and the second camera; And an internal image replacement unit for replacing the perspective area with the one-directional internal image in the external obstacle image.

In one embodiment, the perspective image may include a visual effect in which the one-directional internal image appears as a part of the external obstacle is broken down.

In one embodiment, the data upload module uploads information on the object located in the first space and photographed by the first camera to the real-time streaming server, and the data download module transmits the information from the real-time streaming server The information about the object can be downloaded.

In one embodiment, the user device may be a head mounted display device.

According to the method and system for real-time observation through adaptive matching of internal and external images provided herein, an internal image of a destination streamed in real time through a server is adaptively matched with an obstacle image captured from the outside by a user device By displaying the generated perspective image, it is possible to provide the user with an immersion feeling as if the user views the obstacle directly and looks inside.

This allows a higher immersion feeling in a mixed reality environment, such as a game or telepresence, if the user wears a head-mounted display (HMD) device. In addition, it is possible to provide an intuitive route guidance service without any special operation, so that it can be easily reached to a destination within a complex building structure.

Furthermore, the user can interact with the image and information about the object located at the destination by downloading through the server, thereby providing a higher immersion feeling.

FIG. 1 is a flowchart illustrating a sensory inspection method according to an exemplary embodiment of the present invention.
FIG. 2 is a flowchart illustrating in detail the step of generating the perspective image of FIG.
3A and 3B are diagrams illustrating a user performing a real-time perspective method according to an embodiment.
FIGS. 4A and 4B are views showing a perspective image displayed on a user device when performing a real-time perspective method according to an embodiment.
FIG. 5 is a block diagram illustrating a real-feeling perspective system according to an embodiment.
FIG. 6 is a detailed block diagram of the perspective image generation module of FIG.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. The various embodiments of the present invention are different but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. Furthermore, the position or arrangement of individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which the claims are entitled. In the drawings, like reference numerals refer to the same or similar functions in various aspects.

As used herein, terms used in the present specification are selected from the general terms that are currently widely used, while taking into consideration the functions, but these may vary depending on the intention or custom of the artisan or the emergence of new techniques. Also, in certain cases, there may be a term selected by the applicant at will, in which case the meaning will be described in the description part of the corresponding specification. Therefore, the terms used in the present specification should be interpreted based on the meaning of the term rather than on the name of a simple term, and on the contents throughout the specification.

In addition, the embodiments described herein may be wholly hardware, partially hardware, partially software, or entirely software. As used herein, the terms "unit," "module," "device," "server," or "system" Refers to a computer-related entity. For example, a component, a module, a device, a server, or a system may refer to software such as an application for driving the hardware and / or the hardware constituting part or all of the platform.

Hereinafter, a preferred embodiment of a method for selecting a superimposed virtual object using spatial relationship information will be described in more detail with reference to the drawings.

FIG. 1 is a flowchart illustrating a sensory inspection method according to an exemplary embodiment of the present invention. Referring to FIG. 1, a step of photographing a forward direction internal image of a first space using a first camera capable of 360-degree photographing is performed (S100).

Referring to FIG. 3A, which illustrates devices for photographing the inside / outside of the first space, a first space 1, which is a remote space to be viewed, and a remote device 10, which is installed in the first space, There is shown a user 2 and a user device 20 controlled by the user and an external obstacle 30 blocking between the first space 1 and the user 2. [ The first space is a remote space isolated from the user, for example, a space inside a building or a space beyond a wall to be viewed.

The remote device 10 includes a first camera capable of photographing not only a specific direction inside the first space 1 but also all directions 360 degrees, and is installed so as to be able to photograph all the inside of the first space. In the following embodiments, the area of the first space taken by the first camera is simplified to A, B, C, D for the sake of understanding, but the present invention is not limited thereto. In another embodiment, It can be further segmented using a 360-degree camera. In an embodiment, the outer obstacle 30 may be an obstacle that blocks the field of view such that the user can not directly see the first space, such as building exterior walls or interior walls.

As shown, the user 2 wears the user device 20. As discussed below, the user device 20 may be any type of device, including a second camera, a processor and a display, capable of capturing an external obstacle 30, such as a smartphone, tablet, laptop computer, A head-mounted display (HMD) device, a combination thereof, and the like. Unlike the first camera, the second camera need not be a 360 degree camera.

(For example, A, B, C, and D in FIG. 3A) of the first space and transmits the forward direction internal image data of the first space to the real time streaming server through the wired or wireless network An uploading step is performed (S200). The method of uploading the image data photographed by using the transmission unit in the remote device to the server can be performed by a conventional well-known wired / wireless communication and is not a core technical feature of the present embodiment, so a detailed description will be omitted.

Meanwhile, in step S300, an external obstacle in the first space is photographed using a second camera included in the user device outside the first space. Step S300 may be performed before or after steps S100 to S200, and is not subject to temporal order.

Referring to FIG. 3A, a user 2 photographs an external obstacle 30 in a first space using a second camera included in the user device 20. As described later, in one embodiment, the user can specify an area to be viewed from the external obstacle image, so that the captured external obstacle image E can be displayed on the user device as shown in FIG. 4A.

Next, the step of downloading the forward direction internal image (e.g., A, B, C, D) data of the first space from the real time streaming server to the user device through a wireless or wired network is performed (S400). A method of downloading image data photographed using a receiving unit in a user device from a server can be performed using conventional wired / wireless communication, and is not a core technical feature of the present embodiment, so a detailed description will be omitted.

Then, a perspective image is generated by adaptively matching the forward and backward images of the first space with the outer space of the first space (S500). The image registration technique includes an arbitrary image processing technique in which, when a scene or an object is photographed at a different time or point of view, an image of a different coordinate system is transformed and displayed in one coordinate system. As will be described in detail below, the first space internal image captured by the first camera and the external obstacle image captured by the second camera are matched in real time so that the user can observe the obstacle and see the first space Images can be generated.

In one embodiment, the generating of the perspective image (S500) comprises: designating an area to be viewed from the external obstacle image of the first space (S510); based on the external obstacle image of the first space; (S520) of determining a photographing direction of the second camera based on the motion of the user device tracked by using the inertia measurement unit (IMU) (S520), determining, based on the photographing direction of the second camera, A step S530 of extracting a one-directional internal image from the forward direction internal image of the first space, and a step S540 of generating a perspective image by replacing the one-direction internal image with an area to be viewed from the external obstacle image have.

In operation S510, when the external obstacle image photographed by the second camera is displayed on the user device, the user can designate an area of the external obstacle image to be viewed. The method of designating the region to be viewed may be performed by touching a part of the output image when the user device is a smart phone or a tablet and by using a part of the output image using an input device such as a mouse or a keyboard And in the case of a head-mounted display (HMD), it can be performed by tracking a user's hand gesture using a camera mounted on the HMD or an IMU sensor. In another embodiment, the step S510 may be performed by the processor in the user device by automatically selecting a predetermined area of the obstacle image.

Referring to FIG. 4A, in the external obstacle image E photographed in step S300, a region to be viewed E 'is designated according to step S510. As described above, it is possible to passively select an arbitrary size according to various inputs of a user, or designate an area E 'to be viewed by automatically selecting by a processor.

In step S520, the photographing direction of the second camera can be determined based on the external obstacle image in the first space. This is for extracting a one-directional internal image corresponding to the photographing direction of the second camera, that is, the gaze direction of the user, among the forward direction internal images of the first space photographed in step S100 as described later.

In an embodiment, the live streaming server may include image and direction information of an external obstacle (e.g., building wall) in the first space, and the user device may download the information and compare it with the obstacle image, The photographing direction, that is, the direction of the user's gaze can be determined. In one embodiment, in the case where the user device is an HMD device, the real-time streaming server may include feature tracking information of the first space, and the HMD device may include information obtained from an embedded inertial measurement unit Value and the minutiae tracking information, it is possible to effectively calculate how far the user is from the area to be viewed.

3A and 3B, the obstacle image E captured by the second camera of the user device is analyzed. As a result, when the user selects one of the first spaces A, B, C, ), And by using the minutiae tracking information, it is possible to know how far the user is from the first space.

Accordingly, the unidirectional internal image is extracted (S530) based on the photographing direction of the second camera and the viewable area designated in step S510. For example, if the image coinciding with the second camera photographing direction of the user, that is, the gaze direction of the user out of the forward direction internal images A, B, C, and D of the first space is A, (A ') corresponding to the size and direction of the region E' to be viewed out of the one-way image (A) can be selectively extracted.

Referring to FIG. 4B, a perspective image displayed by replacing a designated viewable area E 'of the external obstacle image E with the unidirectional internal image A' is shown. In one embodiment, as shown in FIG. 4B, the perspective image may include a visual effect in which an inner image is displayed as a part of the outer obstacle is broken down. Such a visualization method may be performed by converting a photographed obstacle image (for example, a wall image of a building) into a texture and generating a polygon to perform a destruction simulation. The visual effect can be utilized in an entertainment field such as a game by allowing a user to immerse himself in a mixed reality environment and realizing different visual effects according to an object to be performed.

By displaying the adaptively matched perspective image on the user device, the user can feel the immersed feeling as if he or she is looking inside the first space by looking at the external obstacle, and the 360 degree camera installed at the destination Only the video information in the field of view corresponding to the viewing direction viewed by the user is selected and streamed, thereby reducing the streaming capacity, thereby improving the streaming speed and stably using the network bandwidth.

In one embodiment, the realistic perspective method described above may further comprise the additional step of allowing the user to interact by acquiring information about an object located in the first space. Specifically, the sensory inspection method includes: photographing an object located in the first space using the first camera; uploading information about the object to the real-time streaming server; Downloading the information about the object from the real-time streaming server, and displaying the information on the user device.

For example, if the first space is a hospital interior, the object may be a storage kiosk (KIOSK) located in the hospital. Since the first camera is a camera capable of photographing 360 degrees, the storage kiosk located in the first space is photographed, and information about the storage kiosk (for example, a description of an object such as " Is uploaded to the real-time streaming server together with the first spatial image data.

When the user selects the object (storage kiosk) in the first spatial image (hospital internal image) displayed on the device, information about the object is additionally displayed on the user device. The selection of the object through the interface can be performed by touching a part of the output image when the user device is a smartphone or a tablet and by using an input device such as a mouse or a keyboard in the case of a laptop computer In the case of a head-mounted display (HMD), it can be performed by tracking a user's hand motion using a camera mounted on the HMD or an IMU sensor.

In this way, the user can interact with the first space and the object more intuitively by downloading the image and information about the object located at the destination in real time through the server, so that a higher immersion feeling can be obtained in the mixed reality environment.

The sensory perspective method according to the above-described embodiments can be implemented in the form of program instructions that can be implemented in an application or through various computer components, and the program can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions that are recorded on a computer-readable recording medium may be those that are specially designed and constructed for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform processing in accordance with the present invention, and vice versa.

Hereinafter, a real-feeling perspective system for implementing a real-reality perspective method according to the above-described embodiments will be described.

FIG. 5 is a block diagram illustrating a real-feeling perspective system according to an embodiment. 5, the real-feeling perspective system includes a remote device 10 for photographing and uploading an internal image of a first space, a real-time streaming server 15 for streaming the first spatial internal image data through a wireless or wired network, And a user device 20 located outside the first space.

In the embodiment, the remote device 10 includes a first camera 100 capable of capturing 360 degrees of the interior of the first space, and a second camera 100 capable of transmitting the forward direction internal video data of the first space to the live streaming server through a wired or wireless network And a data upload module 110 for uploading data. The user device 20 further includes a second camera 200 for capturing an external obstacle in the first space, a data download module (downloading module) for downloading the forward direction internal image data of the first space from the live streaming server 210), a perspective image generation module (220) for generating a perspective image by adaptively matching the forward direction internal image of the first space and the external obstacle image of the first space, and a display (230).

As described above, the first camera 100 includes a first camera capable of photographing not only a specific direction inside the first space but also all directions 360 degrees, So that it can be photographed. The data upload module 110 included in the remote device 10 photographs the forward direction internal images of the first space (for example, A, B, C, and D in FIG. 3A) And is used for uploading the forward direction internal image data of the first space to the real time streaming server 15.

The user device 20 may be any type of device that includes a second camera 200 capable of capturing an external obstacle in a first space, such as a smart phone, a tablet, a laptop computer, a head-mounted display ) Devices, combinations thereof, and the like. Unlike the first camera 100, the second camera 200 need not be a 360 degree camera.

The data download module 210 of the user device 20 may be configured to download the forward direction internal image data of the first space from the real time streaming server 51 to the user device 20 via a wireless or wired network . A method of uploading / downloading video data to / from a real-time streaming server using the wired / wireless network can be performed using a conventional wired / wireless communication, and is not a core technical feature of the present embodiment, so a detailed description will be omitted .

Hereinafter, a perspective image is generated by adaptively matching the forward direction internal image of the first space and the external obstacle image of the first space using the perspective image generation module 220 included in the user device 20 Is explained.

6, the perspective image generation module 220 includes a perspective area designation unit 222 for designating an area to be viewed from an outer obstacle image of the first space, An internal image extracting unit 224 for extracting a one-directional internal image from the forward direction internal image of the first space based on the imaging direction of the perspective region and the second camera, And an internal image substitution unit 225 for replacing the perspective area with the one-directional internal image in the external obstacle image.

The user can designate an area to be viewed from the external obstacle image photographed by the second camera 200 by using the perspective area designation unit 222. [ As described above, the method of designating the area to be viewed can be performed by touching a part of the output image when the user device is a smart phone or a tablet, or by using an input device such as a mouse or a keyboard in the case of a laptop computer Or by selecting a part of the output image. In the case of a head-mounted display (HMD), the HMD can be performed by tracking a user's hand movements using a camera mounted on the HMD or an IMU sensor.

The shooting direction determination unit 223 can determine the shooting direction of the second camera based on the external obstacle image in the first space. In a specific embodiment, a second spatial inward image of the first space photographed based on the motion of the user device tracked based on the external obstacle image of the first space or using the inertial measurement unit (IMU) The photographing direction of the camera, that is, the direction of the user's gaze can be determined. In one embodiment, in the case where the user device is an HMD device, the real-time streaming server may include feature tracking information of the first space, and the HMD device may include information obtained from an embedded inertial measurement unit Value and the feature point tracking information are combined, it is possible to effectively calculate how far the user is from the area to be viewed.

The internal image extracting unit 225 may extract a corresponding one-directional internal image from the forward direction internal image based on the determined photographing direction and the designated viewable region. For example, referring to FIGS. 3A and 3B, an image of the user's second camera photographing direction, that is, an image coincident with the user's gaze direction, among the forward direction internal images A, B, C, If the user designates the partial view area E 'of the external obstacle image E, an internal image A' corresponding to the size and direction of the viewable area E 'of the one-way image A is displayed, Can be selectively extracted.

The internal image substitution unit 225 is used to create a perspective image by replacing the designated perspective area with the one-directional internal image of the extracted first space. In one embodiment, as shown in FIG. 4B, the perspective image may include a visual effect in which an inner image is displayed as a part of the outer obstacle is broken down. The visual effect enables a user to immerse in a mixed reality environment, It can be utilized in the field of entertainment such as games by implementing different visual effects according to the purpose of the game.

As described above, by displaying the adaptively matched perspective image on the user device, the user can feel an enhanced immersion feeling as if he or she is looking inside the first space by looking at an external obstacle, In order to improve the streaming speed and to use the network bandwidth more reliably by only selecting the video information in the field of view corresponding to the viewing direction viewed by the user from the 360 degrees camera, streaming is performed to reduce the capacity for streaming. It does.

In one embodiment, the real-feeling perspective system described above may further include an additional feature that allows the user to interact by acquiring information about an object located within the first space. In this embodiment, the data upload module 110 uploads information about objects photographed by the first camera, which are located in the first space, to the real-time streaming server, The information about the object can be downloaded from the real-time streaming server.

In this way, the user can interact with the first space and the object more intuitively by downloading the image and information about the object located at the destination in real time through the server, so that a higher immersion feeling can be obtained in the mixed reality environment.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments or constructions, It will be understood that the invention can be variously modified and changed without departing from the spirit and scope of the invention.

10: remote device
15: Live streaming server
20: User device
100: First camera
110: Data upload module
200: Second camera
210: Data Download Module
220: perspective image generation module
222: Perspective area designation unit
223: photographing direction judging unit
224: internal image extracting unit
225: internal image replacement unit
230: Display

Claims (11)

Capturing an omni-directional internal image of the first space using a first camera of a remote device located inside the first space;
Uploading forward direction internal image data of the first space to a real time streaming server through a wired or wireless network;
Capturing an external obstacle in a first space using a second camera of a user device located outside the first space;
Downloading the forward direction internal image data of the first space from the real time streaming server;
Generating a perspective image by adaptively matching an image in the forward direction of the first space with an image of the obstacle in the first space; And
And displaying the perspective image on the user device,
Wherein the generating the perspective image comprises:
Determining a perspective area to be replaced with a one-directional inner image of an outer obstacle image of the first space;
Determining a photographing direction of the second camera based on the motion of the user device tracked using the inertia measurement unit (IMU);
Extracting a one-directional internal image from the forward direction internal image of the first space based on the perspective area and the photographing direction of the second camera;
Converting the photographed external obstacle image into a texture and generating a polygon; And
Generating a visual effect in which the boundary polygon of the perspective region is removed and the one-directional internal image is displayed,
Wherein the visual effect includes an effect that the external obstacle is broken and an internal image is displayed at the same time to provide an immersion feeling such that the user views the obstacle and directly sees the obstacle.
delete delete The method according to claim 1,
Capturing an object located in the first space using the first camera;
Uploading information about the object to the real-time streaming server;
Selecting the object through an interface of the user device;
Downloading information about the object from the real-time streaming server; And
Further comprising displaying the information on the user device.
The method according to claim 1 or 4,
Wherein the user device is a head mounted display device.
A computer program stored in a computer-readable recording medium for realizing the real-time perspective method according to claim 1 or claim 4.
A remote device for capturing and uploading an internal image of the first space;
A real time streaming server for streaming video data in real time; And
A user device located outside the first space, the adaptive matching through real-
The remote device includes:
A first camera capable of photographing the inside of the first space through 360 degrees; And
And a data upload module for uploading the forward direction internal image data of the first space to the real time streaming server via a wired or wireless network,
The user device comprising:
A second camera for capturing an external obstacle in the first space;
A data download module for downloading the forward direction internal image data of the first space from the real time streaming server;
A perspective image generation module for generating a perspective image by adaptively matching the forward direction internal image of the first space and the external obstacle image of the first space; And
And a display for displaying the perspective image,
The perspective image generation module includes:
A perspective area designation unit for determining a perspective area to be replaced with a one-directional inner image of an outer obstacle image of the first space;
A photographing direction judging unit for judging the photographing direction of the second camera based on the motion of the user device tracked using the inertia measuring unit (IMU); And
And an internal image extracting unit for extracting a one-directional internal image from the forward direction internal image of the first space based on the perspective region and the photographing direction of the second camera,
Converting the photographed external obstacle image into a texture, generating a polygon,
Wherein the polygon of the boundary region of the perspective region is removed and a visual effect in which the one-
Wherein the visual effect includes an effect that an internal image is displayed at the same time as the external obstacle is broken so as to provide an immersion feeling such that the user views the obstacle and directly sees the obstacle.
delete delete 8. The method of claim 7,
Wherein the data upload module uploads information on an object located in the first space and shot by the first camera to the real time streaming server,
And the data download module downloads information about the object from the real-time streaming server.
11. The method according to claim 7 or 10,
Wherein the user device is a head mounted display device.

Images