TW202411936A - Method and program for providing augmented reality image by using depth data - Google Patents

Abstract

The present invention relates to a method and a program for providing an enhanced reality image using depth data. According to an embodiment of the present invention, the method for providing an enhanced reality image using depth data includes: a virtual image data receiving step, wherein the client receives virtual image data from a server; a display position determining step, wherein the client determines the position where each pixel should be displayed in a real space based on the depth data; and a virtual image data displaying step, wherein the virtual image data is displayed on a display space based on the determined position.

Description

Method and program for providing enhanced reality image using depth data

The present invention relates to a method and a program for providing an enhanced reality image using depth data.

Augmented reality is a technology that superimposes virtual objects such as text or images on the display world to present them as a single image. Augmented reality technology was in the research and development and experimental application stage until the mid-2000s, but has recently entered the practical stage as the technical environment has improved. In particular, with the recent emergence of smartphones and the development of network technology, augmented reality technology has begun to attract attention.

The most common method of realizing augmented reality is to use a smartphone camera to shoot the real world and superimpose a pre-generated computer image on it to output it, so that the user feels that the virtual and real world are fused. This method allows users to easily obtain real images using smartphone cameras, and computer images can also be easily realized through the computing function of smartphones, so most augmented reality applications are realized on smartphones. In addition, with the recent appearance of wearable devices in the form of glasses, the attention to augmented reality technology is increasing.

The enhanced reality image should be enhanced at the exact position in the real space to provide a sense of reality for the user. However, when the spatial layout of the object (e.g., a marker) combined with the enhanced reality image changes due to the movement of the user, if the image frame at a predetermined time point is missed, the enhanced reality image may be arranged at an inappropriate position, displayed unnaturally in the display space, and the sense of reality is reduced due to shaking. Therefore, the present invention is expected to provide a method and program for providing enhanced reality images using depth data as follows: using depth data to arrange virtual image data as a two-dimensional image at an appropriate position in the real space, so that when the user moves or frames are missed, an enhanced reality image with a sense of reality and no shaking can be provided.

Furthermore, the present invention is intended to provide a method and program for providing an augmented reality image using depth data as follows: an augmented reality image can be generated by specifying an area that needs to be transparently processed based on additional depth data for each frame of a virtual reality image without the need for separate mask data.

Furthermore, the present invention is intended to provide a method and program for providing augmented reality images using depth data as follows: virtual reality content is deformed and applied as augmented reality content, without having to separately produce an augmented reality image that is enhanced and displayed in real space separately from the virtual reality image.

The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and those skilled in the art can clearly understand other technical problems not mentioned through the description below.

According to an embodiment of the present invention, a method for providing an augmented reality image using depth data includes: a virtual image data receiving step, in which the client receives virtual image data from a server, wherein the virtual image data includes color data and depth data; a display position determining step, in which the client determines the position where each pixel should be displayed in the real space based on the depth data; and a virtual image data displaying step, in which the virtual image data is displayed on the display space based on the determined position.

Furthermore, as another embodiment, the depth data is included in a separate channel different from the color data channel for each pixel, and the depth data and the color data are transmitted in synchronization.

Furthermore, as another embodiment, the virtual image data is a two-dimensional image that is stored according to each pixel by acquiring depth data of each location when shooting or generating.

Furthermore, as another embodiment, the display position determination step includes the following steps: the client determines a specific depth as a transparency adjustment benchmark; and distinguishes the depth range based on the transparency adjustment benchmark to determine whether to perform transparent processing, wherein the transparency adjustment benchmark is a benchmark for setting the boundary of the content to be displayed on the screen.

Furthermore, as another embodiment, the depth range is a plurality of areas divided based on the plurality of depths by setting a plurality of depths according to the transparency adjustment standard.

Furthermore, as another embodiment, the virtual image data further includes acquired position data and image direction data, and the display position determination step includes the following steps: comparing the current position data acquired by the client and the acquired position data, and comparing the reproduction direction data acquired by the client and the image direction data; and adjusting the position of the pixels in the virtual image data based on the comparison result.

Furthermore, as another embodiment, the display position determination step further includes the following step: the client adjusts the color or chromaticity of each pixel in the virtual image data based on the light illumination direction in the real space.

Furthermore, as another embodiment, the following step is also included: when the client machine is a device that outputs combined image data that is a combination of real image data obtained by a camera and the virtual image data, the client machine corrects the real image data based on an output delay time, wherein the output delay time is the time required from the time the real image data is captured to the time it is output on the screen.

According to another embodiment of the present invention, a program for providing an augmented reality image using depth data is combined with a computer as hardware to execute the above-mentioned method for providing an augmented reality image using depth data and store it in a medium.

According to another embodiment of the present invention, an augmented reality image providing device using depth data includes: a virtual image data receiving unit, receiving virtual image data from a server, wherein the virtual image data includes color data and depth data; a control unit, determining a position where each pixel should be displayed in a real space based on the depth data; and an image output unit, displaying the virtual image data on a display space based on the determined position.

According to another embodiment of the present invention, a method for enhancing reality images using depth data includes the following steps: the server obtains color data of each pixel of the virtual image data provided to the client at a predetermined time point; obtains depth data of each pixel of the virtual image data and stores it; and the server synchronizes the color data and the depth data and transmits them to the client, wherein the depth data is data used when correcting based on the first virtual image data at the first time point if the second virtual image data at the second time point is not received, and the second time point is a time point that passes through the virtual image data transmission cycle from the first time point.

Furthermore, as another embodiment, the virtual image data also includes acquired position data and image direction data, the client compares the current position data with the acquired position data, and compares the reproduced direction data with the image direction data, and adjusts the position of the pixels in the virtual image data based on the comparison result, the current position data and the reproduced direction data are data acquired by the client in real time or per unit time.

According to the present invention as above, there are the following multiple effects:

First, in the case where the spatial layout of objects (e.g., markers) to be combined with augmented reality images changes due to user movement, corrected image data is provided that can replace image frames that are missing at a specific point in time, so that the augmented reality images are displayed at the exact location in the real space so that they can be reproduced naturally without shaking.

Second, the depth data is combined with the two-dimensional virtual image data composed of color data, so that the two-dimensional virtual image data can be naturally displayed in the three-dimensional real space. That is, the enhanced reality image reproduction device recognizes the three-dimensional real space and displays each pixel of the two-dimensional image at an appropriate depth, thereby achieving a three-dimensional enhanced reality effect.

Third, since the level of transparency processing can be determined based on the depth data applied to each pixel when producing virtual image data for virtual display, virtual reality images can be directly applied as augmented reality based on the depth data obtained when the virtual image data is obtained, without producing a separate augmented reality image.

In order to make the above features and advantages of the present invention more clearly understood, embodiments are specifically cited below and described in detail with reference to the accompanying drawings.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The advantages, features and methods for implementing the present invention can be clearly understood by referring to the embodiments described in detail below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and it can be implemented in various forms that are different from each other. The present embodiments are provided only to make the disclosure of the present invention complete and to fully inform the scope of the present invention to those who have basic knowledge in the technical field to which the present invention belongs. The present invention is only defined by the scope of the claims. Throughout the specification, the same figure numbers refer to the same components.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meanings that can be commonly understood by people with basic knowledge in the technical field to which the present invention belongs. In addition, for terms defined in commonly used dictionaries, unless otherwise clearly and specifically defined, they should not be idealized or over-interpreted.

The terms used in this specification are used to illustrate the embodiments and are not intended to limit the present invention. In this specification, unless otherwise specified, the singular form also includes the plural form in the sentence. The terms "comprises" and/or "comprising" used in the specification do not exclude the existence or addition of one or more other constituent elements other than the mentioned constituent elements.

In this manual, "virtual image data" refers to image data produced to realize virtual reality or augmented reality. "Virtual image data" can be generated by photographing real space with a camera, or it can be produced through a modeling process.

In this specification, "first virtual image data" means virtual image data provided from a server to a client at a first time point. In this specification, "second virtual image data" means virtual image data provided from a server to a client at a second time point (i.e., a time point that is a unit time of an image receiving cycle from the first time point).

In this specification, "depth data" refers to a value for depth in three-dimensional space, and is a value assigned to each of divided detail units (for example, each pixel) in specific virtual image data.

In this specification, "color data" refers to data on the color of virtual image data displayed on the screen. For example, color data can be included according to the pixel of virtual image data. In addition, "color data" can be constructed using a predetermined color model that can express colors such as RGB (Red-Green-Blue) color model.

In this manual, "real image data" refers to image data obtained by photographing real space.

In this specification, "client" refers to an augmented reality image reproduction device that receives virtual image data from a server and reproduces it. In other words, "client" refers to all devices that can provide real images directly to the user's eyes or can simultaneously display real image data obtained by shooting real space and augmented reality content to provide.

Below, the method and program for providing an enhanced reality image using depth data according to an embodiment of the present invention are described in detail with reference to the accompanying drawings.

FIG1 is a structural diagram of an enhanced reality imaging system according to an embodiment of the present invention.

FIG2 is a flow chart of a method for providing an enhanced reality image using depth data according to an embodiment of the present invention.

1 and 2 , the method for providing an enhanced reality image using depth data according to an embodiment of the present invention includes the following steps: the client 200 receives virtual image data from the server 100 (S120: virtual image data receiving step); the client 200 determines the position of each pixel to be displayed in the real space based on the depth data (S140: display position determination step); and displays the virtual image data on the real space based on the determined position (S160: virtual image data display step). The following describes a detailed description of each step.

The client 200 receives virtual image data from the server 100 (S120: virtual image receiving step). The virtual image data includes color data and depth data.

As an embodiment, the virtual image data is a two-dimensional image that is stored for each pixel by obtaining the depth data of each location when shooting or generating. That is, the client 200 receives the virtual image data as a two-dimensional image including the depth data and generates it as an enhanced reality image in a three-dimensional real space. Through this, the client 200 can realize an augmented reality with a sense of reality in a three-dimensional real space without receiving a high-capacity three-dimensional image data modeled in three dimensions from the server 100.

As an embodiment, the depth data is included in a separate channel for each pixel that is distinguished from the color data channel. That is, the server 100 transmits the color data and the depth data for each pixel of the virtual image data to the client 200 through the color data channel and the depth data channel. At this time, the server 100 synchronizes the depth data transmitted through the depth data channel and the color data transmitted through the color data channel and transmits them to the client 200. The client 200 obtains the color data and the depth data for each pixel simultaneously received through the color data channel and the depth data channel as the virtual image data at that time point.

The client 200 determines the position of each pixel to be displayed in the real space based on the depth data (S140: display position determination step). That is, the client 200 determines to display the color data at the depth corresponding to the depth data at the real space position corresponding to each pixel position of the virtual image data and allowed to be seen by the user through the client 200. Specifically, in the case where the client 200 is a device that outputs virtual image data on a transparent screen (for example, a glass-type wearable device), the virtual image data corresponds to the screen size of the device, so that the client 200 outputs each pixel of the virtual image data at each corresponding location on the screen based on the depth data.

Furthermore, as another embodiment, as shown in FIG3 , the display position determination step (S140) includes the following steps: the client 200 determines a specific depth as a transparency adjustment reference (S141); and distinguishes a depth range based on the transparency adjustment reference to determine whether to perform transparent processing (S142). That is, in order to output only a desired portion of the desired virtual image data in the real space, the client 200 specifies an output area based on the depth data.

First, the client 200 determines a specific depth as a transparency adjustment criterion (S141). The transparency adjustment criterion may be a depth value corresponding to a boundary line of the content displayed on the screen. For example, as shown in FIG. 4 , when the client 200 desires to display only a character contained in the virtual image data as augmented reality content, the client 200 determines the depth data of the pixel corresponding to the boundary line of the character as the transparency adjustment criterion. Through this, the client 200 can output only the character (e.g., object 2) in the virtual image data to the screen, and the remaining area can be processed as transparent.

Subsequently, the client 200 distinguishes the depth range based on the transparency adjustment standard (S142). That is, the client 200 divides the area into an area closer to and farther from a predetermined depth. The client 200 may also set multiple depths as the transparency adjustment standard. For example. As shown in FIG. 4 , if two depth values (for example, two depth values of depth value A and depth value B (B is a value greater than A)) are set as the transparency adjustment standard, the client 200 divides the depth range into a first area closer to depth A, a second area between depth A and depth B, and a third area farther from depth B.

Then, the client 200 determines whether to perform transparent processing for the distinguished depth range (S142). As one embodiment, when the depth range is distinguished based on a predetermined depth value (for example, depth value A), the client 200 determines the area farther than the depth A as a transparent processing area, so that only the image of the range closer than the depth A is displayed on the screen. And, as another embodiment, when more than three depth ranges are distinguished due to setting more than two transparency adjustment criteria, the client 200 determines whether to process each depth range as transparent. In addition, the client 200 also determines the transparency value for each depth range and processes the image content contained in a specific depth range as semi-transparent so that it can be seen simultaneously with the real space.

By this, the client 200 can extract only the desired content (e.g., an object) in the virtual image data through the depth data and display it on the real image data. That is, the client sets the mask area using the depth data received at the same time in order to display the virtual image at an appropriate position in the real space, so that the client 200 does not need to receive the mask data for masking a part of the area in the virtual image data separately from the server 100. For example, in the case where depth data is only added to an image generated to realize an existing virtual reality image and applied to realize an augmented reality image, the service provider only generates depth data for each pixel synchronized with the existing virtual reality image, and sets a specific depth corresponding to a specific object boundary in the virtual image data, so that the virtual reality image can be realized as an augmented reality image with a partial area being masked.

Furthermore, as another embodiment, as shown in FIG5 , the client 200 applies the region designation data together with the specific depth data as a transparency adjustment standard. The virtual image data may include another object (e.g., the second object in FIG5 ) having the same depth value as the object (e.g., the first object in FIG5 ) that is desired to be enhanced in the real space. At this time, if the area that is not subjected to transparent processing and is exposed to the real space is set only by the depth data, objects that should not be displayed may be displayed in the real space. Therefore, the client 200 receives data (i.e., region designation data) corresponding to the area within the two-dimensional screen that should be processed as transparent from the server 100, and can make the image content of the designated two-dimensional area only the image content contained in the depth range designated as the area to be displayed be displayed on the screen. The depth data can be used to distinguish in detail the portion that should be processed transparently within a specific designated area, so that the server 100 sets the area designation data in the shape of a designated approximate area (for example, a rectangular shape) and transmits it to the client 200.

Furthermore, as another embodiment, in the display position determination step (S140), when the client 200 determines that the direction in which the user is looking is inconsistent with the direction in which the virtual image data is captured, the client 200 may adjust the virtual image data to a form suitable for display on the screen.

To this end, the virtual image data may also include acquisition position data and image direction data. The acquisition position data indicates the camera position when shooting virtual image data or generating virtual image data (i.e., modeling). The image direction data indicates the direction in which the virtual image data is acquired at the acquisition position. For example, in the case of shooting virtual image data using a camera device, the image direction data is the direction in which the camera is facing at a specific acquisition position, and can be acquired by a sensor included in the camera device. And, for example, in the case where the external server 100 generates virtual image data through a modeling process, the image direction data can be a direction determined for the virtual image data when modeling based on the acquisition position.

As shown in FIG6 , an embodiment of a display position determination step (S140) using the acquired position data and the image direction data includes the following steps: comparing the current position data acquired by the client 200 with the acquired position data, comparing the reproduction direction data acquired by the client 200 with the image direction data (S143); and adjusting the position of the pixel in the virtual image data based on the comparison result (S144).

First, the client 200 compares the current position data with the acquired position data, and compares the reproduction direction data acquired by the client 200 with the image direction data (S143). The current position data is position data acquired by the client 200 based on at least one of a plurality of position measurement methods. The reproduction direction data is data acquired by a motion sensor inside the client 200, and is data for the real-time direction toward which the client 200 is facing. The client 200 compares the acquired position and shooting direction (i.e., image direction) of the virtual image data with the current position and the facing direction (i.e., reproduction direction) to calculate a difference value. That is, the client 200 calculates the difference between the position where the image is acquired and the current position (e.g., a change in position in space), and calculates the difference between the orientation direction of the client 200 and the direction where the virtual image data is acquired (e.g., a difference in orientation value).

Then, the client 200 adjusts the position of the pixel in the virtual image data based on the comparison result (S144). The client 200 adjusts the position of each pixel of the virtual image data to conform to the current position data and the reproduction direction data. The client 200 can reflect the comparison result not only in the position of each pixel on the two-dimensional screen, but also in the depth data of each pixel to adjust the depth displayed in the real space.

Through this, even if the position and direction at which the user uses the client 200 to view the augmented reality image in real space are not exactly consistent with the position and direction at which the virtual image data is acquired, the client 200 can adjust the display position of each pixel in the virtual image data within the screen and display the augmented reality content at the exact position.

Furthermore, as shown in FIG7 , in a case where a portion of frames continuously provided at unit time intervals within the virtual image data is missing (for example, in a case where the second virtual image data (i.e., the second frame) that should be provided at a second time point after receiving the first virtual image data (i.e., the first frame) at a first time point is missing), the client 200 compares the acquired position data and the image direction data contained in the first virtual image data with the current position data and the reproduction direction data acquired by the client 200 at the second time point, and adjusts the position of each pixel within the first virtual image data based on the comparison result (i.e., the spatial position difference and the gaze direction difference). The second time point is a time point that passes through a virtual image data transmission cycle from the first time point. That is, the client 200 moves each pixel having color data and depth data included in the first frame (i.e., the first virtual image data) based on a comparison result of the acquired position data and image direction data included in the first frame and the current position data and reproduction direction data acquired by the client 200 at the second time point to generate and provide the second corrected image data that replaces the missing second frame. By this, in the case where the virtual image data provided by the server 100 is missing, by correcting the previous virtual image data instead, augmented reality content can be naturally provided.

Specifically, if the virtual image data (i.e., the second virtual image data) of a specific reproduction time point (i.e., the second time point) is missing and reproduced, the augmented reality content cannot be displayed at the accurate position in the real space, and when the virtual image data is received after the unit time has passed (i.e., the third time point), the augmented reality content will suddenly move and cause the image to shake. That is, if the first virtual image data (i.e., the first frame) is directly changed to the third virtual image data (i.e., the third frame) due to the omission of the second frame, it will be displayed as a sudden movement of the object, which may induce dizziness in the user. By using the first virtual image data at the first time point to provide the corrected image data at the second time point, the unreceived image frame (i.e., the second virtual image data) can be replaced with the substitute frame (i.e., the second corrected image data based on the first virtual image data) to provide the user with an augmented reality content that is not shaken and is accurately positioned in the real space. In particular, in the case where the client 200 receives the virtual image data from the server 100 through wireless communication and a large number of omissions occur, a more useful effect can be provided.

Furthermore, as another embodiment, as shown in FIG9 , the client 200 may generate and provide additional frames to fill in between frames (i.e., virtual image data) generated and provided by the server 100. For example, in the case where 60 frames per second are provided by the server 100, the client 200 may generate additional frames corresponding to the time points between the frames provided by the server 100 to output 120 frames per second. That is, due to various factors such as server performance or Internet bandwidth limitations, the server 100 may provide a limited number of frames (e.g., 60 frames per second) to the client 200. At this time, the client 200 may increase the number of frames per second by itself to generate a more natural image.

The additional frame between the first frame and the second frame (i.e., the 1.5th frame) is generated by correcting the first frame. That is, the client 200 compares the acquired position data and image direction data contained in the first frame (i.e., the first virtual image data) with the current position data and reproduction direction data acquired by the client 200 at the 1.5th time point when the frame is expected to be added, and generates a comparison result (i.e., a spatial position difference and a gaze direction difference). Subsequently, the client adjusts the position of each pixel in the first frame (i.e., the first virtual image data) based on the comparison result to generate a 1.5 frame. Furthermore, as another embodiment, the display position determination step (S140) further includes the following step: the client 200 adjusts the color or chromaticity of each pixel in the virtual image data. That is, the client 200 adjusts the color or chromaticity of each pixel based on the light irradiation direction of the real space or the layout position of each pixel. Through this, the client 200 can naturally display the virtual image data provided by the server 100 in the real space.

The client 200 displays the virtual image data in the real space based on the determined position (S160: virtual image data display step). As an embodiment, when the client 200 is a device (e.g., a glass-type wearable device) that directly views the real space through a transparent display while seeing the virtual image data displayed on the transparent display, the client 200 displays the augmented reality content (e.g., image content that has been masked for the virtual image data based on the depth number) at the position where each pixel is to be displayed on the transparent display.

Furthermore, as another embodiment, when the client 200 is a device that outputs combined image data combining real image data acquired by a camera and the virtual image data (for example, in the case of a smart phone or a tablet computer), the following step is further included: the client 200 corrects the real image data based on the output delay time. The output delay time is the time required from when the real image data is captured to when it is output on the screen. That is, the client 200 reflects the movement of the client 200 to correct the real image data, so that the real space that the client 200 is looking at at the same time point is consistent with the real image data displayed on the screen. For example, when the client 200 acquires real image data of the same size as the screen through a camera, the real image data is shifted and displayed on the screen based on the movement of the client 200. Also, for example, when the client 200 acquires real image data larger than the screen size, the client 200 extracts an area that needs to be output on the screen to reflect the movement of the client 200 from the acquired display image data and displays it.

1, an apparatus for providing an enhanced reality image using depth data according to another embodiment of the present invention includes a virtual image data receiving unit 210, a control unit 220, and an image output unit 230. The apparatus for providing an enhanced reality image according to an embodiment of the present invention corresponds to the client 200 in the method for providing an enhanced reality image using depth data according to an embodiment of the present invention. Hereinafter, detailed descriptions of the already described structures are omitted.

The virtual image data receiving unit 210 enables the client 200 to receive virtual image data from the server 100. The virtual image data is data including color data and depth data. The virtual image data receiving unit 210 may receive the color data and depth data of each pixel through separate channels. That is, the virtual image data receiving unit 210 receives the color data (e.g., RGB value) of each pixel through the color data channel, and receives the depth data of each pixel through the depth data channel.

The virtual image data receiving unit 210 receives the virtual image data through wired or wireless communication. In the case of receiving through wireless communication, the virtual image data receiving unit 210 may correspond to a wireless Internet module or a short distance communication module.

Wireless Internet module refers to a module used for wireless Internet connection, which can be built-in or external to mobile terminals. Wireless Internet technologies include Wireless LAN (WLAN) (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), Long Term Evolution (LTE), and LTE-A (Long Term Evolution-Advanced).

A short range communication module is a module used for short range communication. Short range communication technologies include Bluetooth, Bluetooth Low Energy (BLE), Beacon, Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, etc.

The control unit 220 plays a role in processing the virtual image data received by the virtual image data receiving unit 210. As an embodiment, the control unit 220 determines the position of each pixel that the client 200 should display in the real space based on the depth data.

Furthermore, the control unit 220 determines whether to perform transparent processing on pixels corresponding to each depth based on the depth data. That is, the control unit 220 processes pixels corresponding to the remaining depth range except for the object located in the specific depth range as transparent, so that only the specific object is displayed on the screen.

Furthermore, if the virtual image data of a specific reproduction time point (i.e., the second virtual image data of the second time point) is missing, the control unit 220 generates the correction image data to replace the missing image data using the previous virtual image data (i.e., the first virtual image data of the first time point). Furthermore, when the acquisition position or image direction of the virtual image data (i.e., the image acquisition direction) is inconsistent with the current position of the client 200 and the direction of the client 200 (i.e., the reproduction direction), the control unit 220 generates the correction image data to be corrected and arranged at an accurate position on the display space.

The image output unit 230 displays the virtual image data on the display space based on the determined position. For example, when the client 200 is a device (e.g., a glass-type wearable device) that directly views the real space through a transparent display and sees the virtual image data displayed on the transparent display, the image output unit 230 can be a transparent display that displays the augmented reality content generated based on the virtual image data.

FIG8 is a flow chart of a process for the server 100 to generate an augmented reality image including depth data for a client according to another embodiment of the present invention.

8 , a method for providing an augmented reality image using depth data according to another embodiment of the present invention includes the following steps: the server 100 obtains color data of each division unit of the virtual image data provided to the client 200 at a specific time point (S220); obtains depth data of each division unit of the virtual image data and stores it (S240); and the server 100 synchronizes the color data and the depth data and transmits them to the client 200 (S260).

The server 100 generates virtual image data to be provided to the client 200. The virtual image data includes a plurality of division units, and includes color data and depth data according to each division unit. The division unit may correspond to a pixel of the image data. The depth data is data used when correcting based on the first virtual image data at the first time point if the second virtual image data at the second time point is not received, and the second time point is a time point after the virtual image data transmission cycle from the first time point.

The server 100 obtains color data and depth data for each division unit (eg, pixel) (S220 and S240). The server 100 synchronizes the color data and the depth data and transmits them to the client 200 (S260). The server 100 transmits the synchronized color data and depth data to the client 200 through a separate channel.

Furthermore, as another embodiment, the virtual image data further includes acquired position data and image direction data, and the client machine 200 is characterized by comparing the current position data with the acquired position data, comparing the reproduced direction data with the image direction data, and adjusting the position of the pixel in the virtual image data based on the comparison result, and the current position data and the reproduced direction data are data acquired from the client machine 200 in real time or per unit time.

As described above, the method for providing an augmented reality image according to an embodiment of the present invention can be implemented as a program and stored in a medium to be executed in combination with a computer as hardware.

The aforementioned program may include a code (Code) encoded in a computer language such as C, C++, JAVA, machine language, etc., which can be read by the computer processor (CPU) through the device interface of the computer, so that the computer reads the program to execute the method implemented as a program. Such code may include functional code (Functional Code) related to functions that define the functions required to execute the method, and the functions may include control codes related to the execution steps required for the computer processor to execute in a predetermined sequence. In addition, such code may also include additional information required by the computer processor to execute the function or memory reference codes related to which location in the internal or external memory of the computer the media needs to refer to. Furthermore, when the processor of the computer needs to communicate with any other remote computer or server 100 in order to execute the function, the code may also include communication-related codes for: how to communicate with any other remote computer or server 100 using the communication module of the computer; what information or media needs to be transmitted during communication, etc.

The storage medium is not a medium for storing data for a short time such as a register, a cache memory, a memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device, but are not limited to this. That is, the program can be stored in various recording media on various servers 100 that can be accessed by the computer or in various recording media on the user's computer. In addition, the medium can be distributed on a computer system connected via a network and stored in a distributed manner according to computer-readable code.

According to the present invention as described above, there are the following various effects.

First, in the case where the spatial layout of objects (e.g., markers) to be combined with augmented reality images changes due to user movement, corrected image data is provided that can replace image frames that are missing at a specific point in time, so that the augmented reality images are displayed at the exact location in the real space so that they can be reproduced naturally without shaking.

Second, the depth data is combined with the two-dimensional virtual image data composed of color data, so that the two-dimensional virtual image data can be naturally displayed in the three-dimensional real space. That is, the enhanced reality image reproduction device recognizes the three-dimensional real space and displays each pixel of the two-dimensional image at an appropriate depth, thereby achieving a three-dimensional enhanced reality effect.

Third, since the level of transparency processing can be determined based on the depth data applied to each pixel in producing virtual image data for virtual display, virtual reality images can be directly applied as augmented reality based on the depth data obtained when the virtual image data is obtained, without producing a separate augmented reality image.

The embodiments of the present invention are described above with reference to the attached drawings, but it is understood by those with ordinary knowledge in the technical field to which the present invention belongs that the present invention can be implemented in other specific forms without changing the technical ideas or essential features of the present invention. Therefore, the embodiments described above should be understood as being exemplary in all aspects, rather than restrictive.

100: Server 200: Client 210: Virtual image data receiving unit 220: Control unit 230: Image output unit A, B: Depth value S120, S140, S160, S141, S142, S143, S144, S220, S240, S260: Steps

FIG. 1 is a structural diagram of an enhanced reality image system according to an embodiment of the present invention. FIG. 2 is a flow chart of a method for providing an enhanced reality image using depth data by a client according to an embodiment of the present invention. FIG. 3 is a flow chart of a process for adjusting the transparency of virtual image data based on depth data according to an embodiment of the present invention. FIG. 4 is an example diagram of setting a transparency adjustment benchmark and dividing regions based on depth data according to an embodiment of the present invention. FIG. 5 is an example diagram of inputting region designation data according to an embodiment of the present invention. FIG. 6 is a flow chart of a process for adjusting the spatial position of each pixel according to an embodiment of the present invention. FIG. 7 is an example diagram of generating a missing second frame based on color data and depth data of a first frame according to an embodiment of the present invention. FIG8 is a flow chart of a method for a server to generate and provide virtual image data including depth data according to an embodiment of the present invention. FIG9 is an example diagram showing a process of adding a 1.5th frame between a first frame and a second frame according to an embodiment of the present invention.

S120~S160: Steps

Claims (12)

A method for providing an enhanced reality image using depth data, as a method for providing an enhanced reality image for a client, comprises: a virtual image data receiving step, wherein the client receives virtual image data from a server, wherein the virtual image data includes color data and depth data; a display position determining step, wherein the client determines the position where each pixel should be displayed in the real space based on the depth data; and a virtual image data displaying step, wherein the virtual image data is displayed on the display space based on the determined position. A method for providing an enhanced reality image using depth data as described in claim 1, wherein the depth data is included in a separate channel different from the color data channel for each pixel, and the depth data and the color data are transmitted synchronously. A method for providing an augmented reality image using depth data as described in claim 1, wherein the virtual image data is a two-dimensional image stored per pixel by obtaining the depth data of each location when shooting or generating. The method for providing an augmented reality image using depth data as described in claim 1, wherein the display position determination step includes: The client determines a specific depth as a transparency adjustment benchmark; and Based on the transparency adjustment benchmark, the depth range is distinguished to determine whether to perform transparent processing, Wherein, the transparency adjustment benchmark is a benchmark for setting the boundary of the content to be displayed on the screen. A method for providing an enhanced reality image using depth data as described in claim 4, wherein the depth range is a plurality of areas divided based on the plurality of depths set according to the transparency adjustment standard. A method for providing an enhanced reality image using depth data as described in claim 1, wherein the virtual image data further includes acquired position data and image direction data, wherein the display position determination step includes: comparing the current position data acquired by the client and the acquired position data, and comparing the reproduction direction data acquired by the client and the image direction data; and adjusting the position of the pixel in the virtual image data based on the comparison result. The method for providing an enhanced reality image using depth data as described in claim 6, wherein the display position determination step further includes: The client adjusts the color or chromaticity of each pixel in the virtual image data based on the light illumination direction in the real space. The method for providing an enhanced real image using depth data as described in claim 1 further includes: In the case where the client is a device that outputs combined image data that combines real image data acquired by a camera and the virtual image data, the client corrects the real image data based on an output delay time, wherein the output delay time is the time required from when the real image data is captured to when it is output on the screen. A program for providing an augmented reality image using depth data, wherein, is combined with a computer as hardware and stored in a medium for executing the method of any one of claim 1 to claim 8. An enhanced reality image providing device using depth data, as a device for providing enhanced reality images, includes: A virtual image data receiving unit, receiving virtual image data from a server, wherein the virtual image data includes color data and depth data; A control unit, determining the position of each pixel to be displayed in the real space based on the depth data; and An image output unit, displaying the virtual image data on the display space based on the determined position. A method for providing an augmented reality image using depth data, as a method for a server to generate virtual image data for realizing augmented reality on a client, comprising: The server obtains color data of each pixel of the virtual image data provided to the client at a predetermined time point; The depth data of each pixel of the virtual image data is obtained and stored; and The server synchronizes the color data and the depth data and transmits them to the client, Wherein, the depth data is data used when correcting based on the first virtual image data at the first time point if the second virtual image data at the second time point is not received, Wherein, the second time point is a time point after the virtual image data transmission cycle from the first time point. A method for providing an enhanced reality image using depth data as described in claim 11, wherein, the virtual image data further includes acquisition position data and image direction data, the client compares the current position data with the acquisition position data, compares the reproduction direction data with the image direction data, and adjusts the position of the pixel in the virtual image data based on the comparison result, the current position data and the reproduction direction data are data acquired by the client in real time or per unit time.