KR20230161309A - An augmented reality device for obtaining depth information and a method for operating the same - Google Patents

Abstract

An augmented reality device that corrects depth values based on the direction of gravity measured by an IMU sensor and a method of operating the same are provided to obtain a depth map with high accuracy without an additional hardware module. An augmented reality device according to an embodiment of the present disclosure acquires a depth map from an image acquired using a camera, obtains a normal vector of at least one pixel from the depth map, and uses an IMU sensor. The direction of the normal vector of at least one pixel may be corrected based on the direction of gravity measured by an inertial measurement unit, and the depth value of the depth map may be corrected based on the direction of the corrected normal vector.

Description

Augmented reality device for acquiring depth information and method of operating the same {AN AUGMENTED REALITY DEVICE FOR OBTAINING DEPTH INFORMATION AND A METHOD FOR OPERATING THE SAME}

This disclosure relates to an augmented reality device that acquires depth information and a method of operating the same. Specifically, the present disclosure relates to an augmented reality device that corrects the depth value of each pixel of a depth map based on the direction of gravity and a method of operating the same.

Augmented Reality is a technology that overlays and displays virtual images on the physical environment space or real world objects of the real world. Augmented reality devices using augmented reality technology (for example, Smart glasses are useful in everyday life, such as information retrieval, route guidance, and camera photography. In particular, smart glasses are also worn as fashion items and are mainly used for outdoor activities.

Recently, in order to provide users with a sense of immersion, devices including a depth sensor that acquires depth information representing the sense of space of objects included in a real space composed of three-dimensional space have been widely used. Conventional depth information acquisition technology using a depth sensor includes, for example, Structured Light, Stereo Vision, or Time of Flight. Among the depth information acquisition technologies, the structured light method, stereo vision method, and ToF method are camera-based depth estimation methods, and as the distance from the camera increases, the accuracy of the depth value decreases. In the case of the structured light method or ToF method, which has relatively high accuracy of depth values regarding distance, additional hardware modules such as illuminators are required, and there is a problem of power consumption and additional costs incurred by the hardware modules.

In addition, most augmented reality applications run by augmented reality devices require depth information at all times, which increases power consumption, and due to the nature of augmented reality devices, which are portable devices with a small form factor, heat generation and power consumption increase during the time the device is available. It has a big impact.

In order to solve the above-described technical problem, the present disclosure provides an augmented reality device that corrects depth values based on the direction of gravity. An augmented reality device according to an embodiment of the present disclosure may include a camera, an IMU sensor (Inertial Measurement Unit), at least one processor, and memory. At least one processor may obtain a depth map from an image acquired using a camera by executing at least one command stored in a memory. At least one processor may obtain a normal vector of at least one pixel included in the depth map. At least one processor may correct the direction of the normal vector of the at least one pixel based on the direction of gravity measured by the IMU sensor 120. At least one processor may correct the depth value of the at least one pixel based on the direction of the corrected normal vector.

In order to solve the above-described technical problem, the present disclosure provides a method for an augmented reality device to correct depth values. In one embodiment of the present disclosure, the method may include obtaining a depth map from an image acquired using a camera. The method may include obtaining a normal vector of at least one pixel included in the depth map. The method may include correcting the direction of a normal vector of the at least one pixel based on the direction of gravity measured by an IMU sensor. The method may include correcting the depth value of the at least one pixel based on the direction of the corrected normal vector.

In order to solve the above-described technical problems, the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may store instructions related to obtaining a depth map from an image acquired using a camera. The storage medium may store instructions related to obtaining a normal vector of at least one pixel included in the depth map. The storage medium may store instructions related to correcting the direction of a normal vector of at least one pixel based on the direction of gravity measured by the IMU sensor. The storage medium may store instructions related to correcting the depth value of at least one pixel of the depth map based on the direction of the corrected normal vector.

The present disclosure may be readily understood by combination of the following detailed description and accompanying drawings, where reference numerals refer to structural elements.
FIG. 1A is a conceptual diagram illustrating how the augmented reality device of the present disclosure corrects a depth value.
FIG. 1B is a conceptual diagram illustrating how the augmented reality device of the present disclosure corrects the depth value according to the direction of gravity measured by the IMU sensor.
Figure 2 is a block diagram showing the components of an augmented reality device according to an embodiment of the present disclosure.
Figure 3 is a flowchart showing a method of operating an augmented reality device according to an embodiment of the present disclosure.
FIG. 4 is a flowchart illustrating a method for an augmented reality device to obtain a normal vector for each pixel according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an operation of an augmented reality device acquiring a normal vector for each pixel from a depth map according to an embodiment of the present disclosure.
FIG. 6A is a diagram illustrating an operation of an augmented reality device correcting the direction of a normal vector according to the direction of gravity according to an embodiment of the present disclosure.
FIG. 6B is a diagram illustrating an operation of an augmented reality device correcting the direction of a normal vector in a direction perpendicular to the direction of gravity according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating a training operation in which an augmented reality device acquires a depth map using an artificial intelligence model according to an embodiment of the present disclosure.
FIG. 8 is a flowchart illustrating a method by which an augmented reality device corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.
Figure 9 is a flowchart illustrating a method for an augmented reality device to calculate a loss value according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating an operation of an augmented reality device calculating a loss value according to an embodiment of the present disclosure.
FIG. 11 is a diagram illustrating an operation of an augmented reality device acquiring a depth map using an artificial intelligence model according to an embodiment of the present disclosure.
FIG. 12 is a flowchart illustrating a method by which an augmented reality device corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.
FIG. 13 is a diagram illustrating an operation in which an augmented reality device corrects the depth value for each pixel of a depth map through a plane sweep method according to an embodiment of the present disclosure.
FIG. 14 is a flowchart illustrating a method by which an augmented reality device corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.
FIG. 15 is a diagram illustrating an operation of an augmented reality device correcting the depth value for each pixel of a depth map obtained through a time-of-flight (ToF) method according to an embodiment of the present disclosure.
FIG. 16 is a diagram illustrating a spatial model restored with a depth map obtained in a general manner and a spatial model restored with a depth map acquired by an augmented reality device according to an embodiment of the present disclosure.

The terms used in the embodiments of the present specification are general terms that are currently widely used as much as possible while considering the function of the present disclosure, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. . In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant embodiment. Therefore, the terms used in this specification should not be defined simply as the names of the terms, but should be defined based on the meaning of the term and the overall content of the present disclosure.

Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described herein.

Throughout the present disclosure, when a part “includes” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary. In addition, terms such as "...unit" and "...module" used in this specification refer to a unit that processes at least one function or operation, which is implemented as hardware or software or as a combination of hardware and software. It can be implemented.

The expression “configured to” used in the present disclosure may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. It can be used interchangeably with ", "designed to," "adapted to," "made to," or "capable of." The term “configured (or set to)” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “system configured to” may mean that the system is “capable of” in conjunction with other devices or components. For example, the phrase "processor configured (or set) to perform A, B, and C" refers to a processor dedicated to performing the operations (e.g., an embedded processor), or by executing one or more software programs stored in memory. It may refer to a general-purpose processor (e.g., CPU or application processor) that can perform the corresponding operations.

In addition, in the present disclosure, when a component is referred to as “connected” or “connected” to another component, the component may be directly connected or directly connected to the other component, but in particular, the contrary It should be understood that unless a base material exists, it may be connected or connected through another component in the middle.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily practice them. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

FIG. 1A is a conceptual diagram illustrating a method by which an augmented reality device corrects a depth value according to an embodiment of the present disclosure.

Referring to FIG. 1A , the augmented reality device can estimate the depth value 20 of the actual floor surface 10 through the image acquired using the camera 110. The estimated depth value 20 may be estimated to be a different value from the depth value of the actual floor 10 depending on the positional relationship, such as the physical distance or height difference between the actual floor 10 and the camera 110. For example, the camera 110 includes a left-eye camera and a right-eye camera, and the augmented reality device uses the left-eye image acquired from the left-eye camera and the right-eye image acquired from the right-eye camera to obtain a depth value 20 through a stereo vision method. When estimating , the larger the distance from the camera 110, the larger the error in disparity, that is, the gap between corresponding points between the left-eye image and the right-eye image. For example, even when an augmented reality device estimates the depth value 20 through a Time-of-Flight (ToF) method, the greater the distance from the camera 110, the greater the accuracy of the estimated depth value 20. It falls.

In the embodiment shown in FIG. 1A, the estimated depth value 20 was obtained as a value similar to the depth value of the actual floor surface 10 in the area where the distance to the camera 110 is close, but the distance to the camera 110 is similar to the depth value of the actual floor surface 10. The farther away, the more the depth value can be obtained in the form of an upward slope, unlike the depth value of the actual bottom surface 10.

The augmented reality device can obtain information about the direction of gravity (G) based on the measurement value measured through the IMU sensor (Inertial Measurement Unit) 120. In one embodiment of the present disclosure, the IMU sensor 120 includes a gyroscope, and the augmented reality device can acquire gravity direction (G) information using the gyro sensor included in the IMU sensor 120. there is. The augmented reality device may obtain the corrected depth value 30 by correcting the estimated depth value 20 based on the direction of gravity (G). In the embodiment shown in FIG. 1A, the corrected depth value 30 may have a flat slope like the actual floor surface 10 and may have the same depth value as the actual floor surface 10. A specific method of correcting the depth value based on information about the direction of gravity (G) acquired by the augmented reality device through the IMU sensor 120 will be described in FIG. 1B.

FIG. 1B is a conceptual diagram illustrating how the augmented reality device of the present disclosure corrects the depth value according to the direction of gravity (G) measured by the IMU sensor.

Referring to FIG. 1B, the augmented reality device acquires the depth value of the pixel (operation ①). The augmented reality device may obtain a depth map including a plurality of pixels (p ₁ to p _n ) with a depth value based on the image acquired using the camera 110. . In one embodiment of the present disclosure, the camera 110 includes a left-eye camera and a right-eye camera, and the augmented reality device uses a left-eye image acquired from the left-eye camera and a right-eye image acquired from the right-eye camera to detect depth through a stereo vision method. You can obtain a map. In one embodiment of the present disclosure, the camera 110 is configured as a Time-of-Flight (ToF) camera, and the augmented reality device can acquire a depth map using the ToF camera. However, it is not limited to this, and in another embodiment of the present disclosure, the augmented reality device may acquire a depth map through structured light.

Depth values of pixels included in the depth map may have values different from those of the actual floor surface 10 based on the distance from the camera 110. In the embodiment shown in FIG. 1B, among the plurality of pixels (p ₁ to p _n ), the first pixel (p ₁ ) and the second pixel (p ₂ ) are located in an area (near) relatively adjacent to the camera 110. ) is the same or similar to the depth value of the actual floor surface 10, but the depth value of the pixels (p _n-1 , p _n ) located in the far area from the camera 110 is There may be differences from the depth value of the actual floor surface 10.

The augmented reality device acquires the normal vector (N ₁ to N _n ) of the pixel (operation ②). In one embodiment of the present disclosure, the augmented reality device divides the plurality of pixels (p ₁ to p _n ) into 3 based on the direction vector and depth value of the plurality of pixels (p ₁ to p _n ) included in the depth map. It can be converted to dimensional coordinates. The augmented reality device calculates the cross-product of the three-dimensional coordinate values of adjacent pixels arranged in adjacent positions in the up, down, left, and right directions based on one pixel, thereby dividing each of the plurality of pixels (p ₁ to p _n ). The normal vectors (N ₁ to N _n ) can be obtained.

The augmented reality device corrects the normal vectors (N ₁ to N _n ) based on the direction of gravity (operation ③). The augmented reality device includes an IMU sensor 120 (see FIG. 1A) and can obtain information about the direction of gravity (G) based on measurements measured using a gyroscope of the IMU sensor 120. there is. The augmented reality device may correct the direction of the normal vectors (N ₁ to N _n ) for each pixel according to the direction of gravity (G) or a direction perpendicular to the direction of gravity (G). As a result of the correction, the augmented reality device can obtain the corrected normal vectors (N ₁ ' to N _n ').

The augmented reality device corrects the depth value for each pixel based on the corrected normal vectors (N ₁ ' to N _n ') (operation ④). The augmented reality device may define a plane for each pixel according to the direction of the corrected normal vectors (N ₁ ' to N _n ') and correct the depth value of the pixel based on the plane for each pixel. In one embodiment of the present disclosure, the augmented reality device acquires a depth map using an artificial intelligence (AI) model, and pixels on a plane defined by the corrected normal vectors (N ₁ ' to N _n ') The loss value is calculated based on the depth values of adjacent pixels, and the depth value for each pixel can be corrected by performing training to apply the calculated loss value to the artificial intelligence model. In one embodiment of the present disclosure, the augmented reality device performs a plane sweep according to a plane for each pixel defined based on the direction of the corrected normal vectors (N ₁ ' to N _n '), thereby determining the depth for each pixel. The value can be corrected. In one embodiment of the present disclosure, a planar region is identified based on color information of an RGB image from a plane defined based on the corrected normal vectors (N ₁ ' to N _n '), and adjacent pixels within the identified plane region are identified. The depth value for each pixel can be corrected based on the depth value.

The augmented reality device may acquire a plurality of pixels (p ₁ ' to p _n ') having corrected depth values. Each of the plurality of pixels (p ₁ ' to p _n ') may have a depth value that is the same as or similar to the actual depth of the floor surface 10.

An image-based depth value acquisition method using an image acquired through the camera 110, for example, a structured light method, a stereo vision method, or a Time-of-Flight method using a camera ( There is a problem that the accuracy of the depth value decreases as the distance from 110) increases. In the case of the structured light method or ToF method, which has relatively high accuracy of depth values with respect to distance, additional hardware modules such as illuminators are required, and power consumption and additional costs may be incurred due to the hardware modules. In addition, since most augmented reality applications require depth information at all times, there is a problem in that the power consumption of the augmented reality device increases. Due to the nature of augmented reality devices, which are portable devices with a small form factor, heat generation and power consumption can greatly affect the time the device can be used.

The present disclosure uses the gravity direction (G) information acquired through the IMU sensor 120 (see FIG. 1A) to obtain a depth value with high accuracy without unnecessary power consumption by an additional hardware module (e.g., a light emitter module). The purpose is to provide an augmented reality device and a method of operating the same.

In the embodiment shown in FIGS. 1A and 1B, the augmented reality device corrects the direction of the normal vectors (N ₁ to N _n ) for each pixel based on the gravity direction (G) information acquired using the IMU sensor 120. And by correcting the depth value for each pixel based on the corrected normal vectors (N ₁ ' to N _n '), a depth map with high accuracy can be obtained and power consumption can be reduced. In addition, augmented reality devices generally include the IMU sensor 120 as an essential component, and the augmented reality device according to an embodiment of the present disclosure implements low power consumption while maintaining a small form factor, thereby enabling portability. And it can provide a technical effect of increasing device use time.

FIG. 2 is a block diagram showing components of an augmented reality device 100 according to an embodiment of the present disclosure.

The augmented reality device 100 may be glasses-shaped augmented reality glasses worn on the user's face. By executing an application, the augmented reality device 100 can provide not only real objects within a field of view (FOV) but also virtual image content displayed on a waveguide. The augmented reality device 100 includes, for example, a movie application, a music application, a photo application, a gallery application, a web browser application, an e-book reader application, a game application, an augmented reality application, an SNS application, a messenger application, By executing an object recognition application, etc., virtual image content displayed in each application can be provided to the user.

However, it is not limited to this, and the augmented reality device 100 is implemented as a Head Mounted Display Apparatus (HMD) or an Augmented Reality Helmet worn on the user's head. It could be. However, the augmented reality device 100 of the present disclosure is not limited to the above-described examples. In one embodiment of the present disclosure, the augmented reality device 100 includes mobile devices, smart phones, laptop computers, tablet PCs, e-readers, digital broadcasting terminals, PDAs (Personal Digital Assistants), It may be implemented in various devices such as Portable Multimedia Player (PMP), navigation, MP3 player, camcorder, Internet Protocol Television (IPTV), Digital Television (DTV), or wearable device.

Referring to FIG. 2 , the augmented reality device 100 may include a camera 110, an IMU sensor 120, a processor 130, and a memory 140. The camera 110, IMU sensor 120, processor 130, and memory 140 may each be electrically and/or physically connected to each other. In FIG. 2 , only essential components for explaining the operation of the augmented reality device 100 are shown, and the components included in the augmented reality device 100 are not limited as shown in FIG. 2 . In one embodiment of the present disclosure, the augmented reality device 100 may further include an eye tracking sensor or a display engine.

The camera 110 is configured to acquire an image about an object by photographing the object in real space. In one embodiment of the present disclosure, the camera 110 may include a lens module, an image sensor, and an image processing module. The camera 110 may acquire a still image or video obtained by an image sensor (eg, CMOS or CCD). The image processing module may process a still image or video acquired through an image sensor, extract necessary information, and transmit the extracted information to the processor 130.

In one embodiment of the present disclosure, the camera 110 may be a stereo camera that includes a left-eye camera and a right-eye camera and acquires a three-dimensional image of an object using two cameras. However, it is not limited to this, and the camera 110 may include a single camera or three or more multi-cameras.

In one embodiment of the present disclosure, the camera 110 irradiates light to an object, detects reflected light reflected from the object, and time of flight, which is the time difference between the time when the reflected light is detected and the time when the light is irradiated. It may be a ToF camera that acquires the depth value of the object based on ). When implemented as a ToF camera, the camera 110 can simultaneously acquire an RGB image (1500, see FIG. 15) and a depth map image (1510, see FIG. 15) of the object.

The IMU sensor (Inertial Measurement Unit) 120 measures the movement speed, direction, angle, and gravitational acceleration of the augmented reality device 100 through a combination of an accelerometer, a gyroscope, and a magnetometer. It is a sensor configured to measure. In one embodiment of the present disclosure, the IMU sensor 120 is a three-axis accelerometer that measures acceleration in the row, lateral, and height directions, and measures roll, pitch, and yaw angular velocities. It may include a three-axis angular velocity meter. In one embodiment of the present disclosure, the IMU sensor 120 may measure angular velocity using a gyro sensor and detect the direction of gravity based on the measured angular velocity. The IMU sensor 120 may provide information about the direction of gravity to the processor 130.

The processor 130 may execute one or more instructions of a program stored in the memory 140. The processor 130 may be comprised of hardware components that perform arithmetic, logic, input/output operations, and signal processing. The processor 130 may include, for example, a Central Processing Unit (Central Processing Unit), a microprocessor (microprocessor), a Graphic Processing Unit (Graphic Processing Unit), Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), and Digital Signal Processors (DSPDs). It may consist of at least one of Signal Processing Devices (PLDs), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but is not limited thereto.

In one embodiment of the present disclosure, the processor 130 may include an AI processor that performs artificial intelligence (AI) learning. AI processors may be manufactured in the form of dedicated hardware chips for artificial intelligence (AI), or as part of existing general-purpose processors (e.g. CPU or application processor) or graphics-specific processors (e.g. GPU) for augmented reality devices. It may also be mounted on (100).

Although the processor 130 is shown as one element in FIG. 2, it is not limited thereto. In one embodiment of the present disclosure, the processor 130 may be composed of one or more elements.

The memory 140 may be, for example, a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (e.g., SD or XD memory). etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), or It may be composed of at least one type of storage medium, such as an optical disk.

The memory 140 may store instructions related to functions or operations by which the augmented reality device 100 acquires depth value information of an object. In one embodiment of the present disclosure, the memory 140 includes at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor 130. can be saved. Instructions, algorithms, data structures, and program codes stored in memory 140 may be implemented in, for example, programming or scripting languages such as C, C++, Java, assembler, etc.

The memory 140 may store instructions, algorithms, data structures, or program codes related to the depth map acquisition module 142, the normal vector correction module 144, and the depth value correction module 146. The 'module' included in the memory 140 refers to a unit that processes functions or operations performed by the processor 130, and may be implemented as software such as instructions, algorithms, data structures, or program code. .

In the following embodiment, the processor 130 may be implemented by executing instructions or program codes stored in the memory 140.

The depth map acquisition module 142 is composed of instructions or program code related to the function and/or operation of acquiring a depth map for an object based on an image acquired using the camera 110. In one embodiment of the present disclosure, the depth map acquisition module 142 may be configured to acquire a depth map using an artificial intelligence model. The processor 130 may acquire a depth map by executing instructions or program code related to the depth map acquisition module 142. In one embodiment of the present disclosure, camera 110 may be a stereo camera including a left-eye camera and a right-eye camera. In this case, the processor 130 inputs the left eye image acquired using the left eye camera and the right eye image acquired using the right eye camera into the artificial intelligence model, and uses the artificial intelligence model to calculate the pixel luminance values of the left eye image and the right eye image. A depth map can be obtained by calculating disparity according to similarity. Artificial intelligence models can be implemented as deep neural network models. Deep neural network models include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or It can also be implemented with known artificial intelligence models such as Deep Q-Networks. The deep neural network model used to obtain the depth map may be, for example, DispNet, but is not limited thereto.

In one embodiment of the present disclosure, the processor 130 may obtain a depth map by calculating the disparity between the left-eye image and the right-eye image through a plane sweep method.

In one embodiment of the present disclosure, the camera 110 is configured as a ToF camera, and the processor 130 determines the time of flight of light, which is the time difference between the time when reflected light obtained using the ToF camera is detected and the time when the clock signal is input. (time of flight) can be calculated, and the distance between the location where the reflected light is detected and the augmented reality device 100, that is, the depth value, can be calculated through the calculation between the flight time and the speed of light. The processor 130 may acquire information about the location where light is irradiated and reflected light is detected, and obtain a depth map by mapping the calculated depth value to the obtained location information.

The normal vector correction module 144 has the function of acquiring a normal vector for each of a plurality of pixels included in the depth map and correcting the normal vector according to the direction of gravity obtained by the IMU sensor 120. Or, it consists of instructions or program code related to operation. The processor 130 may acquire the normal vector of at least one pixel and correct the direction of the obtained normal vector by executing instructions or program code related to the normal vector correction module 144. In one embodiment of the present disclosure, the processor 130 converts each of the plurality of pixels into a three-dimensional coordinate value based on the direction vector of the plurality of pixels and the depth value of the plurality of pixels included in the depth map, and the converted By calculating the cross-product of the three-dimensional coordinate values of adjacent pixels in the up, down, left, and right directions based on each pixel, a normal vector for each pixel can be obtained.

The processor 130 may obtain information about the direction of gravity based on a measurement value measured by the gyro sensor of the IMU sensor 120 and correct the direction of the normal vector for each pixel based on the direction of gravity. In one embodiment of the present disclosure, the processor 130 may correct the direction of the normal vector for each pixel not only in the direction of gravity but also in a direction perpendicular to the direction of gravity. For example, for a pixel having a normal vector close to the vertical direction, the processor 130 may correct the direction of the normal vector to a direction parallel to the direction of gravity. For example, for a pixel (e.g., a pixel representing an object such as a wall, pillar, etc.) having a normal vector close to the horizontal direction with respect to the ground, the processor 130 changes the direction of the normal vector in a direction perpendicular to the direction of gravity. It can be corrected.

A specific embodiment in which the processor 130 acquires a normal vector for each pixel and corrects the direction of the obtained normal vector will be described in detail with reference to FIGS. 4, 5, 6A, and 6B.

The depth value correction module 146 is comprised of instructions or program code related to a function and/or operation for correcting the depth value of at least one pixel of the depth map based on the direction of the corrected normal vector. The processor 130 may correct the depth value of at least one pixel by executing instructions or program code related to the depth value correction module 146. In one embodiment of the present disclosure, the processor 130 calculates the loss value of the depth map output during the training process of the artificial intelligence model and performs training to apply the calculated loss value to the artificial intelligence model. By performing this, the depth value of at least one pixel of the depth map can be corrected. The processor 130 may calculate a loss value based on the depth value of a pixel on a plane defined by the corrected normal vector and the depth values of adjacent pixels in position. In one embodiment of the present disclosure, the processor 130 acquires depth values of adjacent pixels based on a plurality of points where a defined plane and a ray vector of the camera 110 meet, and the depth value of the pixel and The difference value between the depth values of adjacent pixels is calculated respectively, and the loss value can be calculated through a weighted sum operation that applies a weight to the difference value calculated for each adjacent pixel.

In one embodiment of the present disclosure, the weight applied to the weighted sum operation is a first weight determined based on the distance between the location of the camera 110 and the location of adjacent pixels in the depth map and the pixel and adjacent pixels in the depth map. It may include a second weight determined based on the difference in luminance values between the two sides. The processor 130 calculates a loss value according to the positional relationship between the plane defined by the corrected normal vector and the camera 110, and trains an artificial intelligence model using the calculated loss value to obtain the depth value of at least one pixel. Specific embodiments for correcting will be described in detail in FIGS. 7 to 10.

In one embodiment of the present disclosure, processor 130 assumes a plane hypothesis for each pixel along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector, and performs a plane sweep according to the hypothesized plane. By performing a plane sweep, the depth value of at least one pixel can be obtained. The processor 130 may correct the depth value of at least one pixel of the depth map using the acquired depth value. A specific embodiment in which the processor 130 corrects the depth value of at least one pixel by performing a plane sweep along a plane defined based on the corrected normal vector will be described in detail with reference to FIGS. 12 and 13.

In one embodiment of the present disclosure, the processor 130 may acquire an RGB image and a depth map image using a ToF camera. The processor 130 may define a plane for each pixel in the depth map image based on the corrected normal vector and identify the plane area in the depth map image based on a region divided according to color information of the RGB image. The processor 130 may correct the depth value of the pixel to be corrected within the identified planar area based on the depth values of adjacent pixels within the same planar area. Here, the 'compensation target pixel' is a pixel whose depth value needs to be corrected, and the depth value difference between a pixel that does not have a depth value in the flat area (pixel for which the depth value has not been acquired) or adjacent pixels in the flat area is based on the pixel. This refers to pixels that exceed a set threshold. 14 and 14 for a specific embodiment in which the processor 130 identifies a planar area by an RGB image in a depth map image acquired using a ToF camera and corrects the depth value of at least one pixel within the identified planar area. This will be explained in detail in Figure 15.

FIG. 3 is a flowchart illustrating a method of operating the augmented reality device 100 according to an embodiment of the present disclosure.

In step S310, the augmented reality device 100 acquires a depth map from an image acquired using a camera. In one embodiment of the present disclosure, the camera may be a stereo camera including a left-eye camera and a right-eye camera, and the augmented reality device 100 uses a left-eye image acquired using the left-eye camera and the right-eye camera. A depth map can be obtained by inputting the acquired right-eye image into an artificial intelligence model and calculating disparity according to the similarity of pixel luminance values between the left-eye image and the right-eye image using the artificial intelligence model. Artificial intelligence models can be implemented as deep neural network models. The deep neural network model used to obtain the depth map may be, for example, DispNet, but is not limited thereto.

In one embodiment of the present disclosure, the augmented reality device 100 may obtain a depth map by calculating the disparity between the left-eye image and the right-eye image through a plane sweep method.

In one embodiment of the present disclosure, the camera is configured as a ToF camera, and the augmented reality device 100 can acquire a depth map based on the time of flight of light obtained using the ToF camera.

In step S320, the augmented reality device 100 obtains a normal vector of at least one pixel included in the depth map. In one embodiment of the present disclosure, the augmented reality device 100 may convert a plurality of pixels into 3D coordinate values based on the direction vector and depth value of each of the plurality of pixels included in the depth map. The augmented reality device 100 may obtain a normal vector for each pixel using the converted 3D coordinate value.

In step S330, the augmented reality device 100 corrects the direction of the normal vector of at least one pixel based on the direction of gravity measured by the IMU sensor. In the case of a pixel having a normal vector close to the vertical direction, the augmented reality device 100 may correct the direction of the normal vector to a direction parallel to the direction of gravity. In the case of a pixel having a normal vector close to the horizontal direction with respect to the ground, the augmented reality device 100 may correct the direction of the normal vector in a direction perpendicular to the direction of gravity.

In step S340, the augmented reality device 100 corrects the depth value of at least one pixel of the depth map based on the direction of the corrected normal vector. When a depth map is acquired using an artificial intelligence model in step S310, the augmented reality device 100 calculates a loss value of the depth map obtained in the training process of the artificial intelligence model, and calculates the calculated loss. The depth value of at least one pixel included in the depth map can be corrected by performing training that applies the value to the artificial intelligence model. The augmented reality device 100 may calculate a loss value based on the depth value of a pixel on a plane defined by the corrected normal vector and the depth values of adjacent pixels in position. In one embodiment of the present disclosure, the augmented reality device 100 acquires depth values of adjacent pixels based on a plurality of points where a defined plane and a ray vector of a camera meet, and the depth value of the adjacent pixel The difference value between the depth values of each pixel is calculated, and the loss value can be calculated through a weighted sum operation that applies a weight to the difference value calculated for each adjacent pixel. In one embodiment of the present disclosure, the augmented reality device 100 is based on a first weight determined based on the distance between the positions of adjacent pixels in the depth map and the position of the camera and the luminance value difference between the pixel in the depth map and the adjacent pixels. The loss value can be obtained through a weighted sum operation using the second weight determined based on the loss value.

In one embodiment of the present disclosure, the augmented reality device 100 may perform a plane hypothesis that defines a plane for each pixel along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector. there is. The augmented reality device 100 may obtain the depth value of at least one pixel by performing a plane sweep along a plane defined by the plane assumption. The augmented reality device 100 may correct the depth value of at least one pixel included in the depth map using the acquired depth value.

When a depth map is acquired using a ToF camera in step S310, the augmented reality device 100 may acquire an RGB image using the ToF camera. In one embodiment of the present disclosure, the augmented reality device 100 defines a plane for each pixel in the depth map based on the corrected normal vector, and defines a plane in the depth map based on a region divided according to color information of the RGB image. Areas can be identified. The augmented reality device 100 may correct the depth value of the pixel to be corrected within the identified plane area based on the depth values of adjacent pixels within the same plane.

FIG. 4 is a flowchart illustrating a method for the augmented reality device 100 to obtain a normal vector for each pixel according to an embodiment of the present disclosure.

Steps S410 and S420 shown in FIG. 4 are steps that embody step S320 shown in FIG. 3. Step S410 shown in FIG. 4 may be performed after step S310 of FIG. 3 is performed. After step S420 shown in FIG. 4 is performed, step S330 of FIG. 3 may be performed.

FIG. 5 is a diagram illustrating an operation of the augmented reality device 100 acquiring a normal vector for each pixel from a depth map according to an embodiment of the present disclosure.

Hereinafter, the operation of the augmented reality device 100 will be described with reference to FIGS. 4 and 5 together.

Referring to step S410 of FIG. 4, the augmented reality device 100 converts the pixels into 3D coordinate values based on the direction vectors of the pixels and the depth values of the pixels included in the depth map. Referring to FIG. 5 together, the processor 130 (see FIG. 2) of the augmented reality device 100 calculates the direction vector and depth value (D _i,j ) of the first pixel 501 from the depth map 500, and a plurality of Direction vectors and depth values (D _{i-1, j} , D _{i+1, j} , D _{i, j-1} , D _{i, j+1} ) of each of the adjacent pixels 502 to 505 may be obtained. In the embodiment shown in FIG. 5, the first pixel 501 is a pixel arranged in the i-th column and j row (j-th row) among the plurality of pixels included in the depth map 500. It represents and can have a depth value of D _i,j . The plurality of adjacent pixels 501 to 505 are pixels arranged adjacent to each other in the up, down, left, and right directions with respect to the first pixel 501, and are respectively D _{i-1, j} , D _i+1 , _j , D _{i, j.} It may have depth values of _-1 , and D _{i, j+1} . For example, the second pixel 502 is a pixel placed in column i-1, row j, and has a depth value of D _{i-1, j} , and the third pixel 503 is a pixel placed in column i+1, row j. It is a pixel placed in and may have a depth value of D _i+1,j . The fourth pixel 504 and the fifth pixel 505 may have depth values of D _{i, j-1} and D _{i, j+1} , respectively.

The processor 130 stores the direction vector and depth value (D _i,j ) of the first pixel 501 and the direction vector and depth value (D _{i-1, j} , D) of each of the plurality of adjacent pixels 502 to 505. Based on _{(i+1, j} , D _{i, j-1} , D _{i, j+1} ), the first pixel 501 and the plurality of adjacent pixels 502 to 505 are displayed in three dimensions in the three-dimensional space 510. It can be converted to coordinate values (P _i,j , P _{i-1, j} , P _{i+1, j} , P _{i, j-1} , P _{i, j+1} ). In one embodiment of the present disclosure, the processor 130 converts the first pixel 501 and the plurality of adjacent pixels 502 to 505 into three-dimensional coordinate values in the three-dimensional space 510 through Equation 1 below: can do.

In the embodiment shown in FIG. 5, the processor 130 converts the first pixel 501 into P _{i and j} through the operation of Equation 1, and converts the plurality of adjacent pixels 502 to 505 into P _i They can be converted to _{-1, j} , P _{i+1, j} , P _{i, j-1} , and P _{i, j+1} respectively.

Referring to step S420 of FIG. 4, the augmented reality device 100 obtains a normal vector for each pixel by calculating the cross-product of the 3D coordinate values of a plurality of adjacent pixels in the up, down, left, and right directions. Referring to FIG. 5 together, the processor 130 of the augmented reality device 100 calculates the 3D coordinate values of a plurality of adjacent pixels in the up, down, left, and right directions based on the 3D coordinate value (P _i,j ) of the first pixel. The cross product of (P _{i-1, j} , P _{i+1, j} , P _{i, j-1} , P _{i, j+1} ) can be calculated. In one embodiment of the present disclosure, the processor 130 may obtain the normal vector (N _i,j ) for the first pixel through the operation of Equation 2.

In the embodiment shown in FIG. 5, the processor 130 calculates the three-dimensional coordinate values (P _{i-1, j)} of the second and third pixels adjacent to the first pixel in the left and right directions through the operation of Equation 2. , P _{i+1, j} ) and the difference between the three-dimensional coordinate values (P _{i, j-1} , P _{i, j+1} ) of the fourth and fifth pixels adjacent to the first pixel in the vertical direction. By calculating and calculating the vector product of the difference, the normal vector (N _i,j ) of the first pixel can be obtained. The normal vector (N _i,j ) may be a vector defining a plane composed of the first to fifth pixels.

FIG. 6A is a diagram illustrating an operation of the augmented reality device 100 according to an embodiment of the present disclosure to correct the direction of the normal vector (N _i,j ) according to the direction of gravity (G).

Referring to FIG. 6A , the augmented reality device 100 may include an IMU sensor 120, a processor 130, and a normal vector correction module 144. The components shown in FIG. 6A show only essential components for explaining the operation of the augmented reality device 100 to correct the direction of the normal vector (N _i,j ), and are components included in the augmented reality device 100. This is not limited as shown in FIG. 6A.

The IMU sensor 120 may acquire information about the direction of gravity (G) based on measurements made using a gyroscope and provide information about the direction of gravity (G) to the processor 130. there is. The processor 130 executes the instructions or program code of the normal vector correction module 144 to determine the normal vector (N _i,j ) based on the information about the gravity direction (G) obtained from the IMU sensor 120. The direction can be corrected. In one embodiment of the present disclosure, the processor 130 determines whether the direction of the normal vector (N _i,j ) is close to the vertical direction or the horizontal direction, and if close to the vertical direction, the direction of gravity (G) The direction of the normal vector (N _i,j ) can be corrected according to the gravity normal vector (-N _G ) having a direction parallel to . In the embodiment shown in FIG. 6A, since the direction of the normal vector (N _i,j ) is close to the vertical direction, the processor 130 changes the direction of the normal vector (N _i,j ) to the gravity normal vector (-N _G ) It can be corrected in a parallel direction according to . As a result of the correction, the processor 130 may obtain the corrected normal vector (N' _i,j ).

The processor 130 may correct the direction of the plane defined by the 3D coordinate value (P i _,j ) of the first pixel according to the direction of the corrected normal vector (N' i _,j ). In one embodiment of the present disclosure, the processor 130 may correct the direction of the plane by the angle difference between the corrected normal vector (N' _i,j ) and the normal vector (N _i,j ) before correction.

FIG. 6B is a diagram illustrating an operation of the augmented reality device 100 according to an embodiment of the present disclosure to correct the direction of the normal vector (N _i,j ) in a direction perpendicular to the direction of gravity (G).

The embodiment shown in FIG. 6B is the same as the embodiment shown in FIG. 6A except for the direction of the normal vector (N _i,j ) of the first pixel and the direction of the corrected normal vector (N' _i,j ). , Redundant explanations are omitted.

The processor 130 executes the instructions or program code of the normal vector correction module 144 to determine the normal vector (N _i,j ) based on the information about the gravity direction (G) obtained from the IMU sensor 120. The direction can be corrected. In one embodiment of the present disclosure, the processor 130 may determine whether the direction of the normal vector (N _i,j ) is close to the vertical direction or close to the horizontal direction. Pixels whose direction of the normal vector (N _i,j ) is close to the horizontal direction may be, for example, pixels representing walls, pillars, etc., or pixels representing objects arranged in the vertical direction. If it is determined that the normal vector (N _i,j ) is close to the horizontal direction, the processor 130 may correct the direction of the normal vector (N _i,j ) according to the direction perpendicular to the direction of gravity (G). In the embodiment shown in FIG. 6B, the direction of the normal vector (N _i,j ) is close to the horizontal direction, so the processor 130 sets the direction of the normal vector (N _i,j ) perpendicular to the gravity direction (G). The direction of the normal vector (N _i,j ) can be corrected. As a result of the correction, the processor 130 may obtain the corrected normal vector (N' _i,j ).

The processor 130 may correct the direction of the plane defined by the 3D coordinate value (P i _{,j) of the first pixel according to the direction of the corrected normal vector (N' i ,j} ). In one embodiment of the present disclosure, the processor 130 may correct the direction of the plane by the angle difference between the corrected normal vector (N' _i,j ) and the normal vector (N _i,j ) before correction.

FIG. 7 is a diagram illustrating a training operation in which the augmented reality device 100 acquires a depth map 730 using an artificial intelligence model 700 according to an embodiment of the present disclosure.

Referring to FIG. 7, the augmented reality device 100 may include a stereo camera consisting of a left-eye camera and a right-eye camera. The augmented reality device 100 may input the left eye image 710L acquired using the left eye camera and the right eye image 710R acquired using the right eye camera to the artificial intelligence model 700. The augmented reality device 100 obtains a depth map 730 by calculating disparity according to the similarity in luminance values between pixels between the left eye image 710L and the right eye image 710R using the artificial intelligence model 700. can do.

In one embodiment of the present disclosure, the artificial intelligence model 700 may be implemented as a deep neural network model. Deep neural network models include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or It can also be implemented with known artificial intelligence models such as Deep Q-Networks. For example, the deep neural network model used to obtain the depth map 730 may be DispNet, but is not limited thereto.

For example, when the artificial intelligence model 700 is implemented as a convolutional neural network model such as DispNet, the artificial intelligence model 700 creates a feature map by convolving each of the left eye image 701L and the right eye image 710R. map) can be extracted. In the feature map extraction process for each of the left eye image 710L and the right eye image 710R, weights may be shared with the same value. The artificial intelligence model 700 accumulates the difference values between the feature maps extracted from the left eye image 710L and the right eye image 710R, respectively, as a 3D cost volume, and up-convolves the accumulated 3D cost volume. It can be trained to output the depth map 730 by performing up-convolution hierarchical refinement. In one embodiment of the present disclosure, the IMU sensor measurement value 720 is applied as an input to the artificial intelligence model 700, and the IMU sensor measurement value 720 may be input to the three-dimensional cost volume. An error may exist between the depth map 730 output during the training process and the groundtruth of the disparity according to the similarity of the pixel luminance values of the left eye image 710L and the right eye image 710R, and augmentation The real-world device 100 may train the artificial intelligence model 700 using the loss value 740 to correct errors in the depth map 730.

The augmented reality device 100 acquires information about the direction of gravity based on the IMU sensor measurement value 720, and generates a normal vector (N'i _,j , FIG. 6A) corrected according to the direction of gravity or a direction perpendicular to the direction of gravity. The loss value 740 can be calculated based on the depth values of pixels on the plane defined by (see and FIG. 6B). A specific embodiment in which the augmented reality device 100 calculates the loss value 740 will be described in detail with reference to FIGS. 8 to 10.

FIG. 8 is a flowchart illustrating a method by which the augmented reality device 100 corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.

Steps S810 and S820 shown in FIG. 8 are steps that embody step S340 shown in FIG. 3. Step S810 shown in FIG. 8 may be performed after step S330 of FIG. 3 is performed.

In step S810, the augmented _reality device 100 generates a loss value ( loss) (740, see FIG. 7) is calculated. A specific embodiment in which the augmented reality device 100 calculates the loss value 740 will be described in detail with reference to FIGS. 9 and 10.

FIG. 9 is a flowchart illustrating a method by which the augmented reality device 100 calculates a loss value 740 according to an embodiment of the present disclosure.

Steps S910 to S940 shown in FIG. 9 are steps that embody step S810 shown in FIG. 8. After step S940 shown in FIG. 9 is performed, step S820 of FIG. 8 may be performed.

FIG. 10 is a diagram illustrating an operation of the augmented reality device 100 to calculate a loss value 740 according to an embodiment of the present disclosure. Hereinafter, step S810 of FIG. 8 will be described with reference to FIGS. 9 and 10 together.

Referring to step S910 of FIG. 9 , the augmented reality device 100 defines a plane composed of a pixel having a corrected normal vector and a plurality of adjacent pixels positionally adjacent to the pixel. Referring to FIG. 10 together, the processor 130 (see FIG. 2) of the augmented reality device 100 calculates the three-dimensional coordinate value (P _{i ,j) of the first pixel having the corrected normal vector (N' i,j} ). and 3D coordinate values (P _{i-1, j, P i+1, j} , P _{i, j-1} , P _{i, j+)} of _{a plurality} of adjacent pixels arranged in positions adjacent to the first pixel in the up, down, left, and right directions. ₁ ) can be defined as a plane consisting of The 3D coordinate value of the first pixel (P _i,j ) and the 3D coordinate value of each of the plurality of adjacent pixels (P _{i-1, j} , P _{i+1, j} , P _{i, j-1} , P _{i , j+1} ) may have different depth values.

In step S920 of FIG. 9 , the augmented reality device 100 acquires depth values of a plurality of adjacent pixels based on a plurality of points where a defined plane and a ray vector of a camera meet. Referring to FIG. 10 together, the processor 130 may identify a plurality of points where the plane defined in step S910 and the ray vector (R) of the camera 110 meet. The ray vector R of the camera 110 may be determined based on the direction of the camera 110 and a positional relationship including at least one of the distance or height between the camera 110 and the plane. The processor 130 determines the depth values (D' _i,j , D' _{i-1, j, D'i+1,j ,} D' _i,j-1 , D' _i,j+) of the identified plurality of points. ₁ ) can be obtained. In the embodiment shown in FIG. 10 , the first depth value (D' _i,j ) may be equal to the depth value of the 3D coordinate value (P _i,j ) of the first pixel. The second depth value (D' _{i-1, j} ) may have the same depth value as the 3D coordinate value (P _{i-1, j} ) of the second pixel. However, the location of the second point having the second depth value (D' _{i-1, j} ) may be different from the location of the 3D coordinate value (P _{i-1, j} ) of the second pixel. Likewise, the third depth value (D' _{i+1, j} ) has the same depth value as the three-dimensional coordinate value (P _{i+1, j} ) of the third pixel, but the location of the third point is 3 of the third pixel. It may be different from the location of the dimensional coordinate value (P _{i+1, j} ).

In step S930 of FIG. 9 , the augmented reality device 100 calculates a difference value between the depth value of a pixel and the depth values of a plurality of adjacent pixels. Referring to FIG. 10 together, the processor 130 calculates the depth value (D' _i,j ) of the first pixel and the depth values (D' _i-1,j , D) of the plurality of adjacent pixels based on Equation 3 below. The difference value (d _pq ) between ' _{i+1, j} , D' _{i, j-1} , D' _{i, j+1} ) can be calculated respectively.

In Equation 3, D' _p represents the depth value (D' _i,j ) of the first pixel, which is a reference pixel with a corrected normal vector (N'i _,j ), and D' _q is the positional difference between the first pixel and the first pixel. represents the depth value (D' _{i-1, j} , D' _{i+1, j} , D' _{i, j-1} , D' _{i, j+1} ) of each of a plurality of adjacent pixels. According to Equation 3, the processor 130 is D' _i,j - D' _{i-1, j} , D' _i,j - D' _{i+1, j} , D' _i,j - D' _{i, j-} d _pq can be obtained by calculating ₁ , and D' _i,j - D' _{i, j+1} , respectively.

In step S940 of FIG. 9, the augmented reality device 100 obtains a loss value through a weighted sum operation that applies a weight to the calculated difference value. Referring to Figure 10, the processor 130 calculates the loss value (Loss _g ) through a weighted sum operation that applies the first weight (w _q ^D ) and the second weight (w _pq ^C ) to the calculated difference value d _pq . can be obtained. In one embodiment of the present disclosure, the loss value (Loss _g ) can be calculated by Equation 4 below.

In Equation 4, the first weight (w _q ^D ) may be determined based on the distance between the positions of a plurality of adjacent pixels in the depth map 730 (see FIG. 7) and the position of the camera 110. In one embodiment of the present disclosure, the first weight (w _q ^D ) is the depth value (D _{i-1, j} , D _{i+1, j)} of a plurality of adjacent pixels in the depth map 730 as shown in Equation 5 below: , D _{i, j-1} , D _{i, j+1} ).

Referring to Equation 5, D _q is the depth value (D _{i-1, j} , D _{i+1, j} , D _{i, j-1} , D _{i, j+)} of a plurality of adjacent pixels in the depth map 730. ₁ ), and the smaller the depth value (D _{i-1, j} , D _{i+1, j} , D _{i, j-1} , D _{i, j+1} ) of the plurality of adjacent pixels, the smaller the first weight ( The value of w _q ^D ) increases. Depth values (D _{i-1, j} , D _{i+1, j} , D _{i, j-1} , D _{i, j+1} ) of a plurality of adjacent pixels, that is, the distance between the adjacent pixels and the camera 110 is close. As the value increases, the value of the first weight (w _q ^D ) may increase. In the depth map 730 extraction method using a stereo camera, the accuracy of the depth value decreases as the distance increases, so Equation 5 is designed to give more weight to pixels that are closer to the camera 110.

In Equation 4, the second weight (w _pq ^C ) may be determined based on the difference in pixel luminance values between the first pixel and a plurality of adjacent pixels in the depth map 730. In one embodiment of the present disclosure, the second weight w _pq ^C may be calculated based on the difference in luminance values between pixels in the depth map 730 as shown in Equation 6 below.

Referring to Equation 6, I _p represents the luminance value of the first pixel in the depth map 730, and I _q represents the luminance value of a plurality of adjacent pixels. As the difference between the luminance value of the first pixel and the luminance values of the plurality of adjacent pixels decreases, the value of the second weight w _pq ^C increases. The more similar the luminance values of pixels on the depth map 730, the higher the probability of having the same or similar depth values, so Equation 6 is designed to give more weight to the difference between the luminance values of pixels in the depth map 730. .

Referring again to FIG. 8, in step S820, the augmented reality device 100 performs training to apply the calculated loss value (Loss _g , see FIG. 10) to the artificial intelligence model to determine the depth of at least one pixel of the depth map. Correct the value.

8-10, the loss value (Loss _g ) is the plane defined by the corrected normal vector (N'i _,j , see FIGS. 6A and 6B) and the ray vector of the camera 110. To be determined based on the depth value (D' _i,j , D' _{i-1, j} , D' _{i+1, j} , D' _{i, j-1} , D' _{i, j+1} ) by (R) You can. In particular, the loss value (Loss _g ) is a first weight (w _q ^D ) determined based on the distance to the camera 110 in the depth map and a second weight determined based on the difference in luminance value for each pixel in the depth map ( Since w _pq ^C ) is applied and calculated, the accuracy of the depth value can be improved through training that applies the loss value (Loss _g ) to the artificial intelligence model.

FIG. 11 is a diagram illustrating an operation of the augmented reality device 100 acquiring a depth map 1130 using an artificial intelligence model 1100 according to an embodiment of the present disclosure.

Referring to FIG. 11, the augmented reality device 100 may include a stereo camera consisting of a left-eye camera and a right-eye camera. The augmented reality device 100 inputs the left eye image 1110L acquired using the left eye camera, the right eye image 1110R acquired using the right eye camera, and the IMU sensor measurement value 1120 to the artificial intelligence model 1100. And, the depth map 1130 can be obtained by performing inference using the artificial intelligence model 1100.

The artificial intelligence model 1100 may be a trained model by applying the loss value 740 (see FIG. 7), as in the embodiment shown in FIG. 7. The augmented reality device 100 inputs the IMU sensor measurement value 1120 measured by the IMU sensor 120 (see FIG. 2) into the artificial intelligence model 1100, and the input IMU sensor measurement value 1120 is displayed in three dimensions. Can be applied to cost volume. In the inference process of the artificial intelligence model 1100, a depth map 1130 may be output through hierarchical refinement, such as up-convolution of the 3D cost volume.

In the embodiment shown in FIG. 11 , the augmented reality device 100 creates a depth map 1130 using an artificial intelligence model 1100 trained using IMU sensor measurements 1120 that include information about the direction of gravity. By obtaining, the depth value accuracy of the depth map 1130 can be improved. In addition, the augmented reality device 100 according to an embodiment of the present disclosure trains the artificial intelligence model 1100 using the IMU sensor measurement value 1120 obtained from the IMU sensor 120, which is an essential component, so additional No hardware modules are required, and this provides the technical effect of realizing low power consumption while maintaining a small form factor.

FIG. 12 is a flowchart illustrating a method by which the augmented reality device 100 corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.

Step S1210 shown in FIG. 12 is a step included in step S330 shown in FIG. 3. Steps S1220 to S1240 shown in FIG. 12 are steps that embody step S340 shown in FIG. 3.

FIG. 13 is a diagram illustrating an operation of the augmented reality device 100 according to an embodiment of the present disclosure to correct the depth value for each pixel of the depth map 1320 through a plane sweep method. Hereinafter, an operation in which the augmented reality device 100 corrects the depth value for each pixel of the depth map 1320 will be described with reference to FIGS. 12 and 13 together.

Referring to FIG. 12, in step S1210, the augmented reality device 100 corrects the directions of the normal vectors of the left eye image and the right eye image based on the direction of gravity measured by the IMU sensor 120 (see FIG. 2). Referring to FIG. 13 together, the augmented reality device 100 may include a stereo camera including a left-eye camera and a right-eye camera. The processor 130 (see FIG. 2) of the augmented reality device 100 acquires gravity direction information from the gyroscope of the IMU sensor 120 (see FIG. 2), and uses the left eye camera based on the obtained gravity direction information. The direction of the normal vector for each pixel of the left eye image 1310L acquired using the right eye camera and the right eye image 1310R acquired using the right eye camera can be corrected. In one embodiment of the present disclosure, the processor 130 may correct the direction of the normal vector for each pixel in the direction of gravity or in a direction perpendicular to the direction of gravity. Since the specific method for correcting the direction of the normal vector is the same as that described in FIGS. 6A and 6B, overlapping descriptions will be omitted.

Referring to FIG. 12, in step S1220, the augmented reality device 100 performs a plane hypothesis along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector. Referring to FIG. 13 , the processor 130 may define a plane for each pixel through a plane hypothesis for each of the left eye image 1310L and the right eye image 1310R. The processor 130 performs a plane assumption according to the direction of the corrected normal vector of each of the pixels included in the left eye image 1310L or the direction perpendicular to the direction of the corrected normal vector, so that each pixel in the left eye image 1310L A plane can be defined. In the same way, the processor 130 performs a plane assumption according to the direction of the corrected normal vector of each of the pixels included in the right eye image 1310R or the direction perpendicular to the direction of the corrected normal vector, thereby ), you can define a plane for each pixel.

Referring to FIG. 12, in step S1230, the augmented reality device 100 obtains a depth value for each pixel by performing a plane sweep along a plane defined by the plane assumption. Referring to FIG. 13 together, the processor 130 searches for a matching point in the right eye image 1310R based on the two-dimensional position coordinate values (x, y) of the reference pixel on the plane defined in the left eye image 1310L. You can perform a plane sweep. In one embodiment of the present disclosure, the processor 130 identifies a plane in the right eye image 1310R that corresponds to a plane defined in the left eye image 1310L through a plane assumption, and determines a plane within the plane identified in the right eye image 1310R. You can search for the matching point. In one embodiment of the present disclosure, the processor 130 selects a reference pixel of the left eye image 1310L within a disparity search range of d ₀ to d _max among pixels in a plane defined in the right eye image 1310R. You can search for the matching point corresponding to . The processor 130 measures the luminance value similarity between the reference pixel within the plane of the left eye image 1310L and the pixels within the corresponding plane of the right eye image 1310R, and calculates the luminance value dissimilarity and disparity between pixels. Based on the graph 1310 showing the relationship between the pixels, the pixel with the lowest luminance value dissimilarity in the right eye image 1310R can be identified as a matching point. The processor 130 may determine the distance ( _d The processor 130 may obtain depth values for each pixel by performing a plane sweep on all pixels of the left-eye image 1310L and the right-eye image 1310R in the manner described above.

Referring to FIG. 12, in step S1240, the augmented reality device 100 corrects the depth value of at least one pixel of the depth map using the acquired depth value. Referring to FIG. 13 together, the processor 130 of the augmented reality device 100 may correct the depth value of the depth map 1320 using the obtained depth value for each pixel.

In the embodiment shown in FIGS. 12 and 13, the augmented reality device 100 corrects the normal vector for each pixel in the direction of gravity and performs a plane sweep to a plane defined according to the corrected normal vector, thereby providing high accuracy. A depth map 1320 can be obtained.

FIG. 14 is a flowchart illustrating a method by which the augmented reality device 100 corrects the depth value for each pixel of a depth map according to an embodiment of the present disclosure.

Steps S1410 to S1430 shown in FIG. 14 are steps that embody step S340 shown in FIG. 3. Step S1410 shown in FIG. 14 may be performed after step S330 of FIG. 3 is performed.

FIG. 15 is a diagram illustrating an operation of the augmented reality device 100 according to an embodiment of the present disclosure to correct the depth value for each pixel of the depth map image 1510 acquired through a time-of-flight (ToF) method. am.

Referring to FIG. 14, in step S1410, the augmented reality device 100 defines a plane for each pixel in the depth map based on the corrected normal vector. In one embodiment of the present disclosure, the augmented reality device 100 may include a ToF camera. A ToF camera irradiates light to an object using a light source, detects reflected light reflected from the object, and determines the time of flight of the object based on the time of flight, which is the time difference between the time when the reflected light is detected and the time when the light is irradiated. It may be configured to obtain a depth value. Referring to FIG. 15 together, the augmented reality device 100 may acquire an RGB image 1500 and a depth map image 1510 of an object using a ToF camera. The processor 130 (see FIG. 2) of the augmented reality device 100 acquires a normal vector for each pixel from the depth map image 1510 and corrects the direction of the normal vector along the direction of gravity or a direction perpendicular to the direction of gravity. You can. The processor 130 may define a plane for each pixel in the depth map image 1510 based on the direction of the corrected normal vector. In one embodiment of the present disclosure, the processor 130 may define a plane composed of a pixel having a corrected normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel.

Referring to FIG. 14, in step S1420, the augmented reality device 100 identifies a flat area in the depth map based on the area divided according to the color information of the RGB image. Referring to FIG. 15 together, the processor 130 of the augmented reality device 100 divides the RGB image 1500 into a plurality of regions 1501 to 1501 based on the color information of each of the plurality of pixels included in the RGB image 1500. 1506). In one embodiment of the present disclosure, the processor 130 may divide the RGB image 1500 into a plurality of regions 1501 to 1506 based on pixel-specific luminance values of a plurality of pixels of the RGB image 1500. The processor 130 calculates the difference value of the luminance value of each of the plurality of pixels of the RGB image 1500, and groups the pixels whose calculated difference value is less than or equal to a preset threshold, thereby creating the RGB image 1500. It can be divided into a plurality of areas (1501 to 1506). The processor 130 may identify a flat area in the depth map image 1510 based on at least one of the location, shape, and size of the plurality of divided areas 1501 to 1506 of the RGB image 1500. there is. In the embodiment shown in FIG. 15, the processor 130 creates a plurality of planar regions 1511 to 1516 corresponding to the plurality of divided regions 1501 to 1506 of the RGB image 1500 from the depth map image 1510. can be identified.

Referring to FIG. 14, in step S1430, the augmented reality device 100 corrects the depth value of a pixel in the identified planar area based on the depth values of adjacent pixels. In one embodiment of the present disclosure, the augmented reality device 100 identifies a pixel to be corrected in a planar area within a depth map, and corrects the depth value of the identified pixel to be corrected based on the depth values of adjacent pixels in the same planar area. can do. In the present disclosure, a 'correction target pixel' is a pixel whose depth value needs to be corrected, and is a pixel whose depth value has not been obtained in a flat area or a pixel whose depth value difference with adjacent pixels in a flat area exceeds a preset threshold. means. In the embodiment shown in FIG. 15, among the plurality of planar areas 1511 to 1516 identified from the depth map image 1510, depth values have not been obtained in the first to fifth planar areas 1511 to 1515. Pixels whose difference in pixel or depth value exceeds a preset threshold may not be identified. At least one correction target pixel 1526 for which a depth value has not been obtained may be identified in the sixth plane area 1516 among the plurality of planar areas 1511 to 1516. The processor 130 may correct the depth value of the identified correction target pixel 1526 based on the depth values of adjacent pixels among a plurality of pixels included in the sixth plane 1516.

In the case of a general ToF camera, there is a problem that the accuracy of the depth value decreases as the distance from the camera increases, or the depth value is not obtained. In the embodiment shown in FIGS. 14 and 15, the augmented reality device 100 corrects the normal vector for each pixel in the depth map image 1510 based on the direction of gravity, and defines a plane based on the corrected normal vector. , Based on the color information for each pixel of the RGB image 1500, the flat area of the depth map image 1510 is identified, and the depth value of the correction target pixel 1526 in the flat area is corrected, improving the accuracy of the depth map. You can do it.

16 shows a three-dimensional space model 1610 restored with a depth map obtained in a general manner and a three-dimensional space model 1620 restored with a depth map acquired by the augmented reality device 100 according to an embodiment of the present disclosure. ) This is a drawing showing.

Referring to FIG. 16, referring to the three-dimensional space model 1610 of the real space 1600 acquired through a conventional depth map acquisition method, the surface of the object is inclined, and the three-dimensional space model 1610 Accuracy is low. The general depth map acquisition method has the problem that the accuracy of the depth value decreases as the distance from the camera increases.

The augmented reality device 100 according to an embodiment of the present disclosure corrects the direction of the normal vector for each pixel along the gravity direction measured by the IMU sensor 120 (see FIG. 2) or a direction perpendicular to the gravity direction, and corrects Since the depth value for each pixel is corrected based on the obtained normal vector, the accuracy and resolution of the obtained depth map can be improved compared to the depth map obtained in a general manner. In addition, the three-dimensional space model 1620 restored with the depth map acquired by the augmented reality device 100 according to an embodiment of the present disclosure can be seen that the direction of the object has been corrected to be perpendicular or horizontal to the direction of gravity. there is.

In addition, the augmented reality device 100 according to an embodiment of the present disclosure obtains gravity direction information using the IMU sensor 120, which is an essential component, and can implement low power consumption while maintaining a small form factor. As a result, the augmented reality device 100 of the present disclosure can provide technical effects that increase portability and device use time and improve user convenience.

The present disclosure provides an augmented reality device 100 that corrects depth values based on the direction of gravity. The augmented reality device 100 according to an embodiment of the present disclosure includes a camera 110 (see FIG. 2), an IMU sensor (Inertial Measurement Unit) 120 (see FIG. 2), and at least one processor 130 (see FIG. 2). , and a memory 140 (see FIG. 2). At least one processor 130 may obtain a depth map from an image acquired using the camera 110 by executing at least one command stored in the memory 140. At least one processor 130 may obtain a normal vector of at least one pixel included in the depth map. At least one processor 130 may correct the direction of the normal vector of the at least one pixel based on the direction of gravity measured by the IMU sensor 120. At least one processor 130 may correct the depth value of the at least one pixel based on the direction of the corrected normal vector.

In one embodiment of the present disclosure, at least one processor 130 converts at least one pixel into a three-dimensional coordinate value based on the direction vector of at least one pixel included in the depth map and the depth value of the at least one pixel. A normal vector can be obtained by converting and calculating the cross-product of the 3D coordinate values of a plurality of adjacent pixels in the up, down, left, and right directions.

In one embodiment of the present disclosure, the camera 110 may include a left eye camera that acquires a left eye image and a right eye camera that acquires a right eye image. At least one processor 130 applies the left-eye image and the right-eye image as input to an artificial intelligence model, and uses the artificial intelligence model to calculate disparity according to the similarity of pixel luminance values of the left-eye image and the right-eye image to create a depth map. can be obtained.

In one embodiment of the present disclosure, the at least one processor 130 uses an artificial intelligence model based on the depth values of a pixel on a plane defined by a corrected normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel. The loss of the acquired depth map can be calculated. At least one processor 130 may correct the depth value of at least one pixel by performing training to apply the calculated loss value to the artificial intelligence model.

In one embodiment of the present disclosure, at least one processor 130 defines a plane consisting of a pixel having a corrected normal vector and a plurality of adjacent pixels positionally adjacent to the pixel, and the defined plane and the camera 110 The depth values of a plurality of adjacent pixels can be obtained based on a plurality of points where ray vectors of ) meet. At least one processor 130 may calculate a difference value between the depth value of the acquired pixel and the depth values of a plurality of adjacent pixels. At least one processor 130 may obtain a loss value by performing a weighted sum operation that applies a weight to the difference value calculated for each plurality of adjacent pixels.

In one embodiment of the present disclosure, the weight is determined based on a first weight determined based on the distance to the camera of each of a plurality of adjacent pixels in the depth map and a luminance value difference between the pixel in the depth map and the plurality of adjacent pixels. It may include a second weight determined based on the weight.

In one embodiment of the present disclosure, at least one processor 130 may obtain a corrected depth map by performing inference by inputting a left-eye image and a right-eye image to a trained artificial intelligence model.

In one embodiment of the present disclosure, the camera 110 of the at least one processor 130 may include a left-eye camera that acquires a left-eye image and a right-eye camera that acquires a right-eye image. At least one processor 130 corrects the direction of the normal vector of at least one pixel of the left-eye image and the right-eye image according to the direction of gravity or a direction perpendicular to the direction of gravity, and determines the direction of the corrected normal vector or the direction of the corrected normal vector. The plane hypothesis can be made along the direction perpendicular to the direction. At least one processor 130 may obtain the depth value of at least one pixel by performing a plane sweep along a plane defined by the plane assumption. At least one processor 130 may correct the depth value of at least one pixel of the depth map using the obtained depth value.

In one embodiment of the present disclosure, the camera 110 is configured as a Time-of-Flight camera (ToF camera), and at least one processor 130 may acquire a depth map using the ToF camera.

In one embodiment of the present disclosure, at least one processor 130 may define a plane for each pixel based on the corrected normal vector. At least one processor 130 may identify a flat area of the plane defined in the depth map based on the area divided according to color information of the RGB image. At least one processor 130 may correct the depth value of at least one pixel of the depth map based on the depth value of adjacent pixels in the identified planar area.

The present disclosure provides a method for the augmented reality device 100 to correct depth values. In one embodiment of the present disclosure, the method may include obtaining a depth map from an image acquired using the camera 110 (S310). The method may include obtaining a normal vector of at least one pixel included in the depth map (S320). The method may include correcting the direction of the normal vector of at least one pixel based on the direction of gravity measured by the IMU sensor 120 (S330). The method may include correcting the depth value of at least one pixel based on the direction of the corrected normal vector (S340).

In one embodiment of the present disclosure, the step of acquiring the depth map (S310) includes inputting a left eye image acquired using a left eye camera and a right eye image acquired using a right eye camera into an artificial intelligence model, and artificial intelligence It may include obtaining a depth map by calculating disparity according to the similarity of pixel luminance values of the left-eye image and the right-eye image using the model.

In one embodiment of the present disclosure, the step of correcting the depth value of the at least one pixel (S340) includes the depth values of the pixel on the plane defined by the corrected normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel. Based on this, it may include calculating the loss of the depth map acquired by the artificial intelligence model (S810). The step of correcting the depth value of at least one pixel (S340) is a step of correcting the depth value of at least one pixel of the depth map by performing training by applying the calculated loss value to the artificial intelligence model. (S820) may be included.

In one embodiment of the present disclosure, calculating the loss value (S810) includes defining a plane consisting of a pixel with a corrected normal vector and a plurality of adjacent pixels positionally adjacent to the pixel (S910), and It may include obtaining depth values of a plurality of adjacent pixels based on a plurality of points where the defined plane and the ray vector of the camera 110 meet (S920). Calculating the loss value (S810) may include calculating a difference value between the depth value of the acquired pixel and the depth values of a plurality of adjacent pixels (S930). The step of calculating the loss value (S810) may include the step of obtaining the loss value by performing a weighted sum operation that applies a weight to the difference value calculated for each of the plurality of adjacent pixels (S940). there is.

In one embodiment of the present disclosure, the weight is a first weight determined based on the distance from the camera 110 of each of the plurality of adjacent pixels in the depth map and the luminance value between the pixel in the depth map and the plurality of adjacent pixels. It may include a second weight determined based on the difference.

In one embodiment of the present disclosure, the method may further include obtaining a corrected depth map by performing inference by inputting the left-eye image and the right-eye image to the trained artificial intelligence model.

In one embodiment of the present disclosure, the step of correcting the direction of the normal vector of the at least one pixel (S330) involves using the left eye image and the right eye camera acquired using the left eye camera according to the direction of gravity or the direction perpendicular to the direction of gravity. It may include a step of correcting the direction of the normal vector of the right eye image obtained using the method. The step of correcting the depth value of at least one pixel (S340) includes performing a plane hypothesis along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector (S1220). can do. The step of correcting the depth value of at least one pixel (S340) includes the step of obtaining the depth value of at least one pixel by performing a plane sweep along a plane defined by the plane assumption (S1230). It can be included. The step of correcting the depth value of at least one pixel (S340) may include a step of correcting the depth value of at least one pixel using the obtained depth value (S1240).

In one embodiment of the present disclosure, the step of acquiring the depth map (S310) may acquire the depth map using a Time-of-Flight camera (ToF camera).

In one embodiment of the present disclosure, correcting the depth value of at least one pixel (S340) may include defining a plane of the pixel based on the corrected normal vector (S1410). The step of correcting the depth value of the at least one pixel (S340) may include the step of identifying a flat area of the plane defined in the depth map based on the area divided according to the color information of the RGB image (S1420). . The step of correcting the depth value of the at least one pixel (S340) may include the step of correcting the depth value of the at least one pixel based on the depth values of adjacent pixels in the identified planar area (S1430).

The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may store instructions related to obtaining a depth map from an image acquired using the camera 110. The storage medium may store instructions related to obtaining a normal vector of at least one pixel included in the depth map. The storage medium may store instructions related to correcting the direction of a normal vector of at least one pixel based on the direction of gravity measured by the IMU sensor. The storage medium may store instructions related to correcting the depth value of at least one pixel of the depth map based on the direction of the corrected normal vector.

A program executed by the augmented reality device 100 described in this disclosure may be implemented with hardware components, software components, and/or a combination of hardware components and software components. A program can be executed by any system that can execute computer-readable instructions.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device.

Software may be implemented as a computer program including instructions stored on computer-readable storage media. Computer-readable recording media include, for example, magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disk, hard disk, etc.) and optical read media (e.g., CD-ROM). (CD-ROM), DVD (Digital Versatile Disc), etc. The computer-readable recording medium is distributed among computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner. The media may be readable by a computer, stored in memory, and executed by a processor.

Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' only means that the storage medium does not contain signals and is tangible, and does not distinguish between cases where data is stored semi-permanently or temporarily in the storage medium. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

Additionally, programs according to embodiments disclosed in this specification may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers.

A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may be a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of the augmented reality device 100 or through an electronic market (e.g., Samsung Galaxy Store). ))) may be included. For electronic distribution, at least a portion of the software program may be stored on a storage medium or created temporarily. In this case, the storage medium may be a storage medium of a server of the manufacturer of the augmented reality device 100, a server of an electronic market, or a relay server that temporarily stores a software program.

The computer program product, in a system comprised of the augmented reality device 100 and/or a server, may include a storage medium of the server or a storage medium of the augmented reality device 100. Alternatively, if there is a third device (eg, a mobile device) in communication connection with the augmented reality device 100, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the augmented reality device 100 to a third device or from a third device to an electronic device.

In this case, either the augmented reality device 100 or a third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the augmented reality device 100 and the third device may execute the computer program product and perform the methods according to the disclosed embodiments in a distributed manner.

For example, the augmented reality device 100 executes a computer program product stored in the memory 140 (see FIG. 2), and another electronic device (e.g., a mobile device) connected to communicate with the augmented reality device 100 is disclosed. It can be controlled to perform the method according to the embodiments.

As another example, a third device may execute a computer program product to control an electronic device communicatively connected to the third device to perform the method according to the disclosed embodiment.

When the third device executes the computer program product, the third device may download the computer program product from the augmented reality device 100 and execute the downloaded computer program product. Alternatively, the third device may perform the methods according to the disclosed embodiments by executing a computer program product provided in a pre-loaded state.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and/or components, such as a described computer system or module, may be combined or combined in a form different from the described method, or other components or equivalents may be used. Appropriate results can be achieved even if replaced or replaced by .

Claims (20)

camera 110;
An Inertial Measurement Unit (IMU) sensor configured to measure the direction of gravity (120);
a memory 140 that stores at least one instruction; and
At least one processor 130 executing the at least one instruction;
Including,
The at least one processor 130,
Obtaining a depth map from the image acquired using the camera 110,
Obtaining a normal vector of at least one pixel included in the depth map,
Correcting the direction of a normal vector of the at least one pixel based on the direction of gravity measured by the IMU sensor 120,
The augmented reality device 100 corrects the depth value of the at least one pixel based on the direction of the corrected normal vector.
According to claim 1,
The at least one processor 130,
Converting the at least one pixel into a three-dimensional coordinate value based on the direction vector of the at least one pixel included in the depth map and the depth value of the at least one pixel,
An augmented reality device 100 that obtains the normal vector by calculating a cross-product of three-dimensional coordinate values of a plurality of adjacent pixels adjacent in the up, down, left, and right directions.
According to any one of claims 1 and 2,
The camera 110 includes a left eye camera for acquiring a left eye image and a right eye camera for acquiring a right eye image,
The at least one processor 130,
Applying the left eye image and the right eye image as input to the artificial intelligence model,
An augmented reality device 100 that obtains the depth map by calculating disparity according to the similarity of pixel luminance values of the left-eye image and the right-eye image using the artificial intelligence model.
According to clause 3,
The at least one processor 130,
Based on the depth values of a pixel on a plane defined by the corrected normal vector and a plurality of adjacent pixels in position adjacent to the pixel, calculate a loss value of the depth map obtained by the artificial intelligence model, ,
An augmented reality device 100 that corrects the depth value of the at least one pixel by performing training to apply the calculated loss value to the artificial intelligence model.
According to clause 4,
The at least one processor 130,
Defining a plane consisting of a pixel having the corrected normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel,
Obtaining depth values of the plurality of adjacent pixels based on a plurality of points where the defined plane meets the ray vector of the camera 110,
Calculating a difference value between the depth value of the acquired pixel and the depth values of the plurality of adjacent pixels, respectively,
An augmented reality device 100 that obtains the loss value by performing a weighted sum operation that applies a weight to the difference value calculated for each of the plurality of adjacent pixels.
According to clause 5,
The weight is a first weight determined based on the distance from the camera 110 of each of the plurality of adjacent pixels in the depth map and the difference in luminance value between the pixel and the plurality of adjacent pixels in the depth map. Augmented reality device 100, including a second weight determined based on .
According to any one of claims 3 to 6,
The at least one processor 130,
An augmented reality device 100 that obtains a corrected depth map by performing inference by inputting the left eye image and the right eye image into the trained artificial intelligence model.
According to claim 1,
The camera 110 includes a left eye camera for acquiring a left eye image and a right eye camera for acquiring a right eye image,
The at least one processor 130,
Correcting the directions of normal vectors of the left-eye image and the right-eye image according to the direction of gravity or a direction perpendicular to the direction of gravity,
Performing a plane hypothesis along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector,
Obtaining a depth value of the at least one pixel by performing a plane sweep along a plane defined by the plane assumption,
The augmented reality device 100 corrects the depth value of the at least one pixel using the obtained depth value.
According to claim 1,
The camera 110 is configured as a Time-of-Flight camera (ToF camera),
The at least one processor 130,
An augmented reality device 100 that acquires the depth map using the ToF camera.
According to clause 9,
The at least one processor 130,
Define a plane for each pixel based on the corrected normal vector,
Identifying a flat area of the plane defined in the depth map based on the area divided according to the color information of the RGB image,
Augmented reality device 100, correcting the depth value based on the depth value of adjacent pixels within the identified planar area.
In the method for the augmented reality device 100 to correct the depth value,
Obtaining a depth map from an image acquired using the camera 110 (S310);
Obtaining a normal vector of at least one pixel included in the depth map (S320);
Correcting the direction of the normal vector of the at least one pixel based on the direction of gravity measured by the IMU sensor 120 (S330); and
Correcting the depth value of the at least one pixel based on the direction of the corrected normal vector (S340);
Method, including.
According to claim 11,
The step of acquiring the depth map (S310) is,
Inputting a left eye image acquired using a left eye camera and a right eye image acquired using a right eye camera into an artificial intelligence model; and
Obtaining the depth map by calculating disparity according to the similarity of pixel luminance values of the left-eye image and the right-eye image using the artificial intelligence model;
Method, including.
According to claim 12,
The step (S340) of correcting the depth value for each pixel of the depth map is,
Calculating the loss of the depth map obtained by the artificial intelligence model based on the depth values of the pixel on the plane defined by the corrected normal vector and a plurality of adjacent pixels in position adjacent to the pixel. Step (S810); and
Compensating the depth value of the at least one pixel by performing training by applying the calculated loss value to the artificial intelligence model (S820);
Method, including.
According to claim 13,
The step of calculating the loss value (S810) is,
Defining a plane consisting of a pixel having the corrected normal vector and a plurality of adjacent pixels that are positionally adjacent to the pixel (S910);
Obtaining depth values of the plurality of adjacent pixels based on a plurality of points where the defined plane meets the ray vector of the camera 110 (S920);
Calculating difference values between the depth value of the acquired pixel and the depth values of the plurality of adjacent pixels (S930); and
Obtaining the loss value by performing a weighted sum operation that applies a weight to the difference value calculated for each of the plurality of adjacent pixels (S940);
Method, including.
According to claim 14,
The weight is a first weight determined based on the distance from the camera 110 of each of the plurality of adjacent pixels in the depth map and the difference in luminance value between the pixel and the plurality of adjacent pixels in the depth map. A method comprising a second weight determined based on .
According to any one of claims 12 to 15,
Obtaining a corrected depth map by performing inference by inputting the left eye image and the right eye image into the trained artificial intelligence model;
A method further comprising:
According to claim 11,
The step (S330) of correcting the direction of the normal vector for each pixel is,
Correcting the directions of normal vectors of a left-eye image acquired using a left-eye camera and a right-eye image acquired using a right-eye camera according to the direction of gravity or a direction perpendicular to the direction of gravity;
Including,
The step (S340) of correcting the depth value of the at least one pixel is,
Performing a plane hypothesis along the direction of the corrected normal vector or a direction perpendicular to the direction of the corrected normal vector (S1220); and
Obtaining the depth value by performing a plane sweep along a plane defined by the plane assumption (S1230); and
Correcting the depth value of the at least one pixel using the obtained depth value (S1240);
Method, including.
According to claim 11,
The step of acquiring the depth map (S310) is,
A method of acquiring the depth map using a Time-of-Flight camera (ToF camera).
According to clause 18,
The step (S340) of correcting the depth value of the at least one pixel is,
defining a pixel plane based on the corrected normal vector (S1410);
Identifying a flat area of the plane defined in the depth map based on the area divided according to the color information of the RGB image (S1420); and
Compensating the depth value of at least one pixel based on the depth value of adjacent pixels in the identified planar area (S1430);
Method, including.
A computer-readable recording medium on which at least one program for implementing the method according to any one of claims 11 to 19 is recorded.

Images