WO2015199502A1 - Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée - Google Patents

Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée Download PDF

Info

Publication number
WO2015199502A1
WO2015199502A1 PCT/KR2015/006591 KR2015006591W WO2015199502A1 WO 2015199502 A1 WO2015199502 A1 WO 2015199502A1 KR 2015006591 W KR2015006591 W KR 2015006591W WO 2015199502 A1 WO2015199502 A1 WO 2015199502A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinate system
camera
sub
depth
augmented reality
Prior art date
Application number
PCT/KR2015/006591
Other languages
English (en)
Korean (ko)
Inventor
우운택
하태진
Original Assignee
한국과학기술원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 한국과학기술원 filed Critical 한국과학기술원
Priority to US15/322,075 priority Critical patent/US10304248B2/en
Priority claimed from KR1020150091330A external-priority patent/KR101865655B1/ko
Publication of WO2015199502A1 publication Critical patent/WO2015199502A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics

Definitions

  • the present invention supports hand interaction with augmented virtual objects in an HMD-based wearable environment equipped with an RGB-D camera based on geometric-based registration coordinate correction using an RGB-D camera for wearable augmented reality authoring. It is about technology to do.
  • a user uses a camera to acquire image feature-camera pose information about the real space, and obtain local reference coordinates (or matched coordinates). After generating, you need to match the coordinate system of the virtual space based on this. However, since the matching coordinate system is generated at an arbitrary position, a process of manually correcting the attitude of the coordinate system is necessary.
  • a three-dimensional virtual object modeled in units of the real space may be augmented accurately in the augmented reality space through a correction process that accurately matches the scale between the real space and the augmented reality space.
  • the GPS / compass sensor-based matching method has a problem that the accuracy of matching is very low due to an error of sensor information, and the 2D object-based matching method requires a pre-learned image.
  • the object is not suitable for any three-dimensional space registration because the object is limited to a simple two-dimensional plane.
  • 3D space-based registration generates a registration coordinate system for augmentation at an arbitrary position, it is necessary to manually correct the coordinate system attitude by the user, and in order to perform such correction, the user has expertise in computer vision / graphics, etc. If this is necessary and the user inputs incorrectly, an error of matching may occur due to incorrect input.
  • Korean Patent Publication No. 10-0980202 relates to a mobile augmented reality system and method that can interact with a three-dimensional virtual object, the camera attached to the terminal, the camera of the terminal
  • the image processing unit for generating a three-dimensional virtual object on the hand the display unit for outputting the image of the three-dimensional virtual object and the hand and the interaction unit for controlling the three-dimensional virtual object in response to the movement of the hand
  • Users can access 3D virtual content anytime, anywhere using a mobile device.
  • the technology is a technique for accessing 3D virtual content using a mobile device, and does not include automatically generating and correcting a registration coordinate system for matching virtual space.
  • the real space we live in is 3d space.
  • Using the interface for the existing 2d display in this real space has a limitation because the order of space is reduced by one.
  • 3d interface technology is required to deal with virtual digital content combined in 3d space.
  • the HMD with a camera provides the user with a first-person view, unlike displays in traditional desktop environments.
  • the hand is an object without a texture.
  • a feature-based object detection / tracking algorithm from color information cannot be applied to finger posture estimation.
  • the task of detecting / tracking a hand and estimating the posture of a finger based on a camera has a challenging condition.
  • the WearTrack system is a wearable system using a magnetic tracker and an HMD equipped with a posture estimation sensor.
  • Systems such as virtual touch screen systems, AR memo, and SixthSense are characterized by 2d interaction based on a 2d image coordinate system. However, this has the disadvantage of not interacting in 3d space because it is 2d based interaction.
  • Tinmith and FingARtips attach additional markers on the glove to estimate hand posture.
  • the size of the separate sensor is very large, it is not suitable for the wearable environment from the user's point of view.
  • a feature point based approach has also been developed. This is a method of estimating finger motion by recognizing a pattern through prior learning.
  • the system locks an RGB-D camera, such as Kinect, to face, and estimates the movement of a user's hand wearing a glove with a specific pattern.
  • RGB-D camera such as Kinect
  • the Digits system demonstrates fingertip tracking for wearable devices.
  • Time of Flight (TOF) depth sensor was worn on the wrist, and the setting was performed to prevent the finger from covering up. It uses simple carving technique to classify fingers and estimate finger posture using the relationship between finger joints.
  • TOF Time of Flight
  • this method has a disadvantage in that the sensor must be attached to an additional part such as the wrist in addition to the HMD.
  • the present invention estimates the finger posture of the bare hand, and aims to estimate the posture of the finger when the finger is bent toward the camera.
  • an object of the present invention is to provide a geometric recognition-based matching coordinate system correction method and apparatus for wearable augmented reality authoring that can automatically generate / correct the matching coordinate system for matching the virtual space based on the actual measurement.
  • a process of generating reference coordinates based on a three-dimensional image including depth information obtained through a camera, and a three-dimensional image including depth information obtained through the camera Dividing a region corresponding to the predetermined object based on depth information and color space transformation of a predetermined object, separating a sub-object having a motion component from the divided region object, and separating Modeling the sub-object and a palm area associated with the sub-object based on a predetermined algorithm to detect a feature point, and based on joint information of the object provided through a predetermined user interface.
  • Process of estimating posture and controlling 3D objects for using augmented reality service Characterized in that it comprises a.
  • a registration coordinate system correction unit for generating reference coordinates (Reference Coordinates) based on a three-dimensional image including depth information obtained through the camera, and a depth information obtained through the camera
  • An object separation unit for dividing a region corresponding to the predetermined object based on depth information and color space transformation of a predetermined object from a 3D image, and a sub-object having a motion component from the object of the divided region
  • An object processor which detects a feature point by modeling the separated sub-object and the palm region associated with the sub-object based on a predetermined algorithm, and skeleton information of the object provided through a predetermined user interface.
  • Augmented reality by estimating a posture of the sub-object based on And a controller for controlling the 3D object for using the service.
  • the matching coordinate system for matching the virtual space is automatically generated / corrected based on the actual measurement, so that the matching coordinate system can be automatically generated and corrected without a correction operation by the user.
  • the present invention can be used as an underlying technology required for authoring augmented reality content in various fields such as augmented reality-based art galleries / museums, classrooms, industries, interior design, etc., because the matching coordinate system can be automatically corrected.
  • FIG. 1 is a flowchart illustrating a method for providing augmented reality interaction service according to an embodiment of the present invention.
  • FIG. 2 is a diagram illustrating a screen to which a user's visual distance perception improvement method is applied when interacting with a bare hand in a head wearable display-based augmented reality environment according to an embodiment of the present invention.
  • FIG. 3 is a flowchart illustrating an operation of correcting a coordinate coordinate system in the augmented reality interaction service providing method according to an exemplary embodiment.
  • FIG. 4 is a detailed block diagram of an operation algorithm for estimating a hand posture in a method for providing augmented reality interaction service according to an exemplary embodiment of the present invention.
  • FIG. 5 is a view illustrating a screen related to visual feedback for improving depth perception in the augmented reality interaction service providing method according to an exemplary embodiment.
  • FIG. 6 is a view illustrating a screen related to a semi-transparent gray shadow and guideline in the augmented reality interaction service providing method according to an embodiment of the present invention.
  • FIG. 7 is a view illustrating a finger joint related position vector in the augmented reality interaction service providing method according to an exemplary embodiment of the present invention.
  • FIG. 8 is a diagram illustrating a screen for an overall operation to which a method for improving visual perception of a user is applied in the augmented reality interaction service providing method according to an exemplary embodiment.
  • FIG. 9 is a diagram illustrating a registration coordinate correction correction method in the augmented reality interaction service providing method according to an embodiment of the present invention.
  • 10 is an example of candidates of a matching coordinate system in 3D space in the method of providing augmented reality interaction service according to an embodiment of the present invention.
  • FIG. 11 is an example of setting a rotation axis of a registration coordinate system in the augmented reality interaction service providing method according to an embodiment of the present invention.
  • FIG. 12 is an example of a scale correction using a distance ratio between a SLAM-based registration coordinate system and a depth camera-based registration coordinate system in a method for providing augmented reality interaction service according to an embodiment of the present invention.
  • FIG. 13 is an example of a position correction in the augmented reality interaction service providing method according to an embodiment of the present invention.
  • FIG. 14 is a view illustrating a rotation correction in a method for providing augmented reality interaction service according to an embodiment of the present invention
  • 15 is a block diagram of an apparatus for providing augmented reality interaction service according to an exemplary embodiment.
  • 16 is a block diagram of a registration coordinate system correcting unit in the apparatus for providing augmented reality interaction services according to an embodiment of the present invention.
  • the present invention relates to providing an augmented reality interaction service, and more particularly, in authoring wearable augmented reality, based on a measurement coordinate system for matching a virtual space using information obtained by an RGB-D camera.
  • Depth information and color space of a predetermined object from a three-dimensional image including depth information for automatically generating / correcting and estimating the pose of the object for interaction with the virtual object in augmented reality.
  • a finger having a motion component and a palm region associated with the finger are modeled through a predetermined algorithm to detect a feature point, and based on the skeleton information of the object provided through a predetermined user interface.
  • 3D customer for using augmented reality service by estimating the posture of the sub object By controlling the user as well as to make available a variety of 3D content, to provide a technique that enables to provide an interface which effectively controls the object in three-dimensional space for the developer.
  • the present invention can automatically generate and correct the matching coordinate system for matching the virtual space on the basis of the actual measurement, it is possible to automatically generate and correct the matching coordinate system without the correction work by the user, furthermore augmented reality-based galleries / museums, classrooms To provide a foundation technology for authoring augmented reality content in various fields such as, industrial, interior design, etc.
  • FIG. 1 is a flowchart illustrating an augmented reality interaction service providing method according to an exemplary embodiment.
  • step 110 reference coordinates are generated based on a 3D image including depth information obtained through a camera.
  • step 110 is performed by analyzing the geometry of the real space using the depth image information photographed for the real space and generating a matching coordinate system for the real space.
  • a matching coordinate system for the real space E.g., Head Mounted Display
  • a coordinate-based coordinate coordinate system based on geometric recognition for augmented reality authoring which is a method for more robustly performing an object pose estimation for interaction with a virtual object in augmented reality described below. Interface supported by.
  • step 112 depth information and color space transformation of a predetermined object are performed from a three-dimensional image including depth information acquired through the camera, and based on this, the object corresponds to the preset object in step 114. Split the area.
  • the predetermined object refers to a hand object, and according to an embodiment of the present invention, the hand object is divided through an operation of steps 112 to 114 from an RGB image and a depth image.
  • the RGB color space is converted from the RGB image to the HSV color space, and the skin color space is saturated and saturated for robust skin region division. Obtained by performing a double threshold on the (value) element.
  • the distance from the depth image by the distance (arm distance) from the camera where the hand is attached to the HMD is set as a threshold.
  • the threshold is set to 60 cm, and the segmented depth image and the color image are aligned using a known calibration.
  • step 116 the sub-object having the motion component is separated from the object in the divided region, and in step 118, the sub-object and the palm region associated with the sub-object are modeled based on a predetermined algorithm to generate a feature point.
  • the feature point includes a finger reference point and an end point of the depth information-based hand, and the end point of the hand is extracted using template matching from a pre-modeled depth template.
  • This operation is performed since the palm and the finger must be separated from each other to estimate the posture of the finger corresponding to the sub-object from the image of the divided hand.
  • the hand image utilizes a mophological operation. Your fingers and palms are separated.
  • the morphological operation is a finger and palm are separated by using erosion and dilation
  • the erosion is an operation of eroding the image from the outside
  • the dilation is inflated in contrast to the erosion
  • the erosion is repeatedly performed, the area of the finger gradually disappears.
  • the palm area can be modeled by expanding the area of the palm by performing a dilation.
  • the center point of the palm is computed through a distance transform, and the calculated center point is the basis for the finger's reference point search.
  • the finger is modeled through the operation of step 116 with the palm, the finger is modeled by the elliptic fitting.
  • the smallest distance between the points of the modeled ellipse (finger) and the center point of the palm is estimated as the reference point of the finger. This makes it possible to find the reference point of the finger even when the finger is bent to some extent.
  • the end point of the hand is detected by using depth information, not only by detecting a fitted ellipse on 2d.
  • the present invention utilizes a known Zero-mean Normalized Cross Correlation (ZNCC) in an image processing application to extract an end point of a hand.
  • ZNCC Zero-mean Normalized Cross Correlation
  • an end point of a hand may be extracted using template matching from a depth-template previously modeled.
  • the red portion of the correlation map of FIG. 6 is the portion that most closely matches the depth template. This approach shows that the fingertip position can be detected even when the finger is bent.
  • the position of the detected fingertip and finger reference point is input to the inverse kinematics algorithm in a later module.
  • step 120 the posture of the sub-object is estimated based on joint information of the object provided through a predetermined user interface to control the 3D object for using augmented reality service.
  • Inverse kinematics is a parameter of joints when a reference coordinate system and an end point position are given. parameter) to estimate the base point obtained from the camera as the origin of the reference coordinate system and set the position of the fingertip to the end point.
  • the rotation matrix of joints is estimated by applying inverse kinematics. Since there are a total of four parameters for moving the finger, there are a total of four parameters to be estimated for each finger.
  • the inverse kinematics algorithm is an inverse-kinematics algorithm based on the damped least-square-method.
  • This algorithm estimates the amount that each joint should change using the difference between the target point (the position of the fingertip obtained from the camera) and the current point (the position of the fingertip of the current model).
  • ⁇ is a parameter of the rotation matrix of the finger joint
  • ⁇ is a damping ratio parameter
  • L1, L2, and L3 are the length of each node of the finger.
  • the 3D object is manipulated by the operation 120.
  • the virtual object manipulation according to the present invention is performed according to the posture of a finger which can be widely used by a user.
  • the posture of the finger being targeted here is a posture mapped from the number of fingers.
  • the tong-shaped hand posture determines the position of the globe. Then, as the operation of pinching and spreading five fingers, the size of the globe was manipulated. From this interaction, a user wearing an HMD with an RGB-D camera can obtain virtual digital information by adjusting the position and size of the virtual globe, which is an augmented virtual object.
  • FIG. 4 The operation algorithm for estimating the posture of the hand for the method of providing augmented reality interaction service according to an embodiment of the present invention described above is shown in FIG. Referring to FIG. 4, the block-specific operation of FIG. 4 is as follows.
  • the hand object is split from the RGB image and the depth image (401, 402).
  • the rgb color space is converted to the HSV color space for robust skin region division.
  • This skin color space is obtained by performing a double threshold on the S and V elements.
  • the hand sets the distance to the threshold (distance) from the camera attached to the HMD, and detects the outline.
  • the palm and the fingers must be separated to estimate the pose of the finger.
  • the hand image performs morphological operations (erosion, dilation) and further associates the subtraction with the dilation, resulting in the separation of the fingers and palms (palm imgae, finger image).
  • the palm image performs distance transform and center and radius extraction for palm center position operation.
  • Palm center position radius, finger position, direction and length
  • the method for providing augmented reality interaction service automatically corrects the attitude of the coordinate system through geometry recognition-based registration coordinate system correction in authoring augmented reality content, and through the flowchart of FIG. 3. Let's take a closer look.
  • FIG. 3 is a flowchart illustrating a method for correcting a geometry-based matched coordinate system in a method for providing augmented reality interaction service according to an exemplary embodiment of the present invention.
  • the method according to the present invention receives depth image information from a depth camera, for example, an RGB-D camera, and receives a region of interest set by a user input (S310 and S320).
  • a depth camera for example, an RGB-D camera
  • the depth image information is information captured and generated by the depth camera, and may include a captured image feature, a posture information of the camera, a distance map image based on depth information, a color, and the like. Can be received after being set by the user input used.
  • the geometry of the ROI is analyzed using the received depth image information, and the first matched coordinate system based on the geometry is generated using the analyzed geometry (S330 and S340).
  • step S330 may perform a geometric analysis for predicting a plane, a tangent, a tangent, an intersection point, etc. for the ROI received from the depth camera, and step S340 may perform a geometry analysis for the analyzed real space or the ROI. Through this, the coordinate system of the real space can be generated.
  • At least one of the plane, the tangent, the tangent, and the intersection point of the region of interest is predicted by analyzing the geometry of the region of interest, and the predicted plane, the tangent, the tangent, the intersection
  • the first registration coordinate system may be generated through at least one of the above, 2) the origin and the direction are calculated through the geometric analysis of the ROI, and the front, Define one of the side and the floor, and generate a first registration coordinate system by correcting the calculated direction sign to match the predetermined left hand coordinate system of the virtual space. It may be.
  • the second registration coordinate system based on the SLAM is corrected using the generated first registration coordinate system based on the geometry, and then the most 3D object is created based on the corrected second registration coordinate system ( S350, S360).
  • the second matching coordinate system may be corrected based on the actual measurement using the distance ratio calculation of the depth camera generating the distance image information and the distance of the SLAM-based camera.
  • the origin position is used to create matching coordinates under various shape conditions having one side, two sides, and three sides. Calculate the direction and.
  • a user uses a mobile input device to determine the center location of an ROI from an RGB-D camera.
  • a radial circular cursor of 50 pixels that controls the area of the depth map image based on the determined center position of the ROI is controlled.
  • the 3D point group is reconstructed in the depth map and a local reference coordinate system, i.e., a first registration coordinate system, is generated.
  • the planes are predicted from a 3D point cloud of the region of interest.
  • the plane estimation may be defined as an optimization problem for predicting the variables a, b, c, and d of the normal vectors constituting the plane equation, as shown in Equation 3 below, random sample consensus (RANSAC) ) Can be estimated through the algorithm.
  • RANSAC random sample consensus
  • the method of determining the three degrees of freedom position in the local reference coordinate is to calculate the 3D coordinates on the intersection line close to the point v o selected by the user in the user's selection area.
  • a point (v *) that minimizes the distance between v o and v i , a point in the 3D point group, is set as the reference position of the coordinate system.
  • This equation is derived by the expansion of the Lagrange Multipliers, and the matrix value is calculated through QR decomposition.
  • the rotation of the coordinate system is used to determine two normal vectors from the predicted planes, eg, the vertical and ground planes, to determine the direction of the coordinate system, eg, the vertical and bottom planes.
  • the direction vector of the crossing line may be set by the cross product of the normal vector, and may be represented by Equation 6 below.
  • Equation 7 Equation 7
  • the least square solution based on the SVD decomposition which is an optimization technique, can be used to calculate the intersection point from the pseudo matrix, and the rotation can be set through the normal vectors of three planes.
  • the directions of the x, y, and z axes are not known exactly because the order and sign of the predicted normal vector may be changed.
  • the order of the normal vectors follows the number of point groups. This is important for graphical rendering in a left hand or right hand based rendering system.
  • the rotation of the coordinate system is aligned in consideration of the rotation information of the RGB-D camera.
  • the normal vector having the minimum angle difference with respect to the direction vector of each camera is found.
  • the normal vector determines the direction vector of the camera. For example, if the i th normal vector N i has a minimum angle difference from the forward camera vector C Front , N i may be set to the z axis.
  • other normal vectors can be defined by the x and y axes, and can correct the direction sign of the coordinates. That is, the direction vector of the camera may be determined by Equation 8 below.
  • C Side and C Ground mean a lateral camera vector and a bottom camera vector.
  • the rotation axis of the registration coordinate system may be set.
  • the scale in order to align the SLAM-based initial local reference coordinates to the depth camera coordinate system-based local reference coordinate system, the scale must be taken into account, and the size of the virtual model may be arbitrarily determined in the SLAM initialization.
  • the distance from the origin coordinate of the SLAM-based coordinate system to the RGB camera is calculated. This is the position vector size of the RGB camera pose matrix and may be expressed in virtual scale units.
  • the scale ratio ⁇ is calculated, and through this process, the scale unit in reality can be applied to augment the virtual object in the SLAM-based virtual reality space as shown in Equation 8. . Therefore, the present invention does not require manual scale correction, and the scale correction is automatically performed.
  • the scale of the SLAM coordinate system is corrected in consideration of the ratio between the scale of the SLAM coordinate system and the scale in reality.
  • Equation 11 the offset shift matrix T P may be utilized to move the RT CtoW to the RT Refine_trans , as shown in FIG. 13. It may be represented by Equation 12 below.
  • RT CtoW refers to a matrix for converting a camera coordinate system into a virtual space coordinate system in a SLAM-based virtual space
  • RT Refine_trans means a corrected local reference coordinate system
  • the virtual object may be augmented based on the coordinate system aligned on the real space scale.
  • Equation 13 the rotation of the current local coordinate system (R Curr ) relative to the rotation (R Init ) of the initial local coordinate system Compute the difference matrix (R Diff ).
  • the calculated difference matrix R Diff may be reflected to correct the RT Refine_trans , which may be reflected as in Equation 13 below.
  • the method of correcting Refine_trans RT taken from the depth estimation coordinate system in order to correct the rotation by reflecting, geometry multiplies the R -1 Curr to RT Refine_trans to offset the current camera rotation R Multiply Depth
  • rotation correction may be performed by multiplying a difference matrix R Diff to reflect camera rotation tracking information relative to initial camera rotation.
  • the present invention uses an RGB-D camera for real-time modeling of an arbitrary space that has not been previously modeled and analyzes a geometric structure, and automatically generates a matching coordinate system based on the actual measurement for wearable augmented reality authoring. This allows the user to easily and precisely augmented reality authoring without additional work on the registration coordinate correction.
  • FIGS. 15 to 16 An apparatus for providing augmented reality interaction service according to an exemplary embodiment of the present invention will be described with reference to FIGS. 15 to 16.
  • 15 is a block diagram of an apparatus for augmented reality interaction service according to an exemplary embodiment.
  • a registration coordinate system corrector 152 an object separator 154, a controller 156, and an object processor 158 are included.
  • the registration coordinate system corrector 152 generates reference coordinates based on a 3D image including depth information obtained through a camera.
  • the object separator 154 may be configured based on the depth information and the color space transformation of a predetermined object from a three-dimensional image including depth information obtained through a camera under the control of the controller 156. Splits the area corresponding to the object.
  • the object separating unit 154 converts the RGB color space of the hand image area corresponding to the predetermined object from the RGB image to the HSV color space for the area corresponding to the predetermined object, and converts the converted HSV color. Segmentation is performed based on the skin color space obtained by performing a double threshold on saturation and value in space.
  • the object separator 154 sets a distance corresponding to the distance between the hand and the camera from a depth image as a threshold value, and corresponds to a result of depth segmentation and RGB segmentation obtained from each image. Based on the intersection, segmentation of the hands is performed.
  • the object processor 158 separates a sub object having a motion component from an object of a region divided by the object separator 154 under the control of the controller 156, and is connected to the separated sub object and the sub object.
  • the palm region is modeled based on a predetermined algorithm to detect feature points.
  • the object processing unit 158 corresponds to a palm area associated with the sub object by using a morphological operation to estimate a posture of a finger corresponding to the sub object from the hand image corresponding to the object.
  • the palm region modeling is performed by separating a palm and a finger.
  • the controller 156 controls the overall operation of the apparatus for providing augmented reality interaction service 150 and estimates a posture of the sub-object based on skeleton information of an object provided through a predetermined user interface to use the augmented reality service. Control 3D objects for
  • the matched coordinate system corrector includes a receiver 160, a generator 162, an enhancer 164, an analyzer 166, and a corrector 168.
  • the receiver 160 receives depth image information from a depth camera or receives or is set or input by a user input.
  • the receiver 160 is a depth from a depth camera, for example, an RGB-D (depth) camera, attached to a glasses display device, such as a head worm display (HWD) worn on a user's head.
  • Depth images may be received, and a region of interest (ROI) in a real space set through a user input may be received.
  • the ROI may be set by user input using a mobile input device.
  • Depth image information is information captured and generated by the depth camera, and may include a photographed image feature, a posture information of the camera, a distance map image based on depth information, and color.
  • the analyzer 166 analyzes the geometry of the real space or the ROI by using the depth image information received by the receiver 160.
  • the analyzer 166 may perform a geometrical analysis for predicting a plane, a tangent, a tangent, an intersection point, and the like, of the ROI received from the depth camera.
  • the generator 162 generates a matched coordinate system for the real space through the geometric structure analysis of the real space or the ROI analyzed by the analyzer 166.
  • the generation unit 162 predicts at least one of the plane, the tangent, the tangent, and the intersection of the ROI through the geometrical analysis of the ROI, and generates the first through the at least one of the predicted plane, the tangent, the tangent, the intersection. You can create a registration coordinate system.
  • the generation unit 162 calculates the origin and direction through the geometry analysis of the ROI, and defines any one of the front, side, and bottom of the predicted plane in consideration of the relationship with the pose of the depth camera.
  • the first registration coordinate system may be generated by correcting the direction code calculated to match the predetermined left hand coordinate system of the virtual space.
  • the corrector 168 measures based on a matched coordinate system generated in advance, for example, a second matched coordinate system to match the virtual space using the matched coordinate system for the real space or the ROI generated by the generator 162. Correct with
  • the second registration coordinate system may be a registration coordinate system generated from a SLAM (Simultaneous Localization and Mapping) algorithm, and the correction unit 168 uses the distance ratio calculation of the depth camera generating the distance and depth image information of the SLAM-based camera.
  • the second registration coordinate system can be corrected based on the measured basis.
  • the augmentation unit 164 is configured to augment the virtual object based on the corrected matching coordinate system, and augment the virtual object to place the augmented virtual object in the space.
  • the augmentation unit 164 may arrange the virtual object in the space by using a user input through the mobile input device.
  • the apparatus acquires depth image information using the RGB-D camera shown in FIG. 9A, and points a place where the user positions the coordinate system using the mobile input device in an interactive manner using a mobile input device.
  • Select clouds As shown in FIG. 9B, a geometric analysis is performed on a region selected by a user, that is, a region of interest, from a distance map image included in the depth image information to predict a plane, a tangent, a tangent, an intersection point, and the like. Create a registration coordinate system for augmented reality space.
  • the origin and direction are calculated by predicting intersections, tangents, and the like through a predetermined optimization method.
  • the origin and direction are calculated by predicting intersections, tangents, and the like through a predetermined optimization method.
  • the initial registration coordinate system generated from the Simulaneous Localization and Mapping (SLAM) algorithm that is, the second registration coordinate system described above, is corrected with the previously calculated registration coordinate system, and then a camera posture is obtained to obtain a virtual image. Augment objects in real space.
  • the distance ratio of the distance unit based on the depth camera and the distance unit of the SLAM-based camera based on the initial matching coordinate system Calculate
  • the virtual object when the distance ratio is applied when augmenting the virtual object, as illustrated in FIG. 9D, the virtual object may be augmented based on the registration coordinate system by reflecting the unit scale of the real space. For example, the user may arrange the virtual object in space using the mobile input device based on the corrected coordinate system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention comprend les étapes consistant à : générer des coordonnées de référence d'après une image tridimensionnelle comprenant des informations de profondeur acquises au moyen d'une caméra ; diviser une zone correspondant à un objet préconfiguré d'après les informations de profondeur et une conversion d'espace couleur de l'objet préconfiguré à partir d'une image tridimensionnelle comprenant les informations de profondeur acquises au moyen de la caméra ; séparer un sous-objet comprenant un composant de mouvement d'un objet de la zone divisée et modeler le sous-objet séparé et une zone de paume reliée au sous-objet d'après un algorithme préconfiguré pour détecter un point caractéristique ; et contrôler un objet tridimensionnel pour utiliser un service d'interaction de réalité augmentée en estimant une position du sous-objet d'après les informations d'articulation d'un objet fourni au moyen d'une interface utilisateur prédéterminée.
PCT/KR2015/006591 2014-06-26 2015-06-26 Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée WO2015199502A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/322,075 US10304248B2 (en) 2014-06-26 2015-06-26 Apparatus and method for providing augmented reality interaction service

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
KR10-2014-0079107 2014-06-26
KR20140079107 2014-06-26
KR10-2015-0008083 2015-01-16
KR20150008083 2015-01-16
KR10-2015-0091330 2015-06-26
KR1020150091330A KR101865655B1 (ko) 2014-06-26 2015-06-26 증강현실 상호 작용 서비스 제공 장치 및 방법

Publications (1)

Publication Number Publication Date
WO2015199502A1 true WO2015199502A1 (fr) 2015-12-30

Family

ID=54938489

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2015/006591 WO2015199502A1 (fr) 2014-06-26 2015-06-26 Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée

Country Status (1)

Country Link
WO (1) WO2015199502A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105979154A (zh) * 2016-06-28 2016-09-28 广东欧珀移动通信有限公司 一种增强现实ar的拍摄方法、装置及智能终端
CN110322484A (zh) * 2019-05-29 2019-10-11 武汉幻石佳德数码科技有限公司 多设备共享的增强现实虚拟空间的校准方法及系统
CN110597442A (zh) * 2019-09-20 2019-12-20 北京华捷艾米科技有限公司 一种手机ar绘画方法及装置
CN110617802A (zh) * 2019-07-26 2019-12-27 北京控制工程研究所 一种星载动目标检测及速度估计方法
CN112164131A (zh) * 2020-09-25 2021-01-01 北京商询科技有限公司 基于Unity引擎的内部结构切面方法、装置、计算机设备
CN112348965A (zh) * 2020-10-27 2021-02-09 维沃移动通信有限公司 成像方法、装置、电子设备及可读存储介质
CN112927330A (zh) * 2021-03-17 2021-06-08 北京七维视觉传媒科技有限公司 用于生成虚拟人体图像的方法和系统
WO2023130047A1 (fr) * 2021-12-30 2023-07-06 Canon U.S.A., Inc. Systèmes et procédés d'appel immersif en réalité virtuelle

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100980202B1 (ko) * 2008-10-30 2010-09-07 한양대학교 산학협력단 3차원 가상물체와 인터랙션이 가능한 모바일 증강현실 시스템 및 방법
KR20120046607A (ko) * 2010-11-02 2012-05-10 한국전자통신연구원 센서리스 기반 손가락 모션 트래킹 방법 및 그 장치
KR20130099317A (ko) * 2012-02-29 2013-09-06 한국전자통신연구원 인터랙티브 증강현실 구현 시스템 및 증강현실 구현 방법
KR20140001167A (ko) * 2012-06-26 2014-01-06 한국과학기술원 웨어러블 증강현실 환경에서 증강현실 서비스 제공 방법 및 장치
KR20140028064A (ko) * 2011-06-06 2014-03-07 마이크로소프트 코포레이션 펼친 손 또는 오므린 손의 인식 시스템

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR100980202B1 (ko) * 2008-10-30 2010-09-07 한양대학교 산학협력단 3차원 가상물체와 인터랙션이 가능한 모바일 증강현실 시스템 및 방법
KR20120046607A (ko) * 2010-11-02 2012-05-10 한국전자통신연구원 센서리스 기반 손가락 모션 트래킹 방법 및 그 장치
KR20140028064A (ko) * 2011-06-06 2014-03-07 마이크로소프트 코포레이션 펼친 손 또는 오므린 손의 인식 시스템
KR20130099317A (ko) * 2012-02-29 2013-09-06 한국전자통신연구원 인터랙티브 증강현실 구현 시스템 및 증강현실 구현 방법
KR20140001167A (ko) * 2012-06-26 2014-01-06 한국과학기술원 웨어러블 증강현실 환경에서 증강현실 서비스 제공 방법 및 장치

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105979154A (zh) * 2016-06-28 2016-09-28 广东欧珀移动通信有限公司 一种增强现实ar的拍摄方法、装置及智能终端
CN110322484A (zh) * 2019-05-29 2019-10-11 武汉幻石佳德数码科技有限公司 多设备共享的增强现实虚拟空间的校准方法及系统
CN110322484B (zh) * 2019-05-29 2023-09-08 武汉幻石佳德数码科技有限公司 多设备共享的增强现实虚拟空间的校准方法及系统
CN110617802A (zh) * 2019-07-26 2019-12-27 北京控制工程研究所 一种星载动目标检测及速度估计方法
CN110597442A (zh) * 2019-09-20 2019-12-20 北京华捷艾米科技有限公司 一种手机ar绘画方法及装置
CN110597442B (zh) * 2019-09-20 2021-03-16 北京华捷艾米科技有限公司 一种手机ar绘画方法及装置
CN112164131A (zh) * 2020-09-25 2021-01-01 北京商询科技有限公司 基于Unity引擎的内部结构切面方法、装置、计算机设备
CN112164131B (zh) * 2020-09-25 2024-04-05 北京商询科技有限公司 基于Unity引擎的内部结构切面方法、装置、计算机设备
CN112348965A (zh) * 2020-10-27 2021-02-09 维沃移动通信有限公司 成像方法、装置、电子设备及可读存储介质
CN112927330A (zh) * 2021-03-17 2021-06-08 北京七维视觉传媒科技有限公司 用于生成虚拟人体图像的方法和系统
CN112927330B (zh) * 2021-03-17 2024-04-26 北京七维视觉传媒科技有限公司 用于生成虚拟人体图像的方法和系统
WO2023130047A1 (fr) * 2021-12-30 2023-07-06 Canon U.S.A., Inc. Systèmes et procédés d'appel immersif en réalité virtuelle

Similar Documents

Publication Publication Date Title
WO2015199502A1 (fr) Appareil et procédé permettant de fournir un service d'interaction de réalité augmentée
US10304248B2 (en) Apparatus and method for providing augmented reality interaction service
US10732725B2 (en) Method and apparatus of interactive display based on gesture recognition
KR101687017B1 (ko) 머리 착용형 컬러 깊이 카메라를 활용한 손 위치 추정 장치 및 방법, 이를 이용한 맨 손 상호작용 시스템
Cipolla et al. Human-robot interface by pointing with uncalibrated stereo vision
US11308347B2 (en) Method of determining a similarity transformation between first and second coordinates of 3D features
US9734393B2 (en) Gesture-based control system
Lee et al. 3D natural hand interaction for AR applications
WO2016028097A1 (fr) Dispositif pouvant être porté
US9911230B2 (en) Method and apparatus for controlling virtual monitor
KR20150028181A (ko) 포인팅 방향 검출 장치 및 그 방법과, 프로그램 및 컴퓨터 판독가능한 매체
US20160086349A1 (en) Tracking hand pose using forearm-hand model
WO2017007166A1 (fr) Procédé et dispositif de génération d'image projetée et procédé de mappage de pixels d'image et de valeurs de profondeur
KR20080065032A (ko) 제스처 인식 기능을 이용한 반투과 거울 기반 사용자인터페이스 시스템
WO2015199470A1 (fr) Appareil et procédé permettant d'estimer la position d'une main au moyen d'une caméra de profondeur de couleur montée sur la tête, et système d'interaction à mains nues associé
Battisti et al. Seamless bare-hand interaction in mixed reality
KR101426378B1 (ko) 깊이 정보를 이용한 프레젠테이션 이벤트 처리 장치 및 방법
Terajima et al. Fast finger tracking system for in-air typing interface
WO2021002530A1 (fr) Terminal portable permettant de générer des informations de structure d'intérieur sur la base d'un pointage de surface de paroi et son procédé de fonctionnement
Usabiaga et al. Global hand pose estimation by multiple camera ellipse tracking
Nabati et al. Camera mouse implementation using 3D head pose estimation by monocular video camera and 2D to 3D point and line correspondences
WO2015167081A1 (fr) Procédé et dispositif permettant de détecter une partie d'un corps humain
WO2023224326A1 (fr) Dispositif de réalité augmentée pour l'acquisition d'informations de profondeur, et procédé de fonctionnement de celui-ci
Watanabe et al. High-speed estimation of multi-finger position and pose for input interface of the mobile devices
WO2022220344A1 (fr) Procédé et dispositif d'interfaçage pour croquis tridimensionnel

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15812580

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 15322075

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15812580

Country of ref document: EP

Kind code of ref document: A1