WO2023120770A1

WO2023120770A1 - Method and apparatus for interaction between cognitive mesh information generated in three-dimensional space and virtual objects

Info

Publication number: WO2023120770A1
Application number: PCT/KR2021/019669
Authority: WO
Inventors: 한상준
Original assignee: 엔센스코리아주식회사
Priority date: 2021-12-22
Filing date: 2021-12-22
Publication date: 2023-06-29
Also published as: KR20230095197A

Abstract

In order to implement augmented reality technology that synthesizes virtual objects in a certain space, the present invention relates to a method that creates pieces of three-dimensional cognitively segmented mesh information for an interaction between a virtual object and updates information on a position, direction, and state of the virtual object such that the virtual object interacts with a real space, thereby enhancing a user's sense of presence and manipulability. More specifically, three-dimensional space information is generated by estimating a geometric relationship by using continuous images obtained from a camera, and cognitive domain segmentation is performed on the images of the current space by using a convolutional neural network, and this is combined with the space information to generate segmented cognitive mesh information. The present invention relates to a method and an apparatus for implementing an interaction between cognitive mesh information generated in a three-dimensional space and virtual objects by continuously updating location information, direction information, and state information in which the virtual objects disposed in the three-dimensional space may physically exist, determining whether the virtual objects can actually exist at the current location by using the cognitive mesh information to cause the virtual objects and the cognitive mesh information to interact with each other, and comparing the positional relationship between the virtual objects and the 3-dimensional mesh information to represent occlusion.

Description

Method and device for interaction between cognitive mesh information generated in 3D space and virtual objects

The present invention is a technology for implementing real-virtual matching that can synthesize virtual objects as if they exist in a real space, and generates three-dimensional mesh information cognitively segmented from the shape of a real space to create a virtual object. Accurate location information that can exist in real space, and visual and physical interaction between virtual objects and real space gives users the illusion that virtual objects exist in real space, increasing the sense of presence. A method and apparatus for providing real-virtual matched content.

In general, augmented reality technology is a technology field derived from virtual reality technology that synthesizes and overlaps virtual objects on a real space and shows them. It can increase the sense of presence by creating an illusion as if the virtual object actually exists in the real space.

As a first example, a 3D point cloud map is created from a depth image obtained using a depth camera, and an object in a real space where augmented content is to be projected is tracked with the 3D point cloud map, and a display device such as a projector is used to create a virtual image in real space. There is a way to directly overlap objects by projecting them and interact with the user.

As a second example, as a method for generating an event by recognizing a user's motion in a three-dimensional real space, the position value of a target object in virtual space is calculated from a depth image obtained using a depth camera to obtain a reference position database and There is a way to compare and generate an event execution signal.

However, since the methods of the examples above have only 3D spatial information obtained through a camera or sensor, the fragmentary mesh information generated through the 3D point cloud map obtained in this way does not separate the space and the object, and the area corresponding to the space There is a limit to obtaining cognitive information, such as unknown properties.

In addition, in order to augment a virtual object in a real space, there is a limit to implementing augmented reality only according to the user's intention, such as placing an object selected by the user in the space, adjusting the size of the object, or moving it. there is.

The above limitation is to provide a user experience with high presence because interaction with real space is ignored in creating content or scenarios in implementing augmented reality, or arbitrary objects can be expressed only according to a pre-planned script. There are significant problems.

In addition, all of the above examples must be accompanied by a device capable of acquiring depth such as a stereo camera or a depth camera to generate spatial information in a 3D real space. Therefore, there is a problem in that the technology such as the examples described above cannot be implemented in a device that does not have a built-in camera capable of obtaining a depth of field.

In order to overcome these limitations, a method of generating a 3D point cloud map that tracks the position of a 3D space by extracting feature points from 2D continuous images of real space and matching the positions of the continuous images has emerged.

However, this technology also constructs a 3D point cloud map of the real space and then augments and expresses the virtual object, without cognitive division of the space, and calculating an anchor point to maintain the virtual object in the real space. However, since the real space and the virtual object do not interact, there is a problem in that the user cannot feel a high sense of presence.

The present invention implements an augmented reality technology that synthesizes virtual objects in an arbitrary space, and first generates 3D mesh information of the real space for interaction between the virtual object and the real space, and through cognitive domain segmentation. The obtained area information is projected onto a 3D space and used again for area division of the 3D mesh information to generate cognitive mesh information, and then virtual objects use the cognitive mesh information to apply physical characteristics to the location of the object. By realizing the decision and movement and realizing the obscuring of objects by considering the positional relationship between virtual objects and objects in real space, users can obtain a high sense of presence, and users and virtual objects interact with each other in real space. Its purpose is to provide a method that can work.

Prior to the present invention, in the prior art, a real space was recognized based on depth information acquired through an infrared ToF camera, an RGB-D depth camera, a stereo camera, etc. and create depth information by estimating the distance to an object such as a real space or object using the positional relationship between adjacent images, and at this time, among the feature points extracted from each image, the distance to the camera is correctly calculated. We analyze the space by implementing the SLAM (Simultaneous Localization and Mapping) method, which collects the feature points that are determined to be estimated and configures them into a 3D point cloud map.

The present invention performs semantic segmentation on the analyzed spatial information and the two-dimensional image at the same acquisition time using a region segmentation method using a convolutional neural network, and as a result, 2 Dimensional cognitive region segmentation information is projected onto the analyzed space, and the meaning of the region is given to cognitive mesh information from the analyzed spatial information.

Cognitive mesh information is used to calculate anchor points in 3D space so that virtual objects can more naturally exist in real space when arranging virtual objects in real space. It helps to render a virtual object in a realistic way in a situation where some or all of its shape should not be visible because it is hidden in space.

In addition, the virtual object provides additional information so that the real space and the virtual objects can interact with each other using cognitive mesh information. For example, if a table in a real space is recognized through cognitive domain segmentation, the virtual object should be able to express its position by fixing it as if it were real on top of the table. Alternatively, if a potted plant in a real space is recognized, the virtual object should be displayed naturally in harmony with the planter's surroundings, and it should be possible to express that the virtual object is hidden behind the pot and partially or entirely invisible. Alternatively, if the mesh information in the real space has a height of the horizontal plane, the virtual objects move in the direction of gravity by applying the physical laws of the real world when implementing the movement of virtual objects, or the direction of movement is changed by bumping into the floor, wall, or object. You should be able to express what is changing.

In this way, in order to express how virtual objects should interact in real space, the depth of the real space is extracted, a 3D point cloud map is created, mesh information is created from the generated spatial information, and 2 By performing cognitive domain segmentation of a dimensional image, 3D projection and combining for domain division of the mesh information, cognitive mesh information divided according to objects or areas is generated, and the gravity direction acquired through the IMU sensor is combined. to add information that can apply the laws of physics, receive an object creation command from a user's request or a pre-designed scenario, find coordinates where the object can exist in real space, and position it in a 3-dimensional space, The real space and virtual objects that implement augmented reality that synthesizes virtual objects on the two-dimensional image obtained from and outputs them to the display implement interaction with each other.

The present invention relates to an electronic recording medium capable of mounting a camera, an IMU sensor, a display, and an algorithm, which is a method for realizing the augmented reality, in order for virtual objects to interact with each other in a real space, and a method for executing the same. It consists of a device composed of processors.

The device is equipped with an algorithm that can implement augmented reality, which includes image acquisition processing, spatial information generation processing, mesh information generation processing, cognitive area division processing, and cognitive mesh information generation processing. generating cognitive mesh information of the real space from the configured image analyzer; An object expression unit that processes a virtual object creation command through a user's input or a pre-designed scenario, calculates the location of a virtual object, and synthesizes the virtual object by combining cognitive mesh information on the acquired image. It is characterized by being composed of composed object expression steps.

The present invention as described above is a method for enhancing a sense of presence by creating an illusion that a virtual object physically exists in a real space to a user, in terms of providing augmented reality content as virtual objects can interact with each other in a real space. It has become possible to construct immersive scenarios.

It is possible to separate the composition of the real space according to each object by using the 3D mesh information by cognitive domain segmentation, so that the virtual object interacts with the object in the real space, and the virtual object is fixed to the real object. It is not limited to being, being attached, being bumped and bounced off, being piled up, or being not limited thereto, and can have various interactions.

Through this, it is expressed that a virtual wall clock is hung on a real wall in a certain space, or a virtual chair is placed around a real table, and the virtual chair is hidden by the real table according to the position where it is placed. It can be implemented with easy operation according to the user's purpose even if it is not elaborately designed in advance to make some or all invisible, so it can be used as augmented reality technology in various fields such as construction, architecture, interior, 3D product design, and games. can

However, the application of the present invention is not limited to the above-described utilization examples, and the present invention can be applied to fields other than the above embodiments without changing the essential content of the present invention.

1 is a flowchart showing the structure of a device composed of a camera, an IMU sensor, and a display constituting the present invention, and each sub-algorithm for implementing the method of the present invention.

Figure 2 is a flow chart describing in detail the processing of the image analysis unit.

Figure 3 is a flow chart describing in detail the processing process of the object expression unit.

4 is a structural diagram detailing a convolutional neural network that performs cognitive domain segmentation.

The present invention will be described in detail with reference to the accompanying drawings.

1 is an electronic recording medium capable of storing a camera D001 capable of obtaining a two-dimensional color image, an image analysis unit S100 composed of an algorithm for realizing augmented reality, and an object representation unit S200. And a processor that can execute it, an IMU sensor (D002) for adding physical environment information about acceleration and attitude to cognitive mesh information, and a video that can reproduce a synthesized image of a virtual object on a two-dimensional image of a real space It is a flow chart of a device with one display (D003) and its method.

Specifically, the image analysis unit (S100) of the above flow chart acquires a two-dimensional image at regular intervals through the image acquisition process (S110) for obtaining an image from the camera (D001), and converts it into an acquired image (R001). Temporarily remember as The acquired image is divided into the two-dimensional image acquired just before and the two-dimensional image currently acquired and stored, and an algorithm that can find feature points of SIFT, SURF, ORB or similar images from these two images, respectively, and create descriptors of the feature points. Extracts feature information using , compares the similarity between feature information between two images with different viewpoints, obtains a pair of closest feature points, and uses the geometric relationship between paired feature points to obtain information from the t-1 frame Get the relative position of the t frame. At this time, spatial information generation processing of registering continuous frames at regular intervals having a sufficiently large number of feature point pairs as key frames and generating a 3D point cloud map as a cluster of feature points obtained from 3D location information from geometric relationships (S120). ) step. Again, the 3D point cloud map obtains the Euclidean distance of each feature point in the 3D space, and for all feature points, at least two or more nearest pairs among other feature points other than the feature point are obtained, and the three paired feature points are connected into a triangle. A mesh information generating process (S130) for generating mesh information made of connections is executed.

The 2D image obtained in the image acquisition process (S110) is subjected to a cognitive domain segmentation process (S140) using the convolutional neural network configured as shown in FIG. 4 to generate 2D cognitive domain segmentation information.

The 2D cognitive region segmentation information can be treated like a flat image. At this time, the 2D location of each corner extracted using a corner point extraction algorithm such as a fast corner detector is determined in the mesh information generation process (S130). The generated mesh information is projected onto a 3D space where it is converted into 3D information, and the Euclidean distance between the projected 3D coordinates is obtained, and the closest points are connected to each other to create a 3D projected cognitive function. Area segmentation information is generated, the distance between each triangle of the mesh information generated in the mesh information generation process (S130) and the 3D cognitive area segmentation information is obtained, and 3D cognitive area segmentation is performed among triangles adjacent to each other. The three-dimensional coordinates of each corner of the triangle on the information side are added to the mesh information to divide the real space so that it can be expressed in detail. Make the area have meaning about what kind of object it represents, obtain information about the device's posture and the acceleration of the device's movement obtained from the IMU sensor (D002), and calculate the direction in which gravity acts at this time, A cognitive mesh information generation process (S150) is executed, which is added to the cognitive mesh information (R002).

The object expression unit (S200), the user touches an arbitrary position of the display (D003), uses a scenario for object appearance information using a timer or clock, or conditions for interacting with a virtual object , or not limited to the method described in the present invention, by receiving a virtual object creation command by a method of instructing object creation, whether or not a virtual object appears in a temporary storage device, location information, direction information, and state information are created. After executing the stored virtual object creation command processing (S210), arranging the virtual object in the location information input through the creation command, and then recalculating whether there is a change value of the physical location over time, The virtual object position calculation process (S220) for updating the location information, direction information or state information is executed, and the physical effect or masking of the object is performed to make the virtual object feel as if it is real in the real space by using the cognitive mesh information (R002). Virtual object synthesis processing (S230) is performed to synthesize the scene of interaction with the acquired image (R001) and render it as if a virtual object exists in a real space, without being limited to the method described in the present invention. This is output to the display (D003).

Claims

In a device equipped with a camera, an IMU sensor, a display, and an electronic recording medium, the step of acquiring continuous images to analyze the 3D real space with the images acquired by the camera, the step of extracting and matching features from the continuous images, A spatial information generation step consisting of generating 3D spatial information by obtaining a geometric relationship with matched feature pairs from the obtained image, generating mesh information by connecting feature points of the 3D spatial information, and cognitive domain in the acquired image. Generating cognitive domain segmentation information by segmentation; combining the spatial information and cognitive domain segmentation information obtained in the above two steps to generate cognitive segmentation information; A method for generating cognitive mesh information by combining the attitude information of the device obtained using the IMU sensor with the cognitive mesh information
Calculating the coordinates of a space in which a virtual object can physically exist in order to synthesize images obtained in a real space, and the cognitive mesh information generated in claim 1 in order for a virtual object to interact in a real space. Updating the position information, direction information, and state information of the virtual object based on , storing the state of the virtual object to preserve the state, and synthesizing the virtual object on the image obtained in the real space. Virtual object synthesis method consisting of