WO2023074999A1

WO2023074999A1 - System for acquiring 3d ar object image in real time, and operating method therefor

Info

Publication number: WO2023074999A1
Application number: PCT/KR2021/017527
Authority: WO
Inventors: 장준환; 박우출; 양진욱; 윤상필; 최민수; 이준석; 송수호; 구본재
Original assignee: 한국전자기술연구원
Priority date: 2021-10-27
Filing date: 2021-11-25
Publication date: 2023-05-04
Also published as: KR20230060596A; KR102539827B1

Abstract

Disclosed are a method for operating a 3D AR object image collecting apparatus supporting 3D AR object image real-time acquisition, and a system to which the method is applied, the method comprising the steps of: acquiring, by the 3D AR object image collecting apparatus from a plurality of cameras, an RGB image and a depth image for a subject; filtering, from the RGB image and the depth image, RGB partial data corresponding to at least some objects in the subject and depth partial data corresponding to the at least some objects; and generating 3D point cloud data on the basis of the filtered RGB partial data and the filtered depth partial data and storing the 3D point cloud data.

Description

3D AR object image real-time acquisition system and its operating method

The present invention relates to 3D AR object acquisition, and more particularly, to a 3D AR object image real-time acquisition system that supports acquiring 3AR object images in real time and a method for operating the same.

Since the advent of Virtual Reality (VR) and Augmented Reality (AR), interest in various smartphone applications has increased, and Mixed Reality (MR) has been used in various fields, such as education and commercial use. It is becoming.

Meanwhile, in the case of augmented reality, since an augmented image is processed based on an image taken by a camera, the amount of data to be processed is very large. Accordingly, in order to implement augmented reality-based content, there is a problem in that use of augmented reality content is not high because a hardware device with high specifications is required or the augmented reality content must be implemented in a very limited environment. In addition, in the case of conventional augmented reality content, since the amount of data processing is large, real-time properties of content applied to augmented reality are poor.

In order to solve the above problems, the present invention provides a 3D AR object image real-time acquisition system and its operating method capable of acquiring a specific AR object image in real time.

On the other hand, the object of the present invention is not limited to the above object, and other objects not mentioned will be clearly understood from the description below.

The present invention for achieving the above object is a subject, a plurality of cameras each acquiring an RGB image and a depth image of the subject, 3D for the RGB image and the depth image acquired in real time by the plurality of cameras A 3D AR object image collecting device generating an AR object image, wherein the 3D AR object image collecting device converts RGB partial data corresponding to at least some objects among the subjects and depth partial data corresponding to the at least some objects to the RGB image and filtering from the depth image, generating and storing 3D point cloud data based on the filtered RGB partial data and the depth partial data.

Here, the 3D AR object image collection device supports the connection of the plurality of cameras, checks the connection state of the plurality of cameras and the on-off state of the plurality of cameras, and controls the driving of the plurality of cameras. , a data acquisition module acquiring the RGB image and the depth image from the plurality of cameras in real time.

In particular, the camera interface may output error information indicating an inactive or unconnected state of a camera providing the RGB image or the depth image among the plurality of cameras.

On the other hand, the 3D AR object image collection device filters the RGB partial data corresponding to a specific object selected from the RGB image and the depth partial data corresponding to the specific object, or filters an object located at a specific depth among the depth images. A correction module filtering the depth part data and the RGB part data corresponding to the object of the specific depth may be included.

In addition, the 3D AR object image collection device includes a data sync module that converts the resolution of the RGB partial data based on the resolution of the depth partial data to match the resolution of the depth partial data with the resolution of the RGB partial data. can do.

According to an embodiment of the present invention, a method for operating a 3D AR object image collection device supporting real-time acquisition of 3D AR object images includes an RGB image and a depth image of a subject from a plurality of cameras. obtaining, filtering RGB partial data corresponding to at least some objects among the subjects and depth partial data corresponding to the at least some objects from the RGB image and the depth image; and generating and storing 3D point cloud data based on partial data.

The method includes checking a deactivation or disconnection state of a camera providing the RGB image or the depth image among the plurality of cameras, and outputting error information corresponding to the deactivation or disconnection state of the camera. can include more.

Meanwhile, the filtering may include filtering the RGB partial data corresponding to a specific object selected from the RGB image and the depth partial data corresponding to the specific object, or the depth corresponding to an object located at a specific depth among the depth images. Filtering the partial data and the RGB partial data corresponding to the object of the specific depth may be included.

In this case, the method may further include converting the resolution of the RGB partial data based on the resolution of the depth partial data so that the resolution of the depth partial data matches the resolution of the RGB partial data.

According to the present invention, a 3D object can be easily created by combining color data and depth data.

In addition, the present invention can provide a real-time 3D AR object that can be applied to various additional services, such as E-Sports relay, home fitness and patient management, and smart factory operation, by solving physical limitations.

In addition, various effects other than the above effects may be disclosed directly or implicitly in detailed descriptions according to embodiments of the present invention to be described later.

1 is a diagram showing an example of a 3D AR object image collection environment capable of acquiring 3D AR object images in real time according to an embodiment of the present invention.

2 is a diagram showing an example of a camera interface of a 3D AR object image collection device according to an embodiment of the present invention.

3 is a diagram showing an example of data change between a data acquisition module and a correction module in a 3D AR object image collection device according to an embodiment of the present invention.

4 is a diagram showing an example of a configuration of a data sync module in a 3D AR object image collection device according to an embodiment of the present invention.

5 is a diagram showing an example of a data generation module in a 3D AR object image collection device according to an embodiment of the present invention.

6 is a diagram illustrating an example of a method for obtaining a 3D AR object image in real time according to an embodiment of the present invention.

In order to clarify the characteristics and advantages of the problem solving means of the present invention, the present invention will be described in more detail with reference to specific embodiments of the present invention shown in the accompanying drawings.

However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention will be omitted in the following description and accompanying drawings. In addition, it should be noted that the same components are indicated by the same reference numerals throughout the drawings as much as possible.

The terms or words used in the following description and drawings should not be construed as being limited to a common or dictionary meaning, and the inventor may appropriately define the concept of terms for explaining his/her invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principle that there is. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention, and do not represent all of the technical ideas of the present invention. It should be understood that there may be equivalents and variations.

In addition, terms including ordinal numbers, such as first and second, are used to describe various components, and are used only for the purpose of distinguishing one component from other components, and to limit the components. Not used. For example, a second element may be termed a first element, and similarly, a first element may be termed a second element, without departing from the scope of the present invention.

In addition, terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "include" or "having" described in this specification are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or the other It should be understood that the above does not preclude the possibility of the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

In addition, terms such as “unit”, “unit”, and “module” described in the specification refer to a unit that processes at least one function or operation, and may be implemented as hardware or software or a combination of hardware and software. Also, "a or an", "one", "the" and similar related words in the context of describing the invention (particularly in the context of the claims below) Unless indicated or otherwise clearly contradicted by context, both the singular and the plural can be used.

In addition to the above-mentioned terms, specific terms used in the following description are provided to aid understanding of the present invention, and the use of these specific terms may be changed in other forms without departing from the technical spirit of the present invention.

In addition, embodiments within the scope of the present invention include computer-readable media having or conveying computer-executable instructions or data structures stored thereon. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. physical storage media such as, but not limited to, any other medium that can be used to store or convey any program code means in a computer system and which can be accessed by a general purpose or special purpose computer system. .

Hereinafter, the system, apparatus and method constituting the present invention will be described in more detail through each drawing.

Referring to FIG. 1 , a 3D AR object image collection environment 10 according to an embodiment of the present invention may include a subject 11, a plurality of cameras 100, and a 3D AR object image collection device 200. Here, the plurality of cameras 100 may be included as one component of the 3D AR object image collecting device 200 .

The subject 11 may be a target to be made into a 3D AR object. For example, the subject 11 may include various targets such as people, animals, objects, tools, and backgrounds. Alternatively, the subject 11 may include at least a part of a photographing environment captured by the plurality of cameras 100 .

The plurality of cameras 100 may be disposed at positions capable of photographing the subject 11 . The plurality of cameras 100 may include at least one RGB camera capable of obtaining an RGB image related to the subject 11 and at least one depth camera capable of obtaining depth data of the subject 11. . The at least one RGB camera and the at least one depth camera are disposed to capture the same subject 11 and may have the same or similar shooting distance and shooting angle within a specified range.

The 3D AR object image collection device 200 acquires RGB images and depth images obtained by the plurality of cameras 100 in real time, performs filtering on a target object on the obtained real-time images, and then performs data synchronization. Point cloud data can be generated and provided through In this regard, the 3D AR object image collection device 200 may include a camera interface 210, a data acquisition module 220, a correction module 230, a data sync module 240, and a data generation module 250. can

In the 3D AR object image collection environment 10 of the present invention described above, the plurality of cameras 100 control the 3D AR object image collection device 200 or the user who controls the 3D AR object image collection device 200 Activated in response to manipulation, the activated plurality of cameras 100 may collect images of the subject 11 and provide the collected images to the 3D AR object image collection device 200 .

Referring to FIG. 2 , the camera interface 210 may include a camera connection unit 211 , a driving unit 212 and an error handling unit 213 .

The camera connection unit 211 may include a configuration for connecting the plurality of cameras 100 and the 3D AR object image collection device 200. For example, the camera connection part 211 may include a wired cable inserted into a connector formed in the plurality of cameras 100 and a connection pin or device connector connected to the wired cable. Alternatively, when the plurality of cameras 100 are configured to support image transfer through wireless communication, the camera connection unit 211 may include a wireless communication interface capable of forming a wireless communication channel with the plurality of cameras 100. can That is, the camera connection unit 211 may include a wired communication interface (eg, connection through a cable) that can be connected to at least one camera or a wireless communication interface that can form a wireless communication channel with at least one camera. . When the plurality of cameras 100 do not include a separate battery, the camera connection part 211 may further include a wire capable of supplying power to the plurality of cameras 100 .

The driving unit 212 may be in charge of setting the number of cameras used, setting values, and controlling On/Off. In this regard, when the plurality of cameras 100 are connected through the camera connection unit 211, the driving unit 212 transmits and receives initial data with the plurality of cameras 100 connected to the identification information of the plurality of cameras 100. , the type and specification information of each camera can be collected. Also, the driving unit 212 may drive an application for driving the plurality of cameras 100 . The driving unit 212 may perform initial setting of the plurality of cameras 100 according to predefined settings and may adjust setting values of each camera according to user manipulation. The drive unit 212 controls turn-on of the plurality of cameras 100 according to user manipulation or execution of a specific application (eg, an application supporting a 3D AR object image providing function), and in response to termination of the corresponding application, a plurality of Turn-off control of the camera 100 may be performed.

The error handling unit 213 may process an error occurring in physical connection with the plurality of cameras 100 or in setting a setting value. In this regard, the camera interface 210 may include a display or audio device capable of displaying information related to connection of the plurality of cameras 100 . The error handling unit 213 outputs a list of the plurality of cameras 100 connected according to the connection of the plurality of cameras 100, and the status of each of the plurality of cameras 100 (eg, initialization completed, the plurality of cameras 100 ) can be output. During this process, the error handling unit 213 may also output identification information and error information about a camera that does not operate normally among the plurality of cameras 100 . The error information may include a problem of a camera in which an error has occurred.

Referring to FIG. 3 , the data acquisition module 220 may acquire at least an RGB image and a depth image from among the plurality of cameras 100 according to a user's manipulation or execution of a specific application. In this process, the data acquisition module 220 checks information on the plurality of cameras 100 that are connected and activated through the camera interface 210, and the camera capable of acquiring RGB images from the plurality of cameras 100 and You can check if there is a camera capable of acquiring depth images. The data acquisition module 220 transmits an error message to the error handling unit 213 of the camera interface 210 when a camera capable of obtaining an RGB image and a depth image, respectively, is not included in the plurality of cameras 100. can be forwarded to The error handling unit 213 receiving the error message may output error information requesting connection or turn-on of a camera of a type that is not currently connected (eg, an RGB camera or a depth camera). When both the RGB camera and the depth camera are connected and the RGB image and the depth image are normally transmitted, the data acquisition module 220 may transmit the acquired RGB image and the depth image correction module 230. At this time, the RGB image and the depth image delivered by the data acquisition module 220 may include images of the entire shooting environment including the subject 11 .

The correction module 230 may perform filtering to acquire only desired data based on the RGB image and the depth image delivered by the data acquisition module 220 . During this process, the correction module 230 may filter subjects at a certain depth by using the depth value. The correction module 230 may separately extract an RGB image of a subject at a certain depth. In this regard, the correction module 230 may perform object classification based on boundary lines in the RGB image, and selectively filter objects having a certain depth based on depth information. Alternatively, the correction module 230 may perform filtering on a corresponding object based on depth information on an object (eg, a human face pattern or a specific animal pattern) including a specific pattern among RGB images. That is, the correction module 230 may filter and obtain only RGB data and depth data of at least some of the objects 11 .

Referring to FIG. 4 , the data sync module 240 may include an info handler 241 , a pixel matching unit 242 and a first data buffer 243 .

The info handler 241 may obtain depth data and RGB data from data transmitted from the correction module 230 . In this process, the info handler 241 may obtain resolution information of depth data and resolution of RGB data from the correction module 230 together. Alternatively, the info handler 241 may obtain the resolution information of the RGB camera that provided the RGB image and the resolution information of the depth camera that provided the depth image from the camera interface 210 .

The pixel matcher 242 (Pixel Matcher) can change the resolution of RGB data. That is, the pixel matching unit 242 converts the size of RGB information among RGB information having different resolutions (resolution: 1920*1080) and depth information (resolution: 1024*1024) to match the depth information to generate a point cloud. can do. The pixel matching unit 242 may store changed information in the first data buffer 243 .

The first data buffer 243 may store RGB data and depth data matched by the pixel matching unit 242 . The RGB data stored here may be information converted to suit the resolution of depth data.

Referring to FIG. 5, the data generation module 250 of the present invention includes a second data buffer 251, a point cloud generator 252, a point cloud data buffer 253, a video output generator 254, and a data transmitter 255. ) and PLY file manager 256.

The second data buffer 251 may receive RGB data and the depth data converted based on the depth data by the data sync module 240 from the first data buffer 243 and temporarily store them. The second data buffer 251 may transmit stored data (eg, resolution-converted RGB data and depth data) to the point cloud generator 252 . At this time, the second data buffer 251 stores the data received from the first data buffer 243 in a specified data format format (eg, a format format defined for point cloud generation or the purpose of using the data) for point cloud generation. format) can be converted and saved.

The point cloud generator 252 may generate point cloud data based on data stored in the second data buffer 251 . For example, the point cloud generator 252 may allocate coordinates in a 3D space to data stored in the second data buffer 251 and process a collection of points for the allocated coordinates. Here, the point cloud generator 252 may perform an overlay process on the point cloud to build 3D data.

The point cloud data buffer 253 may store point cloud data generated by the point cloud generator 252 . Data stored in the point cloud data buffer 253 may include collections of coordinates (or points) corresponding to the 3D AR object image.

The video output generator 254 may generate a video file corresponding to the point cloud data stored in the point cloud data buffer 253, and store the generated video file or transmit it to a designated device. The video generated by the video output generator 254 may include a real-time image of the 3D AR object image. In particular, the video file generated by the video output generator 254 may include real-time video images of 3D AR object images of some objects based on the subject 11 .

The data transmitter 255 may transmit the point cloud data stored in the point cloud data buffer 253 to a designated device. For example, when a 3D AR object image supporting function is performed based on a program such as a video conference or game, the data transmitter 255 transmits the point cloud data corresponding to a specific object to other electronic devices performing the corresponding video conference or game. can be passed to the device.

The PLY file manager 256 may create a 3D polygon file based on the point cloud data. The PLY file manager 256 may define a polygonal file format corresponding to at least some objects of the subject 11 based on the point cloud data. In this process, the PLY file manager 256 may store various properties including color and transparency of an object corresponding to at least a part of the subject 11, surface normal, texture coordinates, and data reliability value. That is, the PLY file manager 256 may perform PLY data conversion corresponding to the point cloud data.

Referring to FIG. 6 , in step 601, the 3D AR object image collection device 200 may activate a plurality of cameras 100 including an RGB camera and a depth camera. During this process, the 3D AR object image collection device 200 may activate the plurality of cameras 100 according to user manipulation or execution of an application designed for a 3D AR object image support function.

In step 603, the 3D AR object image collection device 200 may acquire RGB data and depth data in real time. In this operation, the 3D AR object image collection device 200 may obtain time information on RGB data and depth data together, and maintain time synchronization for each data.

In step 605, the 3D AR object image collection device 200 may perform filtering on acquired data. For example, the 3D AR object image collecting device 200 may filter only depth data of an object located at a specific depth and RGB data corresponding to an object at a corresponding depth based on the depth data. Alternatively, the 3D AR object image collection device 200 may detect a specific object based on the RGB image, and filter RGB partial data corresponding to the object and depth partial data corresponding to the object. After selecting a specific object, the 3D AR object image collecting device 200 may filter data (eg, RGB partial data and depth partial data) of the corresponding object with respect to data obtained in real time.

In step 607, the 3D AR object image collection device 200 may process data sync. For example, the 3D AR object image collection device 200 may synchronize resolution conversion of RGB data based on the resolution of depth data. That is, the resolution of the RGB image acquired by the RGB camera and the depth image obtained by the depth camera may differ depending on the characteristics of the cameras. Accordingly, the 3D AR object image collection device 200 may process a conversion task of synchronizing the resolution of RGB data based on the resolution of depth data. At this time, the 3D AR object image collection device 200 may process resolution conversion (synchronization or matching operation) of the filtered RGB partial data based on the filtered depth partial data.

In step 609, the 3D AR object image collection device 200 may generate and store point cloud data. To this end, the 3D AR object image collection device 200 includes a point cloud generator 252, and uses the point cloud generator 252 to convert the RGB partial data and the depth partial data to points corresponding to 3D coordinates. It can be converted to , and collection processing of the converted points can be performed. In this operation, the point cloud generator 252 may generate a specific file (eg, a video file or a PLY file) by overlaying point clouds corresponding to RGB partial data and depth partial data obtained in real time. .

As set forth above, this specification contains many specific implementation details, but these should not be construed as limiting on the scope of any invention or claimables, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features in

Further, while operations are depicted in the drawings in a specific order, it should not be understood that all illustrated operations must be performed or that those operations must be performed in the specific order shown or in sequential order to obtain desired results. In certain cases, multitasking and parallel processing can be advantageous. Further, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally integrated together into a single software product or packaged into multiple software products. You have to understand that it can be.

The present description presents the best mode of the invention and provides examples to illustrate the invention and to enable those skilled in the art to make and use the invention. The specification thus prepared does not limit the invention to the specific terms presented. Therefore, although the present invention has been described in detail with reference to the above-described examples, a person skilled in the art may make alterations, changes, and modifications to the present examples without departing from the scope of the present invention.

Therefore, the scope of the present invention should not be determined by the described embodiments, but by the claims.

Claims

subject;

a plurality of cameras each acquiring an RGB image and a depth image of the subject;

A 3D AR object image collection device for generating a 3D AR object image for the RGB image and the depth image acquired by the plurality of cameras in real time;

The 3D AR object image collection device

RGB partial data corresponding to at least some of the objects and depth partial data corresponding to the at least some objects are filtered from the RGB image and the depth image, and a 3D point is obtained based on the filtered RGB partial data and the depth partial data. A 3D AR object image real-time acquisition system, characterized in that for generating and storing cloud data.
According to claim 1,

The 3D AR object image collection device

a camera interface that supports connection of the plurality of cameras, checks a connection state of the plurality of cameras and an on-off state of the plurality of cameras, and controls driving of the plurality of cameras;

A 3D AR object image real-time acquisition system comprising a; data acquisition module for acquiring the RGB image and the depth image from the plurality of cameras in real time.
According to claim 2,

The camera interface is

3D AR object image real-time acquisition system, characterized in that for outputting error information indicating an inactive or unconnected state of a camera providing the RGB image or the depth image among the plurality of cameras.
According to claim 1,

The 3D AR object image collection device

filtering the RGB partial data corresponding to a specific object selected from the RGB image and the depth partial data corresponding to the specific object;

A 3D AR object image real-time acquisition system comprising: a correction module for filtering the depth part data corresponding to an object located at a specific depth and the RGB part data corresponding to an object of the specific depth among the depth images.
According to claim 4,

The 3D AR object image collection device

and a data sync module for converting the resolution of the RGB partial data based on the resolution of the depth partial data and matching the resolution of the depth partial data with the resolution of the RGB partial data. acquisition system.
In the method of operating a 3D AR object image collection device supporting real-time acquisition of 3D AR object images,

obtaining, by the 3D AR object image collection device, RGB images and depth images of a subject from a plurality of cameras;

filtering RGB partial data corresponding to at least some objects among the subjects and depth partial data corresponding to the at least some objects from the RGB image and the depth image;

A method of operating a 3D AR object image collection device comprising: generating and storing 3D point cloud data based on the filtered RGB partial data and the depth partial data.
According to claim 6,

Checking a deactivated or unconnected state of a camera providing the RGB image or the depth image among the plurality of cameras;

The operation method of the 3D AR object image collection device, characterized in that it further comprises; outputting error information corresponding to the inactive or unconnected state of the camera.
According to claim 6,

The filtering step is

filtering the RGB partial data corresponding to a specific object selected from the RGB image and the depth partial data corresponding to the specific object; or

Filtering the depth part data corresponding to an object located at a specific depth and the RGB part data corresponding to an object of the specific depth among the depth images; Operating method of a 3D AR object image collection device comprising:.
According to claim 8,

The 3D AR object image collection device further comprising a step of converting the resolution of the RGB partial data based on the resolution of the depth partial data and matching the resolution of the depth partial data with the resolution of the RGB partial data. How to operate.