KR102582349B1 - The apparatus and method for correcting error be caused by overlap of object in spatial augmented reality - Google Patents

Abstract

An error correction device due to object overlap in spatial augmented reality according to an embodiment of the present invention includes an image acquisition unit for acquiring an image containing an object, an object identification unit for identifying an object in the acquired image, and an identified object. An ID allocation unit that allocates the generated ID, a skeletal information extraction unit to extract skeletal information about the identified object, an object tracking unit to track the movement of the object in real time based on the extracted skeletal information, and acquisition. An overlap estimation unit for estimating the degree of overlap between a plurality of objects included in an image, a correction unit for correcting errors in skeletal information about overlapping parts of objects based on the estimated overlap, and a correction unit for correcting errors through the correction unit. It may include an object restoration unit for restoring the object using skeleton information. Additionally, the ID assigned to the restored object may be maintained.

Description

Error correction device and method according to object overlap in spatial augmented reality {THE APPARATUS AND METHOD FOR CORRECTING ERROR BE CAUSED BY OVERLAP OF OBJECT IN SPATIAL AUGMENTED REALITY}

The present invention relates to technology for tracking and processing recognized objects, and more specifically, to an apparatus and method for correcting errors due to object overlap in spatial augmented reality.

Recently, much research has been conducted on overlapping processing technology used to recognize and track people in the field of spatial augmented reality. The overlapping processing technology generated to recognize and track people can be applied to many areas, including human-machine interaction and educational assistance.

Currently used technology for processing overlap phenomenon that occurs in recognizing and tracking people is divided into a method of handling overlap when it occurs as an error, a method of initializing input data, and a method of creating a new overlapped person.

The way to handle it as an error is that only one person must appear, and if more than two people are recognized, an error is generated and the user is asked to have only one person in front, making it impossible for more than two people to participate.

The method of initializing input data can provide stable human skeleton information by initializing the input data, but it is cumbersome in that it must be determined under what circumstances initialization is necessary.

The method of creating new overlapping people is effective for tracking people, but has the disadvantage of making continuous tracking of objects difficult because new unique identification numbers are assigned.

As a response to the above-described problem, the present invention seeks to provide an effective processing technology for overlapping parts to accurately recognize a plurality of objects and continuously and stably track the objects even when objects overlap.

As an embodiment of the present invention, an error correction apparatus and method according to object overlap in spatial augmented reality may be provided.

An error correction device due to object overlap in spatial augmented reality according to an embodiment of the present invention includes an image acquisition unit for acquiring an image containing an object, an object identification unit for identifying an object in the acquired image, and an identified object. An ID allocation unit that allocates the generated ID, a skeletal information extraction unit to extract skeletal information about the identified object, an object tracking unit to track the movement of the object in real time based on the extracted skeletal information, and acquisition. An overlap estimation unit for estimating the degree of overlap between a plurality of objects included in an image, a correction unit for correcting errors in skeletal information about overlapping parts of objects based on the estimated overlap, and a correction unit for correcting errors through the correction unit. It may include an object restoration unit for restoring the object using skeleton information. Additionally, the ID assigned to the restored object may be maintained.

An image according to an embodiment of the present invention may include at least one object, and the object identification unit may identify the object from the image based on depth information of the object.

The object according to an embodiment of the present invention is a person, and skeleton information can be extracted by pairing the coordinates of the start and end points of the object's skeleton.

The degree of overlap according to an embodiment of the present invention may be estimated based on the difference between the skeleton information before an object overlaps with another object and the skeleton information after the object overlaps.

The degree of overlap according to an embodiment of the present invention may be estimated based on the ratio of the skeleton information extracted during overlap to the entire skeleton information of the object.

Additionally, the correction unit determines the skeletal information of the object to be corrected based on the estimated degree of overlap and corrects errors in the determined skeletal information. The error in the skeletal information may be an error in the position value of the object due to the overlap.

A method for correcting errors due to object overlap in spatial augmented reality according to an embodiment of the present invention includes the steps of acquiring an image containing an object, identifying the object in the acquired image, and Allocating the generated ID, extracting skeletal information about the identified object, tracking the movement of the object in real time based on the extracted skeletal information, overlapping between a plurality of objects included in the acquired image It includes a step of estimating a degree, a step of correcting an error in skeletal information about an overlapping part of an object based on the estimated degree of overlap, and a step of restoring an object using the corrected skeletal information, and assigning The ID can be maintained.

Meanwhile, as an embodiment of the present invention, a computer-readable recording medium recording a program for executing the above-described method on a computer may be provided.

By using an error correction apparatus and method for object overlap in spatial augmented reality according to an embodiment of the present invention, stable provision of multimedia content can be realized even in a spatial augmented reality environment in which a large number of people participate, and data about objects can be realized. High accuracy can be achieved during processing.

In addition, it is possible to effectively prevent error message processing due to overlapping objects or distortion or loss of multimedia content.

An error correction method for the overlapping phenomenon of objects recognized based on spatial augmented reality according to an embodiment of the present invention analyzes image information overlapping with an object (e.g., a person) and maps it with multimedia content (e.g., a character, etc.). Error phenomena such as unnatural movements of characters can be prevented from occurring in image processing techniques.

Figure 1 is a block diagram of an error correction device based on object overlap in spatial augmented reality according to an embodiment of the present invention.
Figure 2 shows an ID allocation unit according to an embodiment of the present invention.
Figure 3 shows an overlap estimation unit according to an embodiment of the present invention.
Figure 4 shows an object restoration unit according to an embodiment of the present invention.
Figure 5 shows a case where the first object and the second object do not overlap.
Figure 6 shows a case where the arms and waist portions of the first object and the second object overlap.
Figure 7 shows a case where the arms and legs of the first object and the second object overlap.
Figure 8 shows a case where the first object and the second object overlap more than 90 percent (%).
Figure 9 shows a case where part of the second object overlaps the first object.
Figure 10 is a flowchart showing a method for correcting errors due to object overlap in spatial augmented reality according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

The terms used in this specification will be briefly explained, and the present invention will be described in detail.

The terms used in the present invention are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

When it is said that a part "includes" a certain element throughout the specification, this means that, unless specifically stated to the contrary, it does not exclude other elements but may further include other elements. In addition, terms such as "... unit" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. . In addition, when a part is said to be "connected" to another part throughout the specification, this includes not only the case where it is "directly connected," but also the case where it is connected "with another element in between."

In this specification, “object” may refer to a person, animal, object, etc. included in an image that can serve as a standard element during image processing. Objects included in an image sometimes have similar colors or textures, but often have different colors and textures. Additionally, there are cases where objects are included in a complex background. In this way, backgrounds and objects can coexist in images, and techniques to clearly extract and track objects by distinguishing them from the background can be very important in image processing.

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a block diagram of an error correction device according to object overlap in spatial augmented reality according to an embodiment of the present invention, and Figure 2 shows an ID allocation unit according to an embodiment of the present invention.

An error correction apparatus 1000 according to object overlap in spatial augmented reality according to an embodiment of the present invention includes an image acquisition unit 1100 for acquiring an image containing an object, and an object for identifying the object in the acquired image. An identification unit 1200, an ID allocation unit 1300 for assigning an ID generated to an identified object, a skeletal information extraction unit 1400 for extracting skeletal information about an identified object, based on the extracted skeletal information An object tracking unit 1500 for tracking the movement of an object in real time, an overlap estimation unit 1600 for estimating the degree of overlap between a plurality of objects included in the acquired image, and an object tracking unit 1600 for estimating the degree of overlap between a plurality of objects included in the acquired image. It may include a correction unit 1700 for correcting errors in skeletal information about overlapping parts, and an object restoration unit 1800 for restoring the object using the skeletal information corrected through the correction unit. Additionally, the ID assigned to the restored object may be maintained.

The image acquisition unit 1100 according to an embodiment of the present invention can acquire an image of an object using a photographing device such as an RGB-D camera. The acquired image may be referred to as a real image.

An image according to an embodiment of the present invention may include at least one object, and the object identification unit 1200 may identify the object from the image based on depth information of the object. In other words, the depth information for the real image may have already been acquired by the image acquisition unit 1100, and the object identification unit 1200 may identify the object and the background or the object and other objects based on the acquired depth information. there is. For example, the object identification unit 1200 can identify objects representing backgrounds or objects and people, respectively, and recognize only objects representing people.

The ID allocation unit 1300 according to an embodiment of the present invention may include an object structure identification unit 1310, an object counting unit 1320, and an ID distribution unit 1330. The object structure identification unit 1310 can generate a default value to indicate the structure of the object for each identified object, and identify the structure of the object based on the generated default value. For example, in an object representing a person, the object structure may be identified based on information about the entire skeleton of the object represented around a skeletal point near the lower abdomen or cervical spine. The object structure can be identified by extracting the object's skeleton using the object's depth information.

The object counting unit 1320 can count the number of objects with identified object structures in the image. Depending on the counted objects, whether or not they overlap can be determined and the degree of overlap can be estimated. The ID distribution unit 1330 generates an ID to be assigned to each identified object, and assigns the generated ID to each object, so that the assigned ID can be used to track the movement of the object, determine whether or not it overlaps, etc. The ID given to an object can be set to have a unique value for each object in the video. Therefore, even if the image includes a plurality of objects, each object can be identified through an ID assigned to each object.

Figure 3 shows an overlap estimation unit according to an embodiment of the present invention.

The overlap degree estimation unit 1600 according to an embodiment of the present invention includes a skeleton information collection unit 1610, an overlap judgment criteria generation unit 1620, an overlap skeleton counting unit 1630, and an overlap ratio estimation unit 1640. It can be included.

The object according to an embodiment of the present invention is a person, and skeletal information can be extracted as 25 coordinate pairs by pairing the coordinates of the start and end points of the object's skeleton. For example, the skeletal information of an object may be expressed as a pair of coordinate information of a skeleton extending from a skeletal point in the vicinity of the lower abdomen or the cervical spine as described above. In video or spatial augmented reality, a plurality of objects may overlap each other due to the movement of the objects. According to an embodiment of the present invention, as a criterion for determining overlap, the overlap determination criterion generator 1620 calculates the number of remaining bones excluded from the count due to overlapping with other objects from the total number of collected bones of the object as described above. It can be set (created) as a standard. In other words, whether objects overlap can be determined based on the loss of skeletal information due to overlap. For example, if the number of remaining non-overlapping bones out of the total number of bones is 24 or less, the object may be determined to be overlapped. In other words, whether all or part of the object overlaps can be determined depending on whether the number of non-overlapping skeletons is 24 or more or less.

In order to determine the number of overlapping skeletons, the overlapping skeleton counter 1630 uses the difference between the total number of skeletons and the number of currently identified skeletons (e.g., the number of remaining non-overlapping skeletons) to determine the number of overlapping skeletons. You can.

The overlap ratio estimation unit 1640 can estimate the ratio of the number of overlapping skeletons to the total number of skeletons. The estimated ratio can be used to correct position values due to overlap.

The degree of overlap according to an embodiment of the present invention may be estimated based on the difference between the skeleton information before an object overlaps with another object and the skeleton information after the object overlaps. Additionally, the degree of overlap according to an embodiment of the present invention may be estimated based on the ratio of the skeleton information extracted during overlap to the entire skeleton information of the object.

In addition, according to an embodiment of the present invention, the correction unit 1700 determines the skeletal information of the object to be corrected based on the estimated degree of overlap and corrects the error in the determined skeletal information, and the error in the skeletal information is This may be an error in the position value of the object due to overlap. Skeleton information may be overlapping bone parts, number, position values, etc.

According to an embodiment of the present invention, the location of the skeleton lost due to overlap is determined, and the location of the lost skeleton described above can be estimated using already obtained skeleton information (e.g., basic skeleton information) of the object. . Since the skeletal information of the object is collected by the skeletal information collection unit 1610, the skeletal information of the object can be prevented from being arbitrarily changed even if it overlaps with other objects due to movement of the object. In other words, even if a plurality of objects overlap each other and it is difficult to accurately extract the skeletal information of the object even using depth information, the position value, etc. can be estimated using the basic skeletal information already obtained for the object.

Figure 4 shows an object restoration unit according to an embodiment of the present invention.

The object restoration unit 1800 according to an embodiment of the present invention may include a correction value reception unit 1810, a correction target determination unit 1820, and a calculation unit 1830. The correction value receiver 1810 may collect (receive) data for correction of the position value generated by the correction unit 1700 based on the degree of overlap. Additionally, the correction value receiving unit 1810 may receive reference information for identifying the object structure, skeletal information, etc. from the object structure identification unit 1310, the skeletal information collection unit 1610, etc.

The correction target determination unit 1820 may determine an area to be corrected (eg, overlapping skeleton position) based on the overlapping skeleton portion, number, and position value. The calculation unit 1830 may correct the position value of the overlapping skeleton using the correction value received through the correction value receiver 1810 and perform an operation to restore the object based on the corrected position value. In other words, restoration of the object can be performed using the corrected position value, and according to an embodiment of the present invention, distortion (eg, distortion, etc.) of the object can be minimized in spatial augmented reality compared to the prior art.

FIG. 5 shows a case where the first object and the second object do not overlap, and FIG. 6 shows a case where the arms and waist portions of the first object and the second object overlap (eg, green dotted line portion). In addition, Figure 7 shows a case where the arms and legs of the first object and the second object overlap (e.g., green dotted line portion), and Figure 8 shows a case where the first object and the second object overlap more than 90 percent (%). For example, the green dotted line indicates the case). Figure 9 shows a case where a portion of the second object overlaps the first object (eg, green dotted line portion).

As shown in FIG. 5, when objects do not overlap each other, errors rarely occur during image processing for objects in spatial augmented reality. In contrast, when objects overlap (e.g., green dotted line portion) as shown in FIGS. 6 to 9, errors may occur in object identification, transformation mapping, etc. when processing images of the objects. According to an embodiment of the present invention, the degree to which objects overlap, the overlapping parts (e.g., position values), etc. are identified, and errors in position values that may occur due to such overlap are corrected to restore the object. Objects can be expressed more naturally and accurately than before.

Figure 10 is a flowchart showing a method for correcting errors due to object overlap in spatial augmented reality according to an embodiment of the present invention.

A method for correcting errors due to object overlap in spatial augmented reality according to an embodiment of the present invention includes the steps of acquiring an image containing an object (S100) and identifying the object in the acquired image (S200). , assigning the generated ID to the identified object (S300), extracting skeletal information about the identified object (S400), tracking the movement of the object in real time based on the extracted skeletal information ( S500), estimating the degree of overlap between a plurality of objects included in the acquired image (S600), correcting errors in skeletal information about overlapping parts of objects based on the estimated degree of overlap (S700), and correction It includes a step (S800) of restoring the object using the skeleton information, and the ID assigned to the restored object may be maintained.

Regarding the method according to an embodiment of the present invention, the information regarding the above-described device may be applied. Therefore, in relation to the method, description of content that is the same as that of the device described above has been omitted.

One embodiment of the present invention may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer.
Meanwhile, computer-readable media may be any available media that can be accessed by a computer, and includes both volatile and non-volatile media, removable and non-removable media.
Here, computer-readable media may include computer recording media and communication media.
The recording media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
The communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: first object 20: second object
1000: Error correction device for object overlap in spatial augmented reality
1100: Image acquisition unit 1200: Object identification unit
1300: ID allocation unit
1310: Object structure identification unit 1320: Object counting unit
1330: ID distribution unit
1400: Skeleton information extraction unit 1500: Object tracking unit
1600: Overlap degree estimation unit
1610: Skeleton information collection unit 1620: Overlapping judgment criteria generation unit
1630: Overlapping skeleton counting unit 1640: Overlapping ratio estimation unit
1700: Compensation unit
1800: Object restoration unit
1810: Correction value receiving unit 1820: Correction target determination unit
1830: Arithmetic unit

Claims (8)

In an error correction device due to object overlap in spatial augmented reality,
An image acquisition unit for acquiring an image containing an object;
an object identification unit for identifying the object in the acquired image;
an ID allocation unit that allocates a generated ID to the identified object;
a skeletal information extraction unit for extracting skeletal information about the identified object;
an object tracking unit for tracking the movement of the object in real time based on the extracted skeleton information;
an overlap estimation unit for estimating a degree of overlap between a plurality of objects included in the acquired image;
a correction unit for correcting an error in skeletal information about an overlapping portion of the object based on the estimated degree of overlap; and
An object restoration unit for restoring the object using skeleton information corrected through the correction unit,
An error correction device due to object overlap in spatial augmented reality, wherein the assigned ID is maintained for the restored object.
According to claim 1,
The image may include at least one object,
An error correction device according to object overlap in spatial augmented reality, wherein the object identification unit identifies an object from the image based on depth information of the object.
According to claim 1,
The object is a person,
An error correction device according to object overlap in spatial augmented reality, characterized in that the skeleton information is extracted by pairing the coordinates of the starting point and the end point of the skeleton of the object.
According to claim 1,
The degree of overlap is estimated based on a difference value between skeletal information before the object overlaps with another object and skeletal information after the object is overlapped. Error correction device according to object overlap in spatial augmented reality.
According to claim 4,
The degree of overlap is estimated based on the ratio of the skeleton information extracted during overlap to the entire skeleton information of the object. Error correction device according to object overlap in spatial augmented reality.
According to claim 4,
The correction unit determines the skeletal information of the object to be corrected based on the estimated degree of overlap and corrects an error in the determined skeletal information, and the error in the skeletal information is the position value of the object due to the overlap. An error correction device based on object overlap in error-in space augmented reality.
In a method for correcting errors due to object overlap in spatial augmented reality,
Obtaining an image containing an object;
Identifying the object in the acquired image;
Assigning a generated ID to the identified object;
extracting skeletal information about the identified object;
Tracking the movement of the object in real time based on the extracted skeleton information;
estimating a degree of overlap between a plurality of objects included in the acquired image;
Correcting an error in skeletal information about an overlapping portion of the object based on the estimated degree of overlap; and
Comprising: restoring the object using the corrected skeleton information,
A method for correcting errors due to object overlap in spatial augmented reality, wherein the assigned ID is maintained for the restored object.
A computer-readable recording medium on which a program for implementing the method of claim 7 is recorded.