KR20180001778A - Method and apparatus for object extraction - Google Patents

Abstract

Disclosed are an object extraction method and an apparatus thereof. The object extraction method comprises the steps of: acquiring an image from a camera mounted on a wearable device; extracting a pointer from the image by using pointer feature information; and extracting at least one object from the image by using the extracted pointer. Therefore, an object can be extracted by recognizing a pointing position of a user in a wearable device.

Description

TECHNICAL FIELD The present invention relates to a method and apparatus for object extraction,

The present invention relates to a method and apparatus for extracting an object by recognizing a pointing position of a user in a wearable device.

Recently augmented reality and virtual reality technology have been actively developed and various wearable devices such as a goggle type smart device and a clock type smart device have appeared. Augmented reality technology is augmenting virtual objects in the real world. Eyeglass wearable devices for using applications of the system are attracting attention, and research and development related thereto are actively underway.

Object segmentation is actively being developed to divide an object desired by a user in an image. In a smartphone or a tablet device environment, a technique of separating an object and a background using information input through the finger touch input of a user is generally applied.

On the other hand, gesture recognition technology is actively studied. An interface (natural user interface) that does not require a separate input device by recognizing a user's hand gesture or body motion as an algorithm is utilized in various application technologies.

The object of the present invention is to provide an object extraction method and apparatus for extracting an object by recognizing a pointing position of a user in a wearable device.

According to an aspect of the present invention, there is provided a method of recognizing a pointing position of a user and extracting an object in a wearable device.

According to an embodiment of the present invention, there is provided a method for a wearable device, comprising: (a) acquiring an image from a camera mounted on a wearable device; (b) extracting a pointer from the image using pointer feature information; And (c) extracting at least one object from the image using the extracted pointer.

In the step (c), the characteristic information of the extracted pointer is further considered, and the characteristic information is at least one of a pattern of the pointer, a pointing direction, and a motion vector.

In the step (c), the object may be divided into a foreground object or a background object based on the characteristic information.

The method may further include, after the step (b), calculating a pointing position using the extracted pointer.

The step (c) may extract the object using a super pixel division based on a pixel value of a candidate object included in the pointing position.

The method of claim 1, further comprising: setting a mode prior to the step (a), wherein the mode is one of a transparent mode and a translucent mode, and when the mode is a transparent mode, have.

Wherein the step (c) includes: increasing a probability of extracting a candidate object included in the pointing position when the pointing position of the previous image and the pointing position of the current image are within a threshold range, You can extract it as an object.

In the step (c), when the color of the object included in the pointing position of the previous image and the pointing position of the current image is within the threshold value, the candidate object included in the pointing position may be extracted as the object.

In the step (b), the pointer feature information is at least one of a color and a pattern.

And outputting the extracted object to the augmented reality after the step (c).

According to another aspect of the present invention, there is provided an apparatus for extracting an object to be extracted from a scene of a real world to be looked at by a wearer wearing a wearable device.

According to an embodiment of the present invention, there is provided a wearable device including: a camera for photographing a real-world image taken by a user from a camera mounted on a wearable device; And a processor for extracting a pointer from the image using the pointer feature information and extracting at least one object from the image using the extracted pointer.

And a display unit for reflecting the extracted object to the augmented reality and outputting the reflected augmented reality.

The processor may extract the object by further considering characteristic information of the pointer which is at least one of the pattern of the pointer, the pointing direction, and the motion vector.

The processor may calculate the pointing position indicated by the extracted pointer, and extract the object through superpixel division based on the pixel information of the candidate object included in the pointing position.

The processor may increase the probability of extracting the candidate object when the pointing position and the pointing position of the previous image are within a threshold, and may extract the candidate object as an object when the probability is equal to or greater than a reference value.

Further comprising an input unit configured to set a mode, wherein the mode is a transparent mode or a translucent mode.

When the mode is set to the transparent mode, the processor may move the pointing position based on a user's point of view.

The object extraction method and apparatus according to an embodiment of the present invention can recognize the pointing position of the user in the wearable device and extract the object.

In addition, the present invention can easily extract only an object desired by the user by pointing it with a pointer (e.g., a hand) and apply the extracted object to various augmented reality applications.

1 is a flowchart illustrating an object extraction method according to an embodiment of the present invention;
2 illustrates an image including a pointer according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a pointer in a binary image according to an exemplary embodiment of the present invention. FIG.
4 is a diagram illustrating a result of finding a pointer in an image according to an embodiment of the present invention.
FIG. 5 illustrates a foreground candidate object and a background candidate object according to an embodiment of the present invention. FIG.
6 is a diagram illustrating a result of super-pixel division according to an embodiment of the present invention.
FIG. 7 illustrates a result of assigning a foreground object or a background object to a super-pixel that overlaps a foreground candidate object and a background candidate object according to an embodiment of the present invention. FIG.
8 is a diagram illustrating an object division result according to an exemplary embodiment of the present invention.
9 is a diagram illustrating a result of reflecting an extracted object in an augmented reality according to an embodiment of the present invention.
10 is a view schematically showing a configuration of an object extraction apparatus according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In addition, numerals (e.g., first, second, etc.) used in the description of the present invention are merely an identifier for distinguishing one component from another.

Also, throughout the specification, when an element is referred to as being "connected" or "connected" with another element, the element may be directly connected or directly connected to the other element, It should be understood that, unless an opposite description is present, it may be connected or connected via another element in the middle.

Also, throughout the specification, when an element is referred to as "including" an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise. Also, the terms "a "," module ", and the like in the description mean a unit for processing at least one function or operation, which means that it can be implemented by one or more hardware or software or a combination of hardware and software .

The present invention relates to extracting an object by recognizing a pointing position (pointing position) of a user in a wearable device. Therefore, it should be understood that the object extracting apparatus described below is included in the wearable device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating an object extraction method according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an image including a pointer according to an embodiment of the present invention. FIG. FIG. 4 is a diagram illustrating a result of finding a pointer in an image according to an embodiment of the present invention. FIG. 5 is a diagram illustrating a result of searching for a foreground candidate according to an embodiment of the present invention FIG. 6 is a diagram illustrating a super-pixel division result according to an exemplary embodiment of the present invention. FIG. 7 is a diagram illustrating an example of a foreground candidate object and a background candidate object according to an exemplary embodiment of the present invention. FIG. 8 is a diagram illustrating a result of object division according to an embodiment of the present invention, and FIG. 9 is a diagram illustrating a result of assigning overlapping super pixels to a foreground object or a background object according to an embodiment of the present invention. Fig. 8 is a diagram showing a result obtained by reflecting the extracted object to the augmented reality.

In step 110, the object extraction apparatus 100 is set to a mode.

Here, the mode may be any one of a first mode (e.g., a transparent mode) and a second mode (e.g., a semi-transparent mode).

In the first mode (transparent mode), when the user wears a wearable device (e.g., glasses) and the user points to a specific object with a pointer, the pointer is moved from the image of the real world including the pointer photographed through the glasses .

In contrast, in the second mode (semi-transparent mode), after shooting a scene through a camera provided in a wearable device, the captured image is augmented on the display of the wearable device, and then the user indicates a desired position .

In the first mode (transparent mode), a camera provided in the wearable device includes an operation of pointing a specific object existing in the real world as a pointer, so that the wearable device does not interfere with the view of the user, There is an advantage.

However, the first mode (transparent mode) has a disadvantage in that the accuracy of pointer extraction is lowered because a pointer must be extracted from an image captured in a real world including a large amount of information.

On the other hand, the second mode (semi-transparent mode) has an advantage of high accuracy of object extraction, but disadvantageously hinders the view of the user by displaying the enhanced image and then pointing to the object with the pointer.

Accordingly, the user can preset the mode for object extraction before extracting the object from the image obtained through the camera provided in the wearable device.

Of course, even if the user selects the mode for object extraction in the first mode (transparent mode), it is natural that the object can be automatically operated in the second mode when object extraction is not available through the image analysis .

In the following description, the image should be understood to be a photograph of a real world through a camera mounted on a wearable device worn by the user, without any further description.

The basic operations of the first mode and the second mode are the same. However, in the second mode, the user can point the object while viewing the enhanced image after displaying the enhanced image of the camera. In the second mode, the user can point to the object while checking the position of the pointer indicated by the enhanced mode.

Therefore, the second mode can extract the object more accurately than the first mode.

In step 115, the object extraction apparatus 100 acquires a real-world image from a camera provided in the wearable device.

In operation 120, the object extraction apparatus 100 extracts a pointer from the acquired image.

In the embodiment of the present invention, it is assumed that the pointer is a hand and will be described below. However, it is obvious that pointers can be various objects other than hands.

In order to recognize and extract a pointer, the object extracting apparatus 100 sets an area (hereinafter, referred to as an initial setting area) where a pointer is to be placed in the image, places a pointer in the area, Feature information (e.g., color or pattern) can be obtained.

Thereafter, the object extracting apparatus 100 can recognize and extract the pointer from the image obtained through the camera provided in the wearable device, using the feature information of the pointer.

In more detail, the object extracting apparatus 100 may extract a pointer region by dividing pixels corresponding to the pointer by comparing the similarity between pixel values of the image obtained through the camera and the feature information of the pointer.

FIG. 2 illustrates an image including a hand when a user uses a hand as a pointer. FIG. 3 shows an example of extracting a hand by distinguishing pixels corresponding to the hand based on the feature information of the pointer in the image shown in FIG.

FIG. 3 shows pixels having a probability of 50% or more included in the hand region as pixels corresponding to the hand region. That is, as shown in FIG. 2, each pixel value of an image including a hand is compared with feature information of a pointer, and pixels having a similarity of 50% or more are extracted as a hand region (i.e., a pointer region).

In this specification, it is assumed that the degree of similarity is more than 50% in order to facilitate understanding and explanation. However, the reference value for extracting the similarity between the pixel values of the image and the feature information of the pointer and extracting it into the pointer region is experimentally It can be determined.

In comparing the pixel values of the image with the feature information of the pointer, it is a matter of course that the average value of the feature information of the pointer can be used.

For example, if the feature information of the pointer is a pattern, the degree of similarity may be derived by pattern-matching the image based on the feature information of the pointer. Pattern matching, and the like are obvious to those skilled in the art, so a detailed description thereof will be omitted.

In addition, it is a matter of course that various known algorithms can be applied to the method of recognizing and extracting the hand region from the image.

In step 125, the object extracting apparatus 100 calculates the pointing position of the extracted pointer.

For example, the object extraction apparatus 100 may calculate the outermost coordinates of the extracted pointer as a pointing position. At this time, the outermost coordinate of the extracted pointer may be the end point of the X-axis or Y-axis coordinate including the extracted pointer.

For convenience of calculating the pointing position, the object extracting apparatus 100 can extract the contour of the extracted pointer and calculate the pointing position using the extracted contour line.

4 is a diagram illustrating a result of finding a finger pointed by the user when the user uses the finger as a pointer. Referring to FIG. 4, the end point of the finger pointed by the user in the Y-axis direction may be calculated as a pointing position.

In operation 130, the object extraction apparatus 100 determines whether the mode is the first mode (transparent mode).

If the mode is the first mode, in step 135, the object extraction apparatus 100 moves the pointing position based on the user's point of view.

As described above, since the pointing position is calculated in the real world image including the operation in which the user points the pointer to a specific target in the real world, the transparent mode may cause an error between the camera viewpoint and the user's viewpoint.

Accordingly, the object extracting apparatus 100 can move the calculated pointing position when the mode is the transparent mode, reflecting the user's point of view. Of course, the object extracting apparatus 100 may output the calculated pointing position and then receive the correction position from the user and move the pointing position to the correction position.

However, if the mode is the second mode,

When the mode is a translucent mode, an error is not generated between the camera viewpoint and the user viewpoint by locating the pointer on the displayed real world image. Therefore, in the case of the semi-transparent mode, the process of moving the pointing position may be omitted.

In operation 140, the object extraction apparatus 100 extracts an object using the calculated pointing position.

That is, the object extraction apparatus 100 can extract a candidate object included in the pointing position as an object. Here, the candidate object represents an object different from the pointer. That is, it is natural that the candidate object may be different from the pointer in the feature information (e.g., color, pattern, etc.).

In extracting an object, the object extraction apparatus 100 may extract an object through superpixel segmentation based on the attribute (e.g., color) of the candidate object included in the pointing position.

An example in which super-pixel division is applied is illustrated in Fig.

FIG. 7 illustrates the result of assigning a superpixel overlapping a foreground candidate object and a background candidate object to a foreground object or a background object.

That is, after applying super-pixel division, a foreground object or a background object can be extracted by assigning a foreground object or a background object to a super-pixel which overlaps with a foreground candidate object or a background candidate object.

In addition, a known graph-based algorithm such as Graph Cut may be further applied to divide the object precisely. In the case of the graph cut algorithm, a pixel assigned as a foreground candidate or a background candidate is divided into a foreground or background as it is when an image is divided. Since the graph cut algorithm is obvious to those skilled in the art, a detailed description thereof will be omitted.

In addition, the object extracting apparatus 100 may classify the candidate objects at the pointing positions into different objects according to the characteristics of the pointer.

For example, the object extraction apparatus 100 may extract a candidate object pointed to according to the characteristics of the pointer as a foreground object or a background object. Here, the characteristics of the pointer may be the pointer direction, the shape of the pointer, the motion vector according to the motion of the pointer, and the like.

In the embodiment of the present invention, it is assumed that the characteristic of the pointer is a shape, and the description thereof will be mainly described.

For example, in the case where the pointer is a hand and the index finger indicates a pointing candidate object, the pointing candidate object can be extracted by dividing it into foreground objects. On the other hand, the object extraction apparatus 100 can extract the pointing candidate object as a background object when the pointing candidate object is pointed with the palm side (i.e., when the pointer characteristic is the shape of a palm).

For example, if the object extraction apparatus 100 has the same position within a threshold range in a continuous image (i.e., a frame), the probability that the region included in the pointing position is a region of the foreground object is high And extract it as a foreground object.

In addition, when the pointing position is located in the same color area within the threshold range in the continuous image (i.e., frame), the object extracting apparatus 100 determines the area included in the pointing position as the foreground object, It may be extracted.

On the other hand, when the attribute (e.g., color) of an area in which a pointing position is suddenly changed or a pointing position is rapidly changed in a continuous image (i.e., a continuous frame) It is determined that an error has occurred in the position estimation and it can be determined that the probability of the foreground object is low.

In this way, the object extracting apparatus 100 can extract the foreground object by determining the probability that the area including the pointing position of the pointer is included in the foreground area.

That is, when the pointing position of the pointer of the previous image and the pointing position of the pointer of the current image are equal to each other within a threshold value, the object extracting apparatus 100 increases the probability of extracting a candidate object included in the pointing position, The candidate object can be extracted as a foreground object or a background object.

For example, the object extracting apparatus 100 may increase the extraction probability of the candidate object when the attribute of the candidate object of the current image pointing position and the attribute of the candidate object (e.g., color, etc.) is within the threshold value The candidate object may be extracted as a foreground object or a background object when the probability is higher than a reference value.

In this way, the object extracting apparatus 100 according to an embodiment of the present invention reflects the probability of the candidate object without reflecting the result of dividing the object based on super-pixel division, graph cut, etc., There is an advantage that it can be extracted.

This is applied to the background object in the same manner, so that a duplicate description will be omitted.

FIG. 5 shows an example of extracting a foreground object and a background object from a pointing candidate object by reflecting characteristics of a pointer. The red portion of FIG. 5 represents the area extracted by the foreground object using the pointing position, and the blue portion represents the background object.

FIG. 5 illustrates an example of displaying a foreground object and a background object in a stroke form. FIG. 5 shows a foreground object and a background object in the form of a stroke in order to facilitate understanding and explanation. However, it is natural that a foreground object and a background object may be extracted as a bounding box in addition to the stroke form.

As described above, the object extracting apparatus 100 according to an embodiment of the present invention may not extract foreground objects and background objects so as to perfectly match object shapes. That is, the object extraction apparatus 100 according to an embodiment of the present invention is intended for application to an augmented reality using an extracted object (i.e., at least one of a foreground object and a background object) The object region can be extracted so that each object can be distinguished. By doing so, it is possible to minimize complexity by minimizing the computation complexity to completely separate the shapes of objects (foreground and background objects).

In step 145, the object extraction apparatus 100 selects whether to perform post-processing.

Extracted objects can be incorrectly extracted by various causes. For example, the pointing position of the pointer of the user may be determined incorrectly, and even if the pointing position of the pointer is accurately detected, the position of the candidate object may deviate from the position intended by the user due to hand tremor.

In the case of the transparent mode, there may be an error in moving the pointing position reflecting the error between the camera viewpoint and the user viewpoint, and an error may also occur in the process of extracting the object.

Accordingly, the object extraction apparatus 100 can display whether the object is to be displayed after displaying the extracted object.

If the user selects the post-completion, the object extraction apparatus 100 receives the post-completion information at step 150. Here, the posterior information may be a correction object region. That is, the user can specify the object region to be corrected by checking the displayed object extraction result.

Such post-processing information is input in units of the super-pixel area, so that the time required for the post-processing can be minimized. Obviously, such a post-processing process can be performed a plurality of times.

In step 155, the object extraction apparatus 100 reflects the post-processing information, rearranges the object extraction result, and finally extracts the object. Fig. 8 illustrates the result of object extraction.

In step 160, the object extraction apparatus 100 reflects the extracted object to the augmented reality and displays the augmented reality.

FIG. 9 illustrates the result of reflecting the extracted object to the augmented reality. As shown in FIG. 9, the extracted object is reflected in the augmented reality, so that the user can experience the augmented reality more usefully.

FIG. 10 is a diagram schematically showing a configuration of an object extraction apparatus according to an embodiment of the present invention.

10, an object extraction apparatus 100 according to an exemplary embodiment of the present invention includes a camera 1010, a display unit 1015, a processor 1020, an input unit 1025, a candidate unit 1030, 1035).

The camera 1010 is mounted on a wearable device and is a means for photographing a real world scene to be watched by a user. Of course, the camera 1010 may also take an image of the real world and output the image that reflects the augmented reality.

The object extracting apparatus 100 may be connected to a wearable device and may be connected to a camera mounted on the wearable device. In this case, the object extracting apparatus 100 may include a camera 1010, The captured image may be acquired. In this case, the object extracting apparatus 100 may not include a camera.

The display unit 1015 is means for visually outputting photographed images and the like. For example, when the wearable device is a pair of glasses, the display portion 1015 may be a spectacle lens.

Processor 1020 may include internal components (e.g., camera 1010, display portion 1015, candidate portion 1030, memory 1035) of object extraction apparatus 100 according to an embodiment of the present invention. Etc.).

In addition, the processor 1020 may extract a pointer from the image using the pointer feature information, and extract at least one object from the image using the extracted pointer. Since this is the same as that described in FIG. 1, a duplicate description thereof will be omitted.

The input unit 1025 is a means for receiving various control commands for controlling the object extraction apparatus 100 from a user. For example, the input unit 1025 may select the mode or input the finishing information. The input unit 1025 may be implemented as a touch pad.

The candidate section 1030 is a means for rearranging the extracted object by reflecting the post-registration information inputted from the user. This is the same as that described with reference to FIG. 1, so duplicate descriptions will be omitted.

The memory 1035 stores various algorithms necessary for performing a method of extracting an object to be extracted from a scene of a real world to be looked at by a wearer wearing a wearable device according to an embodiment of the present invention, . ≪ / RTI >

The object extraction method according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording media storing data that can be decoded by a computer system. For example, it may be a ROM (Read Only Memory), a RAM (Random Access Memory), a magnetic tape, a magnetic disk, a flash memory, an optical data storage device, or the like. In addition, the computer-readable recording medium may be distributed and executed in a computer system connected to a computer network, and may be stored and executed as a code readable in a distributed manner.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the following claims And changes may be made without departing from the spirit and scope of the invention.

100: object extraction device
1010: Camera
1015:
1020: Processor
1025:
1030: Candidate Government
1035: Memory

Claims (18)

(a) acquiring an image from a camera mounted on a wearable device;
(b) extracting a pointer from the image using pointer feature information; And
(c) extracting at least one object from the image using the extracted pointer.
The method according to claim 1,
The step (c)
Considering further the property information of the extracted pointer,
Wherein the characteristic information is at least one of a pattern, an instruction direction, and a motion vector of the pointer.
3. The method of claim 2, wherein step (c)
And extracting the object as a foreground object or a background object based on the characteristic information.
The method according to claim 1, wherein, after the step (b)
And calculating a pointing position using the extracted pointer.
5. The method of claim 4,
The step (c)
And extracting the object using a super pixel division based on a pixel value of a candidate object included in the pointing position.
5. The method of claim 4,
Receiving a mode before the step (a), the mode being one of a transparent mode and a translucent mode,
And moving the pointing position based on the user's point of view when the mode is in the transparent mode.
The method of claim 1, wherein the step (c)
When the pointing position of the previous image and the pointing position of the current image are within the threshold range, the probability of extracting the candidate object included in the pointing position is increased, and the candidate object having the probability higher than the reference value is extracted as the object. Object extraction method.
The method of claim 1, wherein the step (c)
Wherein the candidate object included in the pointing position is extracted as the object when the color of the object included in the pointing position of the previous image and the pointing position of the current image is within the threshold value.
The method according to claim 1,
Wherein the pointer feature information is at least one of a color and a pattern in the step (b).
The method of claim 1, wherein, after step (c)
And outputting the extracted object to the augmented reality.
Readable recording medium having recorded thereon a program code for performing the method according to any one of claims 1 to 10.
A camera mounted on the wearable device and configured to photograph a real-world image taken by a user; And
And a processor for extracting a pointer from the image using the pointer feature information and extracting at least one object from the image using the extracted pointer.
13. The method of claim 12,
And a display unit for reflecting the extracted object to the augmented reality and outputting the reflected object.
13. The method of claim 12,
Wherein the processor extracts the object by further considering characteristic information of the pointer which is at least one of a pattern of the pointer, an instruction direction, and a motion vector.
13. The method of claim 12,
The processor comprising:
Wherein the calculating unit calculates the pointing position indicated by the extracted pointer and extracts the object through superpixel division based on pixel information of the candidate object included in the pointing position.
16. The method of claim 15,
The processor comprising:
Wherein if the pointing position and the pointing position of the previous image are within a threshold value, the probability of extracting the candidate object is increased, and if the probability is equal to or greater than the reference value, the candidate object is extracted as an object.
16. The method of claim 15,
Further comprising an input unit for setting a mode,
Wherein the mode is a transparent mode or a semi-transparent mode.
18. The method of claim 17,
Wherein when the mode is set to the transparent mode, the processor moves the pointing position based on a user's viewpoint.

Images