KR102105028B1 - Out-focus apparatus and method using depth estimation - Google Patents

Abstract

Proposed are an out-focus device and a method using depth prediction to realize out-focus by predicting the depth of each object and its surroundings in a screen photographed by a monocular camera. The proposed device comprises: an outline setting unit for setting an outline for a received screen; a marker arrangement unit for arranging a plurality of markers inside the outline; an area dividing unit for dividing an area based on each of the markers inside the outline; a depth map generation unit which generates a depth map in which the area for each of the markers divided by the area dividing unit is placed at its own position inside the outline; and a depth level prediction unit which analyzes the depth map and predicts a depth level for overlapping areas of each marker area.

Description

Out-focus apparatus and method using depth estimation

The present invention relates to an out-of-focus device and method using depth prediction, and more particularly, to an out-of-focus device and method capable of realizing an out-of-focus effect using depth prediction on a screen shot by a monocular camera.

While the imaging device for image capture was limited to cameras or camcorders in the past, it has recently been applied to various devices ranging from mobile phones to laptops.

In addition, the functions of the mounted photographing device are further improved and diversified, so that it is possible to photograph an image close to the actual image, and various effects may be given by image processing the photographed image. For example, it is possible to implement not only various filter effects, but also out-focus effects through processing of the captured image.

Here, the out-focus effect is to make the object to be focused out of the image clear, and to process the rest of the objects or backgrounds to make the region of interest more visible. In other words, the out-focus effect refers to blurring a background image out of a depth of field. This is more apparent when the focal length of the lens is increased, but in general, the focal length of the lens of a camera mounted on a portable communication terminal such as a mobile phone, a smart phone, and a tablet PC is short, making it difficult to optically produce the above-described out-focus effect.

Prior art 1: Republic of Korea Registered Patent No. 10-0657522 (out focusing shooting device and method using a portable terminal) Prior art 2: Republic of Korea Patent Publication No. 10-2015-0002167 (stereo camera out-focusing device and method) Prior Art 3: Korean Patent Registration No. 10-1294735 (image processing method and imaging device using the same) Prior art 4: Republic of Korea Patent No. 10-1214536 (Out-focus execution method using depth information and camera applying the same)

The present invention has been proposed to solve the above-described conventional problems, and provides an out-of-focus apparatus and method using depth prediction to implement depth-of-focus by predicting the depth of each object and its surroundings in a screen shot by a monocular camera Has its purpose.

In order to achieve the above object, the out-focus device using depth prediction according to a preferred embodiment of the present invention, an outline setting unit for setting an outline for the received screen; A marker arrangement unit for arranging a plurality of markers inside the outline; A region dividing unit for dividing regions based on respective markers inside the outline; A depth map generator for generating a depth map in which the marker-divided regions divided by the region dividing unit are placed at their positions inside the outline; And a depth level prediction unit that predicts a depth level of a region overlapping each other in each region of the marker in the depth map.

The depth level predicting unit may predict a depth level of the overlapping region according to the number of overlapping regions with respect to overlapping regions in each region of each marker in the depth map.

The greater the number of overlapping, the lower the depth level of the corresponding area, and the smaller the number of overlapping, the higher the depth level of the corresponding area.

delete

The outline setting unit may set the outline as wide as possible so as to include many objects existing in the screen.

The received screen may be a screen shot by a monocular camera.

A blurring unit performing out-of-focus by performing a blur weighting process on the overlapped region according to the depth level for each overlapping region from the depth level prediction unit may further include a.

The blurring part, the deeper the depth level of each overlapped area, the more clearly the corresponding area can be processed.

On the other hand, an out-focus method using depth prediction according to a preferred embodiment of the present invention, an out-focus method in an out-focus device, comprising: setting an outline for a received screen; Arranging a plurality of markers inside the outline; Dividing a region based on each marker inside the outline; Generating a depth map in which the region for each of the divided regions of the marker is placed at its own position within the outline; And predicting a depth level of the overlapped regions of the marker-specific regions in the depth map.

According to the present invention having such a configuration, by setting an outline on a real-time shooting screen, arranging a plurality of markers within the outline, dividing regions for each marker, and then generating a depth map based on the marker-specific regions and analyzing the depth map, to the depth Defocus effect (blurring) can be provided to complete out-of-focus.

1 is a configuration diagram of an out-of-focus device using depth prediction according to an embodiment of the present invention.
2 is a flowchart illustrating an out-of-focus method using depth prediction according to an embodiment of the present invention.
3 to 20 are screen examples used in the description of the out-focus method using depth prediction according to an embodiment of the present invention.

The present invention can be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, components, parts or combinations thereof described herein, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms, such as those defined in a commonly used dictionary, should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the overall understanding in describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

1 is a configuration diagram of an out-of-focus device using depth prediction according to an embodiment of the present invention.

The out-of-focus device using depth prediction according to an embodiment of the present invention may be mounted in a portable terminal, which is a device including a camera capable of shooting.

Alternatively, the out-of-focus device using depth prediction according to an embodiment of the present invention may be mounted on a stand alone type computer, wherein the stand alone type computer is connected to a portable terminal including a recordable camera or the like. do.

Here, the portable terminal is a wireless communication device that is guaranteed for portability and mobility, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital) All kinds of handhelds such as Assistant), International Mobile Telecommunication (IMT) -2000, Code Division Multiple Access (CDMA) -2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminal, etc. ) Based wireless communication device.

Also, the portable terminal may be composed of various smart devices such as a smart phone, a smart note, a tablet PC, and a wearable computer.

The out-of-focus device using depth prediction according to an embodiment of the present invention includes a screen receiving unit 10, an outline setting unit 12, a marker arrangement unit 14, an area division unit 16, and a depth map generation unit 18, A depth level prediction unit 20 and a blurring unit 22 are included.

The screen receiving unit 10 receives a screen shot by a camera such as a portable terminal. Here, the camera is a monocular camera. You don't need to use a binocular camera.

The outline setting unit 12 sets an outline for the screen received by the screen receiving unit 10. At this time, the outline setting unit 12 may set the outline as wide as possible to include as many objects as possible in the corresponding screen.

The marker arrangement unit 14 arranges a plurality of markers within the outline set by the outline setting unit 12. At this time, the position of each marker to be arranged is fixed in advance. For example, the shape of the marker may be various, such as square, rectangular, circular, and egg-shaped. The shape and size of these markers can be varied as needed.

In order to predict the depth, it is first necessary to select a region to be predicted in the screen, and the marker arrangement unit 14 arranges the markers to select the region to be predicted.

The marker arrangement unit 14 closely arranges a plurality of markers inside the outline to enable the depth level prediction unit 20 to predict the depths of all objects existing inside the outline and the depths around the object.

The region division unit 16 performs regional segmentation based on each marker inside the outline.

delete

When the region is divided by the region dividing unit 16, an object in contact with each marker (that is, an object (subject) in an outline) will be included in the region for each marker.

Here, even if a more detailed description of the region division in the region division unit 16 is not performed, it will be understood that those skilled in the art can fully understand through well-known techniques.

The depth map generating unit 18 generates a depth map in which the region for each marker divided by the region dividing unit 16 is disposed (ie, projected) at its own position inside the outline. Accordingly, a part or all of the regions for each marker may overlap each other on the depth map generated by the depth map generator 18.

The depth level prediction unit 20 analyzes the depth map to predict the depth level of the overlapped regions of each marker region. That is, the depth level prediction unit 20 may predict the depth level of the overlapping region according to the number of overlapping regions of each region of each marker in the depth map.

For example, the depth level prediction unit 20 has a depth level of 0 (up to 9 overlapping) to 9 (no overlapping part), and corresponding overlapping parts according to the overlapping number of areas overlapped with each other in each marker-specific area. Predict depth level. The higher the number of overlapping, the lower the depth level of the corresponding part, and the smaller the number of overlapping, the higher the depth level of the corresponding part. In this case, it is assumed that the case where the depth level is 0 has the shallowest depth, and the case where the depth level is 9 has the deepest depth.

The depth level prediction unit 20 recognizes the most overlapped area as a desired subject, predicts the depth level of the corresponding area as the lowest (ie, depth level 0), and the highest depth level of the least overlapping area ( For example, it can be predicted by depth level 9).

The blurring unit 22 performs out-of-focus by performing blur weighting processing on the overlapping region according to the depth level of each overlapping region from the depth level prediction unit 20.

For example, in the case of performing blur weighting according to the depth level of each overlapped region, the blurring portion 22 is 0 (zero) for a region having a depth level of 0 from the depth level prediction unit 20 It can be set to blur × 0 (zero) by providing the weight of. For a region having a depth level of 1, the blurring unit 22 may provide a weight of 1 to be blur × 1. For a region having a depth level of 2, the blurring unit 22 may provide a weight of 2 to be blur × 2. For a region having a depth level of 3, the blurring unit 22 may provide a weight of 3 to be blur × 3. For a region having a depth level of 4, the blurring unit 22 may provide a weight of 4 to be blur × 4. For a region having a depth level of 5, the blurring unit 22 may provide a weight of 5 to be blur × 5. For a region having a depth level of 6, the blurring unit 22 may provide a weight of 6 to be blur × 6. For a region having a depth level of 7, the blurring unit 22 may provide a weight of 7 to blur × 7. The blurring unit 22 may provide a weight of 8 for a region having a depth level of 8 to be blur × 8. For a region having a depth level of 9, the blurring unit 22 may provide a weight of 9 to be blur × 9.

The blurring unit 22 does not perform blur processing on a region having a depth level of 0. Accordingly, a region having a depth level of 0 will be most vivid.

On the other hand, the blurring unit 22 performs the most blur processing for the region having a depth level of 9. Accordingly, the area with the depth level of 9 will be the most unclear.

In other words, the blurring part 22 processes the corresponding area more clearly as the depth level is lower. That is, the higher the depth level, the more blurring the blurring part 22 is.

2 is a flowchart illustrating an out-of-focus method using depth prediction according to an embodiment of the present invention, and FIGS. 3 to 20 are used for the description of an out-of-focus method using depth prediction according to an embodiment of the present invention This is an example screen.

First, the screen receiving unit 10 receives a screen shot by a camera (ie, a monocular camera) such as a portable terminal in real time (S10).

Subsequently, the outline setting unit 12 sets an outline for a screen received in real time to the screen receiving unit 10 (S12). At this time, the outline setting unit 12 sets the outline 30 as wide as possible so as to include as many objects as possible in the corresponding screen as illustrated in FIG. 3.

Then, the marker arrangement unit 14 arranges one or more markers M within the outline 30 set by the outline setting unit 12 (S14). That is, the marker array unit 14 may include a plurality of markers M1 to M9 as illustrated in FIG. 4 in order to predict all objects existing in the outline 30 and the depths around the object. 30). In FIG. 4, the number of markers is 9, but if necessary, the number and number of markers may be increased or decreased by changing the shape and size of the marker.

Thereafter, the region division unit 16 performs regional segmentation based on each marker existing inside the outline 30 (S16). In the case of region division, an object in contact with each marker (ie, an object (subject) in an outline) will be included in the region for each marker. Of course, if there is no object (subject) in contact with each marker, the object (subject) will not be included in the area for each marker. In FIG. 5, a unique region occupied by the marker M1 and a peripheral region having a color and contrast similar to that of the marker M1 are divided into one region R1. In FIG. 6, a unique region occupied by the marker M2 and a peripheral region having a color and contrast similar to the corresponding marker M2 are divided into one region R2. In FIG. 7, a unique region occupied by the marker M3 and a peripheral region having a color and contrast similar to that of the marker M3 are divided into one region R3. In FIG. 8, a unique region occupied by the marker M4 and a peripheral region having a color and contrast similar to that of the marker M4 are divided into one region R4. In FIG. 9, a unique region occupied by the marker M5 and a peripheral region having a color and contrast similar to the corresponding marker M5 are divided into one region R5. In FIG. 10, a unique region occupied by the marker M6 and a peripheral region having a color and contrast similar to that of the marker M6 are divided into one region R6. In FIG. 11, a unique region occupied by the marker M7 and a peripheral region having a color, contrast, and the like similar to the corresponding marker M7 are divided into one region R7. In FIG. 12, a unique region occupied by the marker M8 and a peripheral region having a color and contrast similar to that of the marker M8 are divided into one region R8. In FIG. 13, a unique region occupied by the marker M9 and a peripheral region having a color and contrast similar to that of the marker M9 are divided into one region R9.

Then, the depth map generating unit 18 places the marker-specific regions R1 to R9 divided in the region dividing unit 16 at their positions inside the outline 30 (ie, projected), and displays the depth map. It is created (S18). In this case, the generated depth map may have the form shown in FIG. 14. Here, some or all of the regions for each marker may overlap each other on the generated depth map.

Subsequently, the depth level prediction unit 20 analyzes the depth map to predict the depth level of the overlapped regions of each marker-specific region (S20). That is, as illustrated in FIGS. 15 to 20, the depth level prediction unit 20 predicts the depth level of the overlapped region according to the overlapping number of regions overlapping each other in each region for each marker in the depth map. The depth level prediction unit 20 may recognize the most overlapped area as a desired subject and predict the depth level of the corresponding area as the lowest. 15, since the region where the marker M5 is present overlaps the most, the depth level prediction unit 20 predicts the depth level of the corresponding region as 0 (zero). Referring to FIG. 16, since the periphery of the region where the marker M5 exists is slightly less than the region where the depth level is predicted as 0, the depth level prediction unit 20 predicts the depth level of the corresponding region as 1. Referring to FIG. 17, since the periphery of the region having the depth level 1 overlaps a little less than the region predicted with the depth level 1, the depth level prediction unit 20 predicts the depth level of the corresponding region to 2. Referring to FIG. 18, since the periphery of the region where the depth level is 2 overlaps a little less than the region where the depth level is predicted as 2, the depth level prediction unit 20 predicts the depth level of the corresponding region as 3. Referring to FIG. 19, since the periphery of a region having a depth level of 3 overlaps a little less than a region predicted by a depth level of 3, the depth level prediction unit 20 predicts the depth level of the corresponding region to 4. Referring to FIG. 20, since the periphery of a region having a depth level of 4 overlaps a little less than a region predicted as a depth level of 4, the depth level predicting unit 20 predicts a depth level of the corresponding region as 5.

Finally, the blurring unit 22 performs out-of-focus by performing a blur weighting process on the overlapped region according to the depth level of each overlapping region from the depth level prediction unit 20 (S22). For example, the blurring part 22 is weighted with blur × 0 (zero) for a part with a depth level of 0, and the blurring part 22 is weighted with blur × 1 for a part with a depth level of 1. , For areas with a depth level of 2, the blurring section 22 is weighted with blur × 2, and for areas with a depth level of 3, the blurring section 22 is weighted with blur × 3, and the depth level is 4 The blurring part 22 can be weighted with blur x 4 for the part, and the blurring part 22 can be weighted with blur x 5 for the part with a depth level of 5.

Accordingly, when the blur weighting process according to the depth level of each part in the blurring unit 22 is completed, out-focusing on the corresponding screen is finally completed. That is, in FIG. 20, an area indicated by a depth of 0 is the sharpest, and an area indicated by a depth of 5 is the most blurred. In other words, the area indicated by the depth of 0 is the clearest, and then gradually becomes more blurred as the area indicated by the depth of 1-> depth of 2-> depth of 3-> depth of 5-> depth of 5.

In addition, the out-of-focus method using depth prediction of the present invention described above can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium can be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention pertains.

As described above, optimal embodiments have been disclosed in the drawings and specifications. Although specific terms are used herein, they are used only for the purpose of describing the present invention and are not used to limit the scope of the present invention as defined in the claims or the claims. Therefore, those skilled in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

10: screen receiving unit 12: outline setting unit
14: marker array 16: area division
18: depth map generation unit 20: depth level prediction unit
22: blurring part

Claims (16)

An outline setting unit that sets an outline for the received screen;
A marker arrangement unit for arranging a plurality of markers inside the outline;
A region dividing unit for dividing regions based on respective markers inside the outline;
A depth map generator for generating a depth map in which the marker-divided regions divided by the region dividing unit are placed at their positions inside the outline; And
And a depth level predicting unit predicting a depth level of a region overlapped with each other of the marker-specific regions in the depth map. The method according to claim 1,
The depth level prediction unit,
Out-of-focus device using depth prediction, characterized in that to predict the depth level of the overlapping region according to the number of overlapping with respect to the region overlapping each other in each region for each marker in the depth map. The method according to claim 2,
An out-focus apparatus using depth prediction, characterized in that, as the number of overlapping numbers increases, a depth level of the corresponding region is low, and as the number of overlapping numbers decreases, a depth level of the corresponding region is high. delete The method according to claim 1,
The outline setting unit,
An out-of-focus device using depth prediction, characterized in that an outline is set as wide as possible so as to include many objects existing in the screen. The method according to claim 1,
The received screen is an out-of-focus device using depth prediction, characterized in that the screen shot by a monocular camera. The method according to claim 1,
Out of focus using depth prediction, characterized in that it further comprises; a blurring unit performing out-of-focus by performing a blur weighting process for a corresponding overlapped region according to a depth level for each overlapping region from the depth level prediction unit; Device. The method according to claim 7,
The blurring portion,
The deeper the depth level of each overlapping region, the more out-of-focus device using depth prediction, characterized in that processing the region more clearly. As an out-focus method in an out-focus device,
Setting an outline for the received screen;
Arranging a plurality of markers inside the outline;
Dividing a region based on each marker inside the outline;
Generating a depth map in which the region for each of the divided regions of the marker is placed at its own position within the outline; And
And estimating a depth level of the overlapped regions of the marker-specific regions in the depth map. The method according to claim 9,
The step of predicting the depth level,
Out of focus method using depth prediction, characterized in that to predict the depth level of the overlapping region according to the number of overlapping with respect to the overlapping region in each region of each marker in the depth map. The method according to claim 10,
An out-of-focus method using depth prediction, characterized in that as the number of overlapping numbers increases, the depth level of the corresponding region is low, and as the number of overlapping numbers decreases, the depth level of the corresponding region is high. delete The method according to claim 9,
The step of setting the outline,
Out-of-focus method using depth prediction, characterized in that the outline is set as wide as possible so as to include a large number of objects existing in the screen. The method according to claim 9,
The received screen is an out-of-focus method using depth prediction, characterized in that the screen shot with a monocular camera. The method according to claim 9,
After the step of predicting the depth level,
Performing out-of-focus by performing a blur weighting process on the overlapped region according to the depth level for each overlapped region in the step of predicting the depth level; Focus method. The method according to claim 15,
Step of performing the out-focus,
The deeper the depth level of each overlapping region, the more out-of-focus method using depth prediction, characterized in that processing the region more clearly.

Images