KR101590515B1 - Method for correcting photograghy composition - Google Patents

Abstract

A method for correcting a composition of a photograph before shooting is disclosed. An embodiment of the present invention includes: displaying an image on a camera display window; Forming a saliency map of an image displayed on the display window and confirming a subject using a region of interest (ROI); Detecting an outline of the identified subject using an Active Contour Model; The method of claim 1, wherein the step of identifying the object with the region of interest (ROI) comprises the steps of: dividing the display window into three sections, each having a horizontal and a vertical length, and positioning the outline of the object at an intersection of the grid Generating a low-level feature map using the characteristics; Calculating a ROI map from the low-level characteristic map; And extracting a region of interest, which is a core natural effect, from the calculations.

Description

[0001] METHOD FOR CORRECTING PHOTOGRAPHY COMPOSITION [0002]

One embodiment of the present invention relates to a method for correcting a composition of a photograph before photographing.

Mobile devices have become a common device that is widely used by many people, offering performance comparable to PCs with the development of semiconductor nano processes, low power and high performance processors. In recent years, individuals are using mobile devices such as mobile phones and tablet PCs. As social networking services (SNS) become commonplace, cameras mounted on mobile devices have become essential functions.

As the resolution increases, cameras and digital cameras of mobile devices provide a variety of shooting support functions and auxiliary functions such as face recognition, panorama, and the like when shooting. Many of these functions have been provided for a long time ago, but most users only know the guidelines for the triple division technique, but do not know the functions and roles.

The three-way technique uses the concept of Golden Section to assist the camera to ensure that the subject being photographed is placed in a stable position in the photograph.

The structure of the Golden Split is useful for aesthetic analysis of a given human sculpture or a given static state of nature, but there is a limit to the dynamic state analysis. Understanding the dynamic nature of nature, such as growth, development, and progress, from the perspective of the golden divide, is essential to understand the golden helix structure.

Figure 1 is an illustration of an example of golden splitting.

Referring to FIG. 1, a rectangle having a golden ratio is represented by squares A, B, C, D, E, F, G ... And so on. These squares A, B, C, D, E, F ... Theoretically, the progression to the back can proceed infinitely toward the point Q, and each of the squares is divided into golden segments. For example, square A forms a golden partition with the rest of the entire square (square EBCF), and square B forms a golden partition with square HCFL.

When a 1/4 circle (arc) existing in each square is drawn around a square Q having a golden partition, the arc of a spiral structure as shown in FIG. 1 will be displayed. The connected form of these arcs is called the Golden Spiral, and its progression can extend to infinity. If we measure the mutual ratios of each of the connected arcs of this golden helix, we can easily see that the golden ratio is embedded.

In photographs, this golden ratio can be maintained by using a ternary technique. The object placed close to the intersection through the triangulation in the photo gives a strong impression. In addition, when it is placed at an inclination with respect to the intersection, the balance is seen, and the elements placed at three points of the intersection form a bold triangle.

In order to apply this comforting golden ratio to a picture, three horizontal and vertical lines of the screen are displayed with two vertical lines and a horizontal line, and four intersections are formed. When the subject is placed at one of the intersections, A screen of division is formed.

2 is a view showing an intersection where a position of a subject is to be placed using a golden segment. FIG. 3 shows an example of an actual photograph using a technique of trisecting.

In order to provide a stable feeling of such a composition, a technology capable of providing a sense of stability by inducing a user to correct the composition automatically by analyzing the position of the subject when photographing is being developed.

A problem to be solved by the present invention is to provide a method for correcting a composition of a pre-shooting photograph, which can automatically correct a composition so as to give a feeling of a stable composition by analyzing the position of a subject when photographing.

A method of correcting a picture of a pre-shooting picture according to the present invention includes: displaying an image on a camera display window; Forming a saliency map of an image displayed on the display window and confirming a subject using a region of interest (ROI); Detecting an outline of the identified subject using an Active Contour Model; The method of claim 1, wherein the step of identifying the object with the region of interest (ROI) comprises the steps of: dividing the display window into three sections, each having a horizontal and a vertical length, and positioning the outline of the object at an intersection of the grid Generating a low-level feature map using the characteristics; Calculating a ROI map from the low-level characteristic map; And extracting a region of interest, which is a core natural effect, from the calculations.

At this time, the process of forming the saliency map may be performed using any one of color, brightness, motion, harmony, or direction of the image.

delete

In addition, the step of setting a region of interest, which is a core natural result from the calculation, may be performed by using any one of a multi-scale contrast, a center-surround histogram (CSH) CRF) of the received signal.

delete

According to an embodiment of the present invention, there is an effect that it is possible to provide a method of correcting a composition of a pre-shooting photograph, which can automatically correct a composition so as to give a feeling of a stable composition by analyzing the position of a subject when photographing.

Figure 1 is an illustration of an example of golden splitting.
2 is a view showing an intersection where a position of a subject is to be placed using a golden segment.
FIG. 3 shows an example of an actual photograph using a technique of trisecting.
4A and 4B are views showing an example of applying the Rule of Third.
5A and 5B are views showing before and after the application of Rule of Third, respectively.
FIGS. 6A and 6B are views showing the state before and after applying the cropping, respectively.
Figs. 7A and 7B show examples of photographs in which the composition is unstable.
8 shows segmentation of a color image. ((a) source image, (b) user-specified seeds, (c) segmentation results,
Figure 9 shows the evolution stages of the bacteria.
Figure 10 shows Background and Foreground in the image.
11 is a diagram illustrating an example of image segmentation using the Grab Cut algorithm.
12 shows the image segmentation and the minimum cut.
13 shows a process of clustering colors separated by a minimum cut.
FIG. 14 is a flowchart illustrating a method of correcting a composition of a photograph before photographing according to an embodiment of the present invention.
FIG. 15 illustrates a process of setting a region of interest in a method of correcting a picture of a photograph before photographing according to an embodiment of the present invention.
16 is a diagram showing an example of a salience map for each step.
17 shows Skin pixels and Cb in YCbCr color.
Figure 18 shows an image area merged with adjacent boxes.
FIG. 19 shows a result obtained by removing regions that are not related to the similarity of faces in consideration of ranking.
FIG. 20 is a representation of various ROIs, which are set based on a maximum value and a box in which an actual object is accurately positioned in ROI.
FIG. 21 shows a result of performing color segmentation based on three clusters.
FIG. 22 shows a process of clustering with three clusters.
23 shows the result of the image according to the constant k value.
Fig. 24 shows the mapping result of color segmentation and saliency map.
Figure 25 shows an example where the scenario is not applied.
FIG. 26 shows the difference between an image to which a general rule of thirds is applied and an image to which a scenario is additionally applied.
FIG. 27 shows an example of a photograph in the case where the detected ROI is apart.
28 shows a result of comparing a result image obtained by detecting a line segment on a horizontal line and applying the original image and the scenario.
Figure 29 shows an over cropped image resulting from no such scenario being applied.
30 shows the contour detection of the subject.
31 shows an example in which a contour of a subject is detected so as to induce a correction of a composition of a camera in an actual photographing by the camera, and a position of the subject corresponding to the composition is shown as a contour so as to correct the composition of the photograph.

Hereinafter, embodiments of the present invention will be described in detail. However, the present invention is not limited thereto, and the present invention is only defined by the scope of the following claims.

First, a description will be made of a composition of a photograph and a method of extracting necessary data from a photograph before explaining a composition correction method of a pre-shooting photograph according to an embodiment of the present invention.

Most of the photos taken using a digital camera or a camera mounted on a mobile device are not stable. In the process of holding a subject viewed by the human eye in a rectangular area called a photograph, the photographer should correct the composition so that the position of the subject is stable, but in general, a non-specialist who does not study the photograph professionally, It is not easy to do.

There are a few ways to compose a picture as a technique for taking pictures more like an expert. These methods include Rule of Thirds, Cropping, Viewpoint, Experimentation, Balancing Elements, Leading Lines, Symmetry and Patterns, Background, Depth, and Framing.

For example, Balancing Elements is important because the background can be monotonous and void, even if objects in the picture are placed at the recommended rate.

The Leading Lines are that people's eyes should be taken into account when taking pictures because they tend to understand the image along the central line that exists in the picture.

Symmetry and Patterns have a lot of symmetry and pattern in natural images or artifacts, which is effective in reducing human vision. Therefore, when photographs are taken, the photographs are constructed taking into consideration the properties and patterns that may exist in the images.

Background is a good way to turn off time if the object is in front of a simple background, so if you want to emphasize an object, you can simply take the background.

Depth effectively uses the depth of foreground, background, and middle foreground to emphasize the desired objects through Depth and brings the overall atmosphere in a stable manner.

Framing is a way to emphasize the object you want to emphasize.

The methods described so far are very important when composing photographs, and they are components that have great influence in the process of viewing photographs through human eyes. However, at the moment of photographing, correcting the image by adding such elements in a digital camera or a mobile device can not obtain correct result due to the calculation ability and memory limitation.

Therefore, in the present invention, we will use Rule of Thirds and Cropping, which are methods of obtaining images through photographing and processing images by post-processing to maximize esthetics. These two methods are designed to make your photos look more professional and allow objects in the photos to be placed at a comfortable viewing position to maximize aesthetics.

4A and 4B are views showing an example of applying the Rule of Third.

4A and 4B, it is easy to see how the subject is divided into three parts and where the subject is located. The photographs of this composition can enhance the concentration of the viewer and provide comfort in the overall picture composition.

5A and 5B are views showing before and after the application of Rule of Third, respectively.

5A and 5B are photographs (Fig. 5A) in which the subject is placed in the center of the photograph, respectively, and photographs (Fig. If the subject that the photographer intends to emphasize is the subject, the gaze will stay on the right photograph where the subject is positioned through composition correction.

On the other hand, Cropping to be applied together with the Rule of Third is a typical method of cutting out a region of interest in the image and composing the image into the corresponding region. It has been widely used as a processing technique for region of interest (ROI) in general.

FIGS. 6A and 6B are views showing the state before and after applying the cropping, respectively.

Referring to FIGS. 6A and 6B, cropping is applied and the ROI is centered on the image.

Figs. 7A and 7B show examples of photographs in which the composition is unstable.

Referring to FIGS. 7A and 7B, as can be seen intuitively, the two photographs show an unstable feeling due to an incorrect composition. It is an object of the present invention to reconstruct such an image in accordance with a triple segmentation technique, thereby improving the esthetics and automatically correcting the composition stably and sophisticatedly.

On the other hand, image segmentation is used in the process of image processing in applications such as medical image analysis or photo editing.

Let's look at the mutual growth cut method by segmentation technique of color image.

8 shows segmentation of a color image. (a) source image, (b) user-specified seeds, and (c) segmentation results. Figure 9 shows the evolution stages of bacteria.

Referring to FIG. 8, in the growth cut method, 'bacteria' starts to grow as a seed pixel, and spreads to the entire image region.

Then, as shown in FIG. 9, each cell repeats a plurality of processes, and performs segmentation process while attempting to 'attack' neighboring cells.

Grab Cut is an image segmentation technique based on the graph cut method. It is a technique that accurately separates objects within a given range when the user inputs initial information about the objects to be separated from the image. The user sets the rectangle window in the object area and executes the algorithm. Grab Cut is an innovative segmentation technique for segmentation of boundaries and boundaries in images.

Most segmentation techniques use the edge and region information contained in the image. And, like most segmentation techniques, Grab Cut also encapsulates information in the image.

The general segmentation technique uses only one of the edge information and the area information, but the Grab Cut method uses both of them to find the optimal segmentation result. To perform the segmentation, a graph is drawn where the pixels of the image are represented, and two additional special nodes, the sink node and the source node, are created.

Figure 10 shows Background and Foreground in the image.

Referring to FIG. 10, the background and the foreground are labeled in the image.

11 is a diagram illustrating an example of image segmentation using the Grab Cut algorithm.

Referring to FIG. 11, a user may directly designate an area to segment the background and foreground.

In order to segment the colors in the Min Cut image, among the techniques of clustering by assigning a weight (w) to distances at each node, which means each pixel of an image, the Minimum Cut method is a method of clustering We use Adjacency Matrix and Weight Matrix for the weight w using distance from the weights.

12 shows the image segmentation and the minimum cut. 13 shows a process of clustering colors separated by a minimum cut.

12, clustering is repeated according to the degree of similarity of colors, and is divided into blocks constituting a certain cluster as shown in FIG.

Hereinafter, a method of correcting a composition of a pre-shooting photograph according to an embodiment of the present invention will be described.

FIG. 14 is a flowchart illustrating a method of correcting a composition of a photograph before photographing according to an embodiment of the present invention.

A method of correcting a pre-shot image according to an embodiment of the present invention includes a step of inputting an original image (S100), forming a saliency map from the original image, and performing an image segmentation process (S300) of extracting a region of interest by merging the region of interest map and the image obtained by performing the segmenting process, and applying a triangulation method to the extracted region of interest, (step S400).

First, in the method of correcting the pre-shooting image according to an embodiment of the present invention, a process of forming a saliency map from an original image includes a top-down approach of searching for an area of interest by a user and a visual stimulation There is a bottom-up approach to extract regions of interest based on features.

In the present invention, a Saliency Map is derived from various elements that affect the perception of a region of interest such as color, luminance, motion, texture, and orientation of an image using elements most suitable for use.

To obtain the Saliency Map, follow the steps below. The first step is to extract features. That is, various low-level feature maps are constructed. The next step is to calculate the Saliency Map. Saliency Map is calculated by the feature maps. Finally, the critical area from the calculations is set as the area of interest. These approaches are basically based on biological or computational approaches.

First, the biological approach is to combine or combine the feature maps of brightness, color, and direction with the bottom up method. And, the computational approach basically uses feature maps but approaches based on normalization graph. This is a combination of partially biologic and computational methods. By constructing a multiple pyramid image for various features and finding the center surround difference, the value of the image represented by the feature values can be confirmed. Since these values are represented by various values depending on the features existing in the image, the optimal saliency map should be derived by combining multiple values rather than using only one value. In the present invention, a map reflecting each characteristic is generated by using a complex feature, and a Saliency Map which is most suitable for an automatic composition correction algorithm is derived.

FIG. 15 illustrates a process of setting a region of interest in a method of correcting a picture of a photograph before photographing according to an embodiment of the present invention.

Brightness value, which is the most influential factor in human vision, is a key criterion for constructing a saliency map. Therefore, an intensity image is created by the following equation (1), and this image is used as a feature map.

--- (1)

--- (One)

The human neural circuit that transmits color information to the brain converts the information by the three cones into the color information of the opposite pair of 'red / green' and 'yellow / blue'. In order to obtain the color feature map from the input image, four channels of R (red), G (green), B (blue), Y (yellow) . In the present invention, two color feature maps are created using the above four channels by the following equation (2).

--- (2)

--- (2)

Here, RG and BY are color information of the opposite pair of 'red / green' and 'yellow / blue'. Each Feature Map is transformed into a Conspicuity Map, which is characterized by the highlighting of the most prominent parts from the periphery. The following equation (3) shows the process of performing the Gaussian pyramid to obtain the Conspicuity Map.

--- (3)

--- (3)

When each feature is represented by F, the nine-dimensional Gaussian pyramid P is obtained by gradually sub-sampling the feature map F using a Gaussian filter G and performing low pass filtering.

For each feature P, six intermediate multi-size Conspicuity maps are obtained by the following equation (4).

--- (4)

--- (4)

(5), which has a unique feature.

--- (5)

--- (5)

The visual attention model constructs a saliency map by determining and combining weights for each Conspicuity Map.

--- (6)

--- (6)

Where n is the number of Conspicuity Maps. In the present invention, the dynamic weighting value ω is determined dynamically according to the importance of the Conspicuity Map used to construct the Saliency Map.

16 is a diagram showing an example of a salience map for each step.

Referring to FIG. 16, when the background and the object show distinct color differences, the LUV space is converted into a good LUV space, and the mean difference is derived through the center surround difference with the brightness value using the Saliency Map and the method described above A typical example of a Saliency Map is shown.

The most important area of interest in the photographic image is the human face. Therefore, the face of a person has to have a high value among the regions of interest, so the face detection is separately performed to detect the face region of the person. The following shows the process of detecting face area through Face Detection.

The technique of detecting a person's skin color in an image is a very important and useful method for detecting a face region. To extract such skin color, convert it to YCbCr color model when input image is RGB method. In the case of the RGB color model, the error probability is high in detecting the skin color as the light condition changes. The RGB color model can be converted to the YCbCr method through the following equation (7).

--- (7)

--- (7)

The scatter pattern of the skin pixels of the detected skin color and the scatter pattern of histograms of Cb and Cr are shown in FIG. 17 below.

After detecting the facial region, the result image is obtained through color segmentation, and the binary region created by removing the small regions of the obtained image is integrated with the previously obtained image using Edge Detection technology, Small areas can be removed. After removing the small region, the coordinates of the location determined by the skin color are selected using the image, and the area of the rectangular shape is designated by using the color-based segmentation technique. At this time, each selection area has its own window size.

In the case of neck, arm, and hand located under the face of the skin color detected area, the box-merge algorithm is used to combine the adjacent box areas into a large box area. These results are shown in Fig.

The image matching algorithms are applied to the detected face regions to rank the regions, and considering the distance and the window size, the region having the similarity in ranking is different from the ranking in the other regions The face area can be detected as a result.

FIG. 19 shows a result obtained by removing regions that are not related to the similarity of faces in consideration of ranking.

We use Saliency Map to find the region of interest in the image and find the maximum value for the block of a certain size. However, there is no criterion to determine the extent of the region of interest.

FIG. 20 is a representation of various ROIs, which are set based on a maximum value and a box in which an actual object is accurately positioned in ROI.

In order to set the correct ROI, it is necessary to derive the ROI from the Saliency Map.

In the present invention, an ROI is detected by combining two results of a color segmentation technique and a saliency map (S200).

More specifically, Color Segmentation is a method of dividing a region of an image by grouping similar colors into a group using density and degree of aggregation between colors.

FIG. 21 shows a result of performing color segmentation based on three clusters.

Color Segmentation is a method of segmentation using the density and degree of coupling between colors. In the present invention, after separating k colors arbitrarily by k average points, groups of k are divided into k groups close to the points, These clusters were segmented by applying a K-means K-Means Clustering Algorithm that repeatedly divides the group by setting averaging point.

The K-Means algorithm is a distance-based clustering technique that decomposes a set of n objects into k clusters. The similarity of these clusters is determined by measuring the average value of the objects that can be seen as the center of gravity of the cluster in the cluster, and the data near the reference point are grouped into one cluster.

FIG. 22 shows a process of clustering with three clusters.

Referring to FIG. 22, the process of clustering includes determining (S210) an arbitrary number of clusters, assigning initial values or cluster centers to each cluster, setting a position (S210) (S200) (using the Euclidean distance), and resetting the center of the new cluster to have a minimum value based on the data divided into the clusters; If the location of the center of the newly obtained cluster is the same as that of the existing cluster, the algorithm may be terminated.

The image by this algorithm can show different results depending on the cluster number value k. That is, it is necessary to select a suitable constant value for use in automatic composition correction, and to combine with saliency map to derive an optimal ROI.

23 shows the result of the image according to the constant k value.

Referring to FIG. 23, when the number of clusters is 2, the edge of the image can be distinguished so as to be distinguishable. Accordingly, in the present invention, k = 2 is set as an appropriate value for use in automatic composition correction Respectively.

The color segmentation result is mapped to the saliency map result to find the optimal ROI (S300).

Fig. 24 shows the mapping result of color segmentation and saliency map.

The cropping process is performed so as to comply with the Rule of Thirds criterion using the ROI obtained from the image by the method proposed by the present invention (S400).

If cropping is performed by finding only the intersection nearest to the ROI obtained at this time, a corrected result image of the unintended composition may be obtained. Therefore, in order to preserve the elements inherent in the image, it is necessary to carry out cropping for the composition correction by applying a scenario according to the characteristic of each image.

In addition, we compare the obtained ROI with the horizontal and vertical dimensions of the image, measure the non-degree of the image, and apply the Rule of Thirds to maximize the stability and aesthetics of the image. However, the image depends on various factors such as the type of the object, the size of the ROI, and the human face recognition area. Therefore, we use a method of cropping an image based on various scenarios in order to obtain a result image of a proper configuration of various images.

Figure 25 shows an example where the scenario is not applied.

FIG. 25 shows a case where the face region is correctly selected as the ROI by Face Detection, but the entire body is not included in the resultant image. Therefore, such an image should be selected and processed appropriately.

In the case of Face Detection, cropping is applied to the Rule of Thirds, but the method of improving the stability of the photo by protecting the body area is used.

FIG. 26 shows the difference between an image to which a general rule of thirds is applied and an image to which a scenario is additionally applied.

When two ROIs or faces are detected in the image, a method of comparing the distance between the two faces and the size of the image is used. If two faces are closely attached, find and apply the mean of ROI of two faces. If two faces are far away from the image, apply automatic composition correction while preserving two faces in the image.

FIG. 27 shows an example of a photograph in the case where the detected ROI is apart.

If a line segment such as a horizon is detected in the image, the line segment is placed in a line segment divided into Rule of Thirds to maximize the aesthetics and stability of the image. In this method, the correction process should be applied while preserving the image as much as possible by finding a segment mainly represented exclusively in the image.

28 shows a result of comparing a result image obtained by detecting a line segment on a horizontal line and applying the original image and the scenario.

As described above, if the human face is the most important ROI, but the human face occupies the entire area of the image, cropping the image only to the ROI results in an unnatural image.

Therefore, in such a case, the method of maintaining the image as it is is used to improve the stability of the image.

Figure 29 shows an over cropped image resulting from no such scenario being applied.

The position of the subject is displayed on the camera display window so that you can take a picture of a stable composition using the above-described automatic composition correction technique. In order to display the position of the object on the camera display window, the object should be detected in the saliency map, and then the contour of the object should be detected as shown in FIG.

31 shows an example in which a contour of a subject is detected so as to induce a correction of a composition of a camera in an actual photographing by the camera, and a position of the subject corresponding to the composition is shown as a contour so as to correct the composition of the photograph.

The active contour model (ACM), known as the Snake algorithm, is used to detect the contour of the subject.

The snake algorithm is a mathematical operation that minimizes the energy on the image surface. The snake algorithm can divide the background and the object by controlling the contour of the object to continuously follow the contour of the moving object. Therefore, ACM basically appears as a form of shrinking the area. If ACM is applied to the ROI area of the subject, the ROI area can be transformed into the shape of the subject and segmented.

The snake algorithm extracts the outer edge of the object and displays it on the screen by the above-mentioned rule of thirds, thereby inducing the user to take a picture of a better composition.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. As will be understood by those skilled in the art. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (5)

Displaying an image on a camera display window;
Forming a saliency map of an image displayed on the display window and confirming a subject based on a region of interest (ROI);
Detecting an outline of the identified subject using an Active Contour Model;
And arranging the contour of the object at an intersection of the grids,
The step of confirming the subject with the region of interest (ROI)
Generating a low-level feature map using the characteristics;
Calculating a ROI map from the low-level characteristic map; And
And extracting a region of interest, which is a core natural effect, from the calculation. The method according to claim 1,
Wherein the process of forming the saliency map is performed using any one of color, brightness, motion, harmony, and orientation of an image. delete The method according to claim 1,
The step of establishing a region of interest,
The method comprising the step of setting a Conditional Random Field (CRF) using any one of a multi-scale contrast, a Center-Surround Histogram (CSH), and a spatial distribution of colors. How to correct the composition. delete

Images