KR101031083B1 - Top-down visual attention method of determining visual attention region in a still image - Google Patents

Abstract

The present invention relates to a visual focusing method, and more particularly, to a top-down visual focusing method for determining a visual focus area of a still image, the feature map of the still image for each of the predetermined visual features determining a feature map and a conspicuity map; Determining a maximum stimulus region from the stimulus map; Determining a feature map of a target object image for each of the visual features; Evaluating the similarity between the feature map of the target object image and the feature map of the maximum stimulus region; Selecting visual features based on the evaluated similarity; And combining the stimulus maps of the selected visual features to obtain a saliency map.

In the present invention, even when the main visual features of the target object are not identified in the image, the probability of visual focusing being successfully performed is very high. The present invention has been confirmed through experiments that there is a performance improvement of about 57% compared to the conventional top-down VOCUS algorithm.

Focus, top-down, top-down

Description

TOP-DOWN VISUAL ATTENTION METHOD OF DETERMINING VISUAL ATTENTION REGION IN A STILL IMAGE}

The present invention relates to a visual focusing method, and more particularly, to a visual focusing method for determining a visual focusing area of a still image.

Therefore, the present invention can be widely used in intelligent robots, cameras, various broadcasting equipments based on image processing, etc., which are equipped with image sensors, and thus the present invention can be said to extend to various technical fields to which image processing technology is applied. .

The visual attention model is a computational model of the human visual system that performs the desired task by extracting only the necessary information from many objects that reach the visual receptor. It was first proposed by Koch and Ullman in 1985.

The model proposed by Koch and Ullman computes several features in parallel and integrates their conspicuities into a saliency map, which many researchers have since studied. It was also the basis of the proposed visual focus model. That is, the overall framework for obtaining feature maps (hereinafter referred to as "feature maps") and conspicuity maps (hereinafter referred to as "stimulus maps") from images, and determining saliency maps (hereinafter referred to as "stimulus maps") from them Does not deviate significantly from the original proposed visual focus model, but various methods have been intensively proposed to obtain a stronger, adaptive, and reliable stimulus map.

However, most of the research has been directed to the process of focusing on the stimulating visual information of the image itself, the so-called "bottom-up" visual focusing method, and several stimulating regions within the image. The study of so-called "top-down" visual focusing methods that mentally concentrates on one of them has been relatively poorly conducted.

The top-down visual focus method takes into account that the emotions, motivations, desires, and sensations of a person are important factors in determining the visual focus area, but in fact, modeling human mental activity is very complicated and complex. Only a simple top-down visual focusing method has been proposed that performs visual focusing using information about objects.

A typical example is the Visual Focus Detection Algorithm, "Visual Object Detection with a CompUtational Attention System (VOCUS)" (hereinafter referred to as the "VOCUS algorithm"), published by S.Frintrop in 2006.

The VOCUS algorithm includes both bottom-up and top-down visual focusing methods. Among the top-down visual focusing methods, the maximum stimulus region of the target object to be searched is determined and the weight is determined, and the weight is applied to the stimulus map of the image. This is a way of performing visual focus by obtaining a map.

However, when the VOCUS algorithm is actually applied to various images to perform visual focus, when the maximum stimulus region or the main visual feature of the target object is hidden by other objects in the image, an appropriate stimulus map cannot be obtained. Experimental results show that there is a high probability of failure to concentrate.

In other words, the VOCUS algorithm does not consider special conditions in the image, and obtains a stimulus map by giving a high weight to the main visual features of the target object. For certain situations that cannot be discerned in the image because of being obscured by an object, an effective and appropriate stimulus map could not be obtained, resulting in a failure to concentrate on vision.

The present invention has been made in accordance with the problems of the conventional top-down visual focusing method described above, and by determining and selecting from a given image a visual feature that can best find the target object without using the maximum stimulus region of the target object, This is to ensure that the visual focus can be successfully performed even in some hidden cases.

In addition, the present invention is to provide a new histogram comparison method instead of the histogram intersection technique that has been mainly used to evaluate the similarity of the conventional image in order to evaluate the similarity more accurately.

The above object is a visual focusing method for determining a visual focus region of a still image according to an aspect of the present invention, wherein the feature map and the stimulus map of the still image for each of the predetermined visual features determining a conspicuity map; Determining a maximum stimulus region from the stimulus map; Determining a feature map of a target object image for each of the visual features; Evaluating the similarity between the feature map of the target object image and the feature map of the maximum stimulus region; Selecting visual features based on the evaluated similarity; And combining the stimulus maps of the selected visual features to obtain a saliency map.

Here, the maximum stimulation region may be determined to be an area including positions representing values greater than or equal to a predetermined threshold with respect to a position representing the maximum value of the stimulus map.

In the visual focusing method, the similarity evaluation may be performed based on the similarity of the histogram extracted from the feature map of the target object image and the maximum stimulus region. The similarity evaluation may include selecting, for each of the target object image and the maximum stimulus region, a bin having a larger value than the left and right bins among the extracted histograms as a peak bin; Calculating a ratio of the size and distance of other peak bins based on the peak bin having the maximum value; And calculating the similarity represented by the ratio of the size and distance calculated for each of the target object image and the maximum stimulation region, thereby increasing the accuracy of the similarity evaluation.

Further, in the visual focusing method of the present invention, selecting the visual features based on the evaluated similarity, preferably selects a predetermined number of visual features according to the order of high similarity.

Further, in the visual focusing method of the present invention, selecting visual features based on the evaluated similarity, it is preferable to select visual features having a similarity value that is greater than a predetermined reference value.

In the present invention, even when the main visual features of the target object are not identified in the image, the probability of visual focusing being successfully performed is very high. As will be described later, the present invention has been confirmed through experiments that there is a performance improvement of about 57% compared to the conventional top-down visual focus algorithm.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a visual focusing method according to an exemplary embodiment of the present invention.

Referring to Fig. 1, the visual focusing method according to an embodiment of the present invention starts from preparing a target object image and a still image (ST-1).

There is no particular limitation in the present invention with respect to the process or cause of selecting the object as the target object. If the robot performs the task of finding the object, the target object may be determined by the user's command. However, the present invention is not limited to the case where the target object is determined while the robot executes the user's command.

Here, the target object may be determined from a human mental cause, and a specific object may be targeted from human emotions, desires, motivations, moods, and the like in the top-down visual focusing method as described above. Therefore, the target object of the present invention may be one or more objects determined through a computational model or algorithm of the top-down view concentration method.

Images related to the target object may be stored and stored in advance, or may be updated through learning. In the present embodiment, a process of selecting the target object as a milk pack and performing the visual focusing method of the present invention on the target object image as shown in FIG. 2 will be described in detail later.

The still image is an image to be visually focused. The still image may be a surrounding image of the robot taken in real time, or may be selected from among pre-stored images. The present invention is not particularly limited in the process of generating a still image. In the present embodiment, a process of performing visual focus on the image shown in FIG. 3 will be described.

Referring back to FIG. 1, the visual focusing method of the present invention determines a feature map and a conspicuity map for a still image as a next step (ST-2).

The most common method of determining a feature map is to linearly filter an input image. Generally, a still image is selected based on visual features such as color, intensity, and orientation. Obtained by linear filtering.

For example, for brightness I, it is determined in the following way:

 I = (r + g + b) / 3

The channels (R, G, B) for each of red, green, and blue are determined by the following equation:

R = r- (g + b) / 2, G = g- (r + b) / 2, B = b- (r + g) / 2

Here, r, g, and b represent red, green, and blue components of the pixel.

Since the linear filtering method has been introduced to the visual focus model first proposed in 1985 and is still usefully used, a more detailed description thereof will be omitted for simplicity.

4 illustrates a feature map obtained by linearly filtering the still image of FIG. 3 according to an exemplary embodiment of the present invention.

The feature map shown in Fig. 4 is composed of two kinds of brightness (intensity on, intensity off), four patterns (orientation 0 °, 45 °, 90 °, 135 °) and four colors (red, green, blue, yellow). Based on a total of 10 primitive visual features.

The method of obtaining the stimulus map is also a well-known technique as in the case of the feature map, and the center-surround difference is most commonly used. Various methods have been proposed for more specific formulas for implementing this, and one of them is briefly described as follows.

For example, in the case of brightness, the stimulus map can be determined by the following equation:

I (c, s) = | I (c) -I (s) |

Here, c represents the center position and s represents the peripheral position of c.

In addition, in the case of color, a center-periphery difference technique is applied by pairing colors reflecting a color double-opponent relationship. For example, red at the central location is suppressed by green and vice versa at the peripheral location, which can be determined by the formula:

RG (c, s) = | (R (c) -G (c))-(G (s) -R (s)) |

As described above, a method of determining a feature map and a stimulus map is well known, and the present invention is not limited to a specific method among the well-known methods, and thus a detailed description thereof will be omitted. More specific information is available from a paper published in Koch and Ullman in 1985, a paper published by S.Frintrop in 2006 (VOCUS), and a paper published by Itti in 1998 (A Model of Saliency-Based Visual Attention for Rapid Scene Analysis). Reveal that it can be obtained.

From the feature map shown in FIG. 4, the stimulus map as shown in FIG. 5 can be obtained:

When the rate of change of each visual feature is large, the stimulus is evaluated to be high, and the stimulus map can be obtained by expressing the degree of stimulus for the visual feature in brightness.

Referring back to FIG. 1, in the next step, the visual focusing method of the present invention performs a procedure for determining the maximum stimulation region from the stimulus map (ST-3).

In the present embodiment, in determining the maximum stimulation region, the maximum stimulation region is determined up to a position including the position where the degree of stimulation is greatest and indicating a predetermined reference value or more.

However, it will be appreciated that the method of determining the maximum stimulus area can be selected based on various criteria based on the degree of stimulus. For example, a method of selecting a region having a constant size based on the maximum position of the magnetic pole, or selecting a region having the largest average value of the magnetic poles with respect to the region having the predetermined size may be applied.

6 is an image showing a maximum stimulation region selected according to an embodiment of the present invention. Note that since the area is included as a region indicating a stimulus degree above the reference value, the size of the maximum stimulation region may not match the size of the target object image. For the sake of understanding, the process so far to determine the maximum stimulus region of FIG. 6 from the still image of FIG. 3 is clearly shown in FIG.

In the next step of the visual focusing method of the present invention, the feature map of the target object image is determined (ST-4). Accordingly, the feature map for the target object image of FIG. 2 is listed in FIG. 8 for each visual feature.

In the next step, the feature map of the target object image is compared with the feature map of the maximum stimulus region determined in the previous step, and evaluation of the similarity is performed (ST-5).

As a method of evaluating the similarity between images, a method of evaluating the similarity of a color histogram after extracting a color histogram is mainly used. The most common is a histogram intersection technique. The present invention does not exclude how to apply the histogram intersection technique to visual features other than color to obtain the same conclusions. However, in order to compensate for the drawbacks of the histogram intersection method, the present invention proposes a more effective method of assessing similarity, which will be described later.

9 is a view illustrating a process in which feature maps of a target object image and feature maps of a maximum stimulus region are compared with each other based on their histograms, and visual features are listed according to the magnitude of similarity obtained as a result of the comparison according to an embodiment of the present invention. It is a conceptual diagram that is clearly expressed.

The magnitude of similarity obtained through the above process is compared with each other, and a predetermined number of visual features having high similarity are selected (ST-6). For example, based on the similarity obtained in FIG. 9, five visual features having a large similarity to 10 kinds of visual features are arranged. As shown in FIG. 10, 'orientation 45', 'orientation 135', 'green', ' blue ',' red 'can be selected.

Next, the visual focusing method of the present invention combines the stimulus maps of the selected visual features to form a final stimulus map (ST-7).

FIG. 10 illustrates a process of constructing a final stimulus map by combining stimulus maps of five visual features selected according to the magnitude of similarity, and performing concentration based on the stimulus map. Referring to FIG. 10, five visual features selected as having high similarity, that is, stimulus maps corresponding to 'orientation 45', 'orientation 135', 'green', 'blue', and 'red' are combined. You can see that the map is constructed.

Since the stimulus map of the present invention obtained as described above is composed by evaluating visual features of the target object shown in the still image with similarity, and combining the stimulus maps of the visual features most similarly reflected, the still image is the target object. It allows the visual focus to be successfully performed even in situations where the unique visual features are not exhibited.

Finally, the visual focus region is determined from the stimulus degree map (ST-8).

The method of performing the visual focus from the stimulus map is generally performed by focusing the highest stimulus area first, but a lot of researches on how to determine the area to be focused first or next have been conducted. Techniques are already known. Since the present invention has a main feature in the method of constructing the stimulus map, a detailed description of the method of performing the visual focus on the stimulus map will be omitted.

11A and 11B show the order of visually focused areas when performing visual focus on stimulus maps obtained by applying the top-down VOCUS algorithm and the present invention, respectively, from the target object images and still images of FIGS. 2 and 3.

In FIG. 11A to which the VOCUS algorithm is applied, it can be seen that the first area that is visually focused is the right bottom of the packaging material of the product called 'lipton', and the milk carton is concentrated for the third time. Therefore, when the first focus of the target object is called success, the VOCUS algorithm can be said to fail in concentration.

In contrast, in FIG. 11B to which the present invention is applied, the first area that is visually focused is a milk carton.

Therefore, the present invention can be evaluated to be successful in concentration, which shows that the present invention shows better results than the top-down VOCUS algorithm when the maximum stimulus area or key visual features of the target object image are obscured. will be. These differences are similar to those of the VOCUS algorithm that does not obtain a good stimulus map in the case where the main visual features are masked by evaluating and weighting the visual features of the target image in advance. After evaluating, the combination of the stimulus maps for the visual features exhibiting high similarity will be attributed to the configuration differences of the present invention which solved the disadvantages indicated by the VOCUS algorithm.

In order to further compare the effects of the top-down VOCUS algorithm and the top-down visual focusing method of the present invention, the objects listed in FIG. 12A are randomly arranged in the environment as shown in FIG. The experiment to apply was performed repeatedly.

For performance comparison, visual focusing is performed based on a saliency map obtained through each algorithm, and the success rate is compared to the number of experiments by evaluating 'success' when the target object is included in five concentration areas. Was calculated.

The following table shows the success rate for each target object by algorithm.

In the above table, the success rate of the present invention was shown to be high. More specifically, it can be seen that the present invention has a success rate of 57% higher than that of the top-down VOCUS algorithm.

On the other hand, the excellent success rate was also affected by using the unique histogram analysis technique of the present invention instead of the histogram intersection technique that is commonly used in analyzing the similarity of the feature map.

Hereinafter, a method of determining similarity of feature maps based on a histogram according to an embodiment of the present invention will be described in detail.

13A and 13B are conceptual views illustrating a histogram processing process for explaining a histogram analysis method according to an exemplary embodiment of the present invention.

As shown in FIG. 13A, first, a histogram is extracted from a feature map of a target object image and a maximum stimulus region. The size of the bin used in this histogram can be chosen arbitrarily. Since many methods have already been proposed for determining the appropriate size, it will be easy for those skilled in the art to select the size of the bin without difficulty.

From the histogram extracted from the target object image and the feature map of the maximum stimulus region, a so-called peak bin is selected, and the remaining bins are deleted to obtain a histogram composed of only peak bins. The peak bin is defined as a bin having a value larger than the size of the bin located at the left / right of successive bins.

Next, for each of the target object image and the feature map of the maximum stimulus region, the peak bin having the largest value among the peak bins of the histogram is selected as the 'maximum peak bin'.

And, as illustrated in FIG. 13B, for each of the feature map of the target object image and the maximum stimulus region, the ratio P _Oi , P _Mi of the size of each of the peak bins _Bi is based on the maximum peak bin MB. , And obtain a distance d _i spaced from the maximum peak bin, and compare the results with the results of each of the feature maps of the target object image and the maximum stimulus region to quantify the similarity.

For example, the size ratio and distance of the first peak bin located to the right from the maximum peak bin are P ₁ and d ₁ , respectively, and the indexes O and M are respectively displayed for the target image and the feature maps of the maximum stimulus region. Similarity (D _m1 , D _d1 ) can be quantified by the following formula:

The equations can be generalized as follows for the i-positioned peak bin from the maximum peak bin:

Here, MB denotes the size of the maximum peak bin, B _i denotes the size of the i-th located peak bin, and d _i denotes the distance from the maximum peak bin to the i-th peak bin.

The similarities evaluated for each peak bin are all collected to numerically represent the similarity of the entire histogram. The collection method can be variously selected. For example, the average value of the similarities for the size and distance can be obtained and added by weighting, or the method can be determined in various ways.

Therefore, the present invention is the main feature of the method for evaluating the similarity of the histogram by obtaining the similarity of the relative size and distance to the maximum peak bin after obtaining the peak bin, and is not limited or limited to a specific aggregation method.

In the embodiment of the present invention, the histogram similarity was evaluated by calculating the similarity between the size and the distance, and calculating the weighted average by applying the distance from the maximum peak bin as the weight.

Although the embodiments of the present invention have been described so far, those skilled in the art will understand that modifications, substitutions, and the like of the embodiments are possible without departing from the technical spirit of the present invention.

For example, obtaining a feature map of the target object image in the flowchart shown in FIG. 1 does not necessarily have to be performed in the sequence shown in FIG. Those skilled in the art will appreciate the fact that the step of obtaining the feature map of the target object image and the previous steps have no logical relationship.

In addition, among the partial methods or algorithms mentioned in the embodiments of the present invention, many of them may not be sufficient requirements or may be replaced by other algorithms to form the technical idea of the present invention. Therefore, many techniques that are not mentioned in addition to the techniques introduced in the embodiments of the present invention can be used, and the technical concept of the present invention should be understood to include all cases of using these techniques.

Therefore, the embodiments of the present invention should be understood as illustrative and should not be construed as limiting the technical spirit of the present invention. The technical spirit of the present invention is defined by the description of the appended claims, and the scope should be regarded as the equivalents of the invention described in the claims.

1 is a flow chart illustrating a top-down visual focusing method according to an embodiment of the present invention;

2 is an image selected as a target object image according to an embodiment of the present invention;

3 is a still image for performing visual focus according to an embodiment of the present invention;

FIG. 4 is a view listing feature maps obtained by linear filtering the still image of FIG. 3 according to an embodiment of the present invention; FIG.

FIG. 5 lists, by visual feature, the stimulus maps obtained from the feature maps of FIG. 4; FIG.

FIG. 6 lists, by visual feature, a maximum stimulus region selected from the stimulus maps of FIG. 5 in accordance with an embodiment of the present invention; FIG.

FIG. 7 is a diagram illustrating the process of determining the maximum stimulation region of FIG. 6 from the still image of FIG. 3; FIG.

FIG. 8 is a view listing feature maps of the target object image shown in FIG. 2 by visual feature according to an embodiment of the present invention; FIG.

9 is a view illustrating a process in which feature maps of a target object image and feature maps of a maximum stimulus region are compared with each other based on their histograms, and visual features are listed according to the magnitude of similarity obtained as a result of the comparison according to an embodiment of the present invention. Easy-to-understand conceptual diagrams;

FIG. 10 is a conceptual diagram illustrating a process of constructing a final stimulus map by combining stimulus maps of five visual features selected according to the magnitude of similarity, and performing concentration based thereon; FIG.

11A and 11B are views showing the order of visually focused areas when visual focusing is performed on stimulus maps obtained by applying the top-down VOCUS algorithm and the present invention from the target object image and the still image of FIGS. 2 and 3, respectively. ;

12A and 12B respectively show images and environments of objects used in an experiment for comparing the performance of the VOCUS algorithm and the visual focusing method of the present invention; And

13A and 13B are conceptual views illustrating a histogram processing process for explaining a histogram analysis method according to an exemplary embodiment of the present invention.

Claims (6)

In the visual focusing method for determining the visual focus area of the still image, Determining a feature map and a conspicuity map of the still image for each of predetermined visual features; Determining a maximum stimulus region from the stimulus map; Determining a feature map of a target object image for each of the visual features; Evaluating the similarity between the feature map of the target object image and the feature map of the maximum stimulus region; Selecting visual features based on the evaluated similarity; And Combining the stimulus maps of the selected visual features to obtain a saliency map, The similarity evaluation is evaluated according to the similarity of the histogram extracted from the feature map of the target object image and the maximum stimulation region. The method of claim 1, And the maximum stimulus region is determined as an area including positions representing values above a predetermined threshold with respect to the position representing the maximum value in the stimulus map. delete The method of claim 1, The similarity evaluation, Selecting, for each of the target object image and the maximum stimulus region, a bin having a larger value than the left and right bins of the extracted histograms as a peak bin; Calculating a ratio of the size and distance of other peak bins based on the peak bin having the maximum value; And And calculating a similarity represented by the ratio of the size and distance calculated for each of the target object image and the maximum stimulation region. The method according to any one of claims 1, 2 or 4, And selecting the visual features based on the evaluated similarity is selecting a predetermined number of visual features according to the order of high similarity. The method according to any one of claims 1, 2 or 4, Selecting visual features based on the evaluated similarity, selecting visual features having a similarity value that is greater than a predetermined reference value.

Images