KR20110056999A - Method and apparatus for malicious photo filtering using semantic features of digital photo - Google Patents

Abstract

PURPOSE: A harmful image discriminating and blocking apparatus is provided to efficiently model big harmful meaning through a meaning based feature and to efficiently block a harmful image. CONSTITUTION: A pre-processing unit(2000) divides an input image into one or more areas. A visual feature generator(3000) generates visual feature about the divided areas. A meaning feature generator(4000) generates the meaning feature from visual features by area. A harmful image blocking unit(5000) decides the final harmfulness of the image. A meaning feature modeling unit(6000) models meaning feature models which are stored in a meaning feature model storage(4300).

Description

Method and apparatus for identifying and blocking harmful images using semantic-based features {Method and apparatus for malicious photo filtering using semantic features of digital photo}

The present invention relates to a method and apparatus for determining and blocking harmful images, and more particularly, to a method and apparatus for identifying and blocking harmful images using a semantic based feature.

The present invention is derived from a study conducted as part of the Ministry of Knowledge Economy [Task Management No .: 2009-F-054-01, Task name: Development of harmful multimedia content analysis / blocking technology].

Recently, with the proliferation of information spaces such as the Internet, IPTV, and social networks, the distribution and consumption of multimedia contents are rapidly increasing. In particular, due to the development of the Internet and the development of content production devices for individual users, the distribution of contents generated by non-specialists as well as contents generated by existing experts is rapidly increasing. In the case of user created contents (UCC), unlike the contents created by existing experts, reliable metadata (data describing the data, eg, EPG, PPG) that describes the content Not provided. With the development of the Internet and content production technology, the amount and accessibility of user-generated content has increased, but the lack of information on the content raises the problem of indiscriminate distribution of harmful contents such as pornography and violent materials and the inability to block them effectively. have.

In order to solve this problem, many studies have been conducted to block harmful contents. Existing methods for blocking harmful content can be classified into IP based blocking method, text based blocking method and content based blocking method of video. First, the IP-based blocking method is to block harmful content based on the IP address where the harmful content exists. However, not only is the growth rate of content generated in the Internet space fast, but the content's IP address can be very flexible and changeable. Therefore, it is not enough to block harmful content based on restrictive IP address. Second, the character-based blocking method is a method of blocking harmful content by using character information such as title, description, and surrounding words of the content. There is also a problem in the way of blocking content that has words related to such harmful content. First, words can contain multiple meanings. Even words related to the harmful content may appear with the general content in a different meaning. Accordingly, non-hazardous content may be blocked. Second, the word blocking method does not detect words that are intentionally mislabeled to avoid blocking. Finally, there is not enough word set related to the harmful meaning to block the harmful content, there is a problem to block the harmful content.

In order to solve this problem, many researchers have studied content-based parental content blocking technology. Content-based harmful content blocking technology is a technology that blocks harmful content by analyzing the content of the content to determine whether harmful content such as exposure exists. However, the existing content-based harmful content blocking technology has been based on low-level features based on visual factors such as color and texture of human skin. However, the harmful meanings (violence, exposure, ...) are vague and complex to model as low-level features. The challenge is to overcome the limitations of content-based parental content blocking technology based on these low-level features.

An object of the present invention is to solve the problems of the existing harmful image determination and blocking system, and to obtain semantic features for determining harmful meaning in an image, and to provide a method and apparatus for identifying and blocking harmful content based on the same. There is this.

In order to achieve the above object, the harmful image blocking method of the present invention includes a preprocessing step of dividing an input image into a plurality of areas, a visual feature generation step of generating visual features of meaning for each of the divided areas, and A semantic feature modeling step of modeling a semantic feature using the visual feature, a semantic feature generation step of generating semantic features of each divided region based on the generated visual feature, and merging the generated semantic feature A harmfulness determination step of determining a harmfulness of the input image based on a merged meaning feature, a hazard modeling step of modeling harmfulness based on the semantic feature, and the merged semantic feature, and the harmfulness determination step And a harmful image blocking step of blocking the inputted image. .

The visual feature generating step may include a visual feature extraction step of extracting a visual feature of meaning with respect to each of the divided regions, and a feature normalization step of normalizing the extracted visual feature. In the extracting step, visual features of meaning may be extracted using color, texture, shape information or MEPG-7 descriptors of the input image.

In the semantic feature modeling step and the hazard modeling step, each of the divided regions may be modeled using a support vector machine (SVM), and in the merging of the semantic features, the divided regions may be modeled. It is preferable to merge the semantic features with the semantic features having the largest confidence value among the semantic features. In addition, in the harmfulness determination step, when the reliability of the harmful concept is more than a predetermined threshold value can be determined as a harmful image.

The present invention also provides an image input unit for receiving an image, a preprocessor for dividing the input image into a plurality of regions, a visual feature generator for generating visual features of meaning for each divided region, and A semantic feature modeling unit for modeling a semantic feature using the visual feature, a semantic feature generation unit for generating semantic features of each divided region based on the generated visual feature, and merging the generated semantic feature Meaning feature merging unit, a hazard modeling unit for modeling the harmfulness based on the semantic feature, a harmfulness determination unit for determining whether the input image is harmful based on the merged semantic features, and the harmfulness determination unit is determined to be harmful It can also be achieved by a harmful image blocking device including a harmful image blocking unit for blocking the input image received .

The preprocessing unit may include an area template model storage unit for storing an area template model, and an area divider for dividing the input image into a plurality of areas using the area template model. The generation unit may include a visual feature extraction unit for extracting the visual features of each of the divided regions, and a feature normalization unit for normalizing the extracted visual features.

According to the present invention, ambiguous harmful meanings such as exposure that are difficult to model with content-based visual feature values such as color, texture, and shape can be effectively modeled using semantic based features of small meanings that constitute a large harmful meaning such as chest and hips. By doing so, harmful images can be blocked more effectively.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various forms, and only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully convey the scope of the invention to those skilled in the art, and the present invention is defined only by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, “comprises” and / or “comprising” refers to the presence of one or more other components, steps, operations and / or elements. Or does not exclude additions.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, Figure 1 is a block diagram showing the configuration of a harmful image blocking device according to an embodiment of the present invention.

As shown in FIG. 1, the apparatus for blocking harmful images according to an exemplary embodiment of the present invention may include an image input part (1000), a pre-processing part (2000), and a visual feature generation unit (3000). ), A semantic feature generation part (4000), a semantic feature modeling part (Semantic feature modeling) 6000, a malicious image filtering part (5000), and a harmful modeling part (Malice modeling, 7000) It is configured to include.

The image input unit 1000 receives an image requiring harmful determination. According to an exemplary embodiment of the present invention, the image input unit 1000 may receive content generated by a user through the Internet or receive content stored in an arbitrary remote or local storage device. The embodiment of the present invention assumes a content file type such as JPEG, BMP, GIF, etc., which is widely used as a file type of media content, but is not limited to the content file type of the above example. In this case, the harmful image includes violent and pornographic images.

2 is a view showing the configuration and function of the preprocessor 2000 of the harmful image blocking device according to an embodiment of the present invention.

As shown in FIG. 2, when there is an image input through the image input unit 1000, the preprocessor 2000 extracts an input image based on an area template model to extract spatial distribution features of meanings in the image. Split into one or more areas. To this end, the preprocessor 2000 includes a region template model 2200 and a region segmentation 2100. Each of the segmented regions 2300 divided by the region divider 2100 enters an input of the visual feature generator 3000.

3 illustrates a region segmentation template of an image according to an embodiment of the present invention. According to an embodiment of the present invention, as shown in FIG. 3, the image may be divided into six templates. The six region segmentation based templates are represented by Equation 1 below.

Here, R (t) represents the t-th regional partition template.

If the input image I has a size of horizontal w and vertical h, the coordinates of each region segmentation template are expressed by Equation 2 below.

Here, left (t) represents the x-axis coordinate of the left side of the t-th template, top (t) represents the y-axis coordinate of the upper left of the t-th template, and right (t) represents x of the right side of the t-th template Bottom (t) represents the y-axis coordinate of the bottom right of the t-th template.

The coordinates of each template shown in [Equation 2] are as shown in [Equation 3] below, and FIG.

The input picture I divided according to the region division template may be represented by Equation 4 as follows.

4 is a view showing the configuration and function of the visual feature generating unit 3000 of the harmful image blocking device according to an embodiment of the present invention.

As shown in FIG. 4, the visual feature generator 3000 generates a visual feature for each of the divided regions 2300 obtained from the preprocessor 2000, and for this purpose, the visual feature generator 3000 is generated. ) Includes visual feature extraction (3100) and feature normalization (3200). That is, the divided region 2300 is extracted with the visual feature through the visual feature extractor 3100, and each extracted visual feature is normalized by the feature normalizer 3200.

The visual feature value used may be expressed by Equation 5 as follows.

는 사용된 특징값의 개수를 나타낸다. 본 발명의 실시예에서는 시각적 특징값을 추출하는 방법에 있어서 영상의 색상, 질감, 모양 정보 등을 기본으로 이용하는 것을 포함하며, 기본적으로 MPEG-7 기술자(descriptors)를 사용하여 특징값을 추출하는 방법을 사용한다. 그러나, 시각적 특징값을 추출하는 방법이 MPEG-7 기술자에 한정되지 않고 BovW(bag of visual words) 및 일반적인 시각적 기술자도 사용할 수 있음은 물론이다. 또한 추출된 특징값을 정규화하여 여러 시각 특징을 사용함에 있어서, 특징값들이 다양한 범위와 차원을 가짐에도 일부 지배적인 특징에만 의존하지 않도록 한다. here,

Denotes the number of feature values used. In an embodiment of the present invention, a method of extracting a visual feature value includes basically using color, texture, and shape information of an image, and basically extracting the feature value using MPEG-7 descriptors. Use However, the method of extracting the visual feature values is not limited to MPEG-7 descriptors, but of course, the bag of visual words (BovW) and general visual descriptors can be used. Also, in using the various visual features by normalizing the extracted feature values, the feature values do not depend only on some dominant features even though they have various ranges and dimensions.

The visual feature value extracted from the region of the image segmented by the template R is expressed by Equation 6 below.

According to an embodiment of the present invention, the harmful image blocking device includes modeling a semantic feature of a region included in each segmented region of an image based on a given region-based feature value, and the function includes a semantic feature modeling unit 6000. Is performed by

5 illustrates a configuration and a process for modeling semantic features in a harmful image blocking device according to an embodiment of the present invention.

The semantic feature modeling unit 6000 functions to model semantic feature models to be stored in the semantic feature model storage unit 4300. The semantic feature modeling unit 6000, as shown in FIG. 5, uses the semantic feature modeling unit 6100 and the non-hazardous semantic feature modeling unit 6200 through the preliminary training of the respective images through the semantic feature model 4300: 4310. 4320). The semantic feature models 4300 and 4320 are used as the semantic feature vector generator 4000 as a semantic model trained in advance for harmful content determination.

In order to generate the semantic feature model 4300, first, semantic features included in the purpose of harmful and harmless semantic determination are defined. If the meaning characteristic is L Meaningful and harmless features are expressed by Equation 7 as follows.

Where L (l) denotes the l- th semantic feature and N _semantic is the total number of semantic features.

In defining semantic features to determine harmful meanings, semantic features are defined to include harmful meaning features that support harmful meanings and non-hazardous meaning features that do not support harmful meanings. For example, in determining meaning features that determine harmful meanings, meaning features such as breasts, abdomen, hips, and genitals supporting harmful meanings, clothes, swimwear, underwear, buildings, roads, You can use semantic features such as trees, sky, sea, etc.

Next, in order to model the above-described semantic features, learning sample images for each meaning are collected. The visual feature value is extracted from the collected image, and the extracted feature value is learned using a discriminator. An example of the discriminator used here may be a support vector machine (SVM), but the implementation of the present invention is not limited to the SVM. Each learned SVM _semantic is expressed as shown in [Equation 8].

Here, SVM _L represents the SVM learned for semantic feature L. The input of the SVM _semantic uses the visual feature vector X described above.

The semantic feature generator 4000 generates a semantic feature from the visual features 3300 for each region obtained from the visual feature generator 3000, and the semantic feature generated by the semantic feature modeler 6000 described above. It consists of a semantic feature model storage unit 4300, a region-specific semantic feature generation unit 4100, and a semantic feature combiner 4200 storing a model.

The area-specific semantic feature generator 4100 generates area-specific semantic features from the area-specific visual features 3300 obtained from the visual feature generator 3000. When a semantic feature for each region is generated, the semantic feature combiner 4200 generates a semantic feature that represents an image based on the semantic feature.

6 is a diagram schematically illustrating a process of extracting a harmful / harmful meaning feature vector for each region. For each divided region R, the semantic characteristic can be obtained by the following [Equation 9].

Where X _R represents the visual feature vector of the divided region R, and F _L (R) represents the reliability of the semantic feature L of the divided region R. The semantic feature indicates how much the split region image includes the meaning of the semantic feature.

This process performs SVM on all semantic features selected as harmful / harmless (4310, 4320). Based on the reliability results obtained, a semantic feature vector 4120 for the divided region R is generated. The semantic feature vector generated as described above may be expressed by Equation 10 as follows.

Here, F _semantic (R) represents the values for all the semantic features extracted for the partition region R.

As described above, the semantic feature is merged through the semantic feature merging unit 4200 to obtain the semantic feature of the entire image using the semantic feature measured for each divided region of the image. According to an embodiment of the present invention, a method for merging semantic features of divided regions, including merging semantic features of respective regions into semantic features for the entire image. The feature merging process for selecting the largest confidence value among the semantic features of each region is expressed as in Equation 11 below.

는 입력 영상을 나타내고,

는 입력 영상의 의미 L에 대한 병합된 의미 특징 값을 나타낸다. 위와 같은 병합 과정은 본 발명의 하나의 실시예이다. 한편, 영상이 여러 개의 의미 특징값을 가질 수도 있으며, 이 때에는 신뢰값의 경계치를 두어 영역의 의미 특징의 신뢰값이 경계치 이상의 값을 가지는 경우 전체 영상의 의미로 병합할수 있다. here

Represents the input image,

Denotes a merged semantic feature value for the semantic L of the input image. The merging process as described above is one embodiment of the present invention. On the other hand, the image may have a plurality of semantic feature values. In this case, if the confidence value of the semantic feature of the region has a value greater than or equal to the boundary value of the confidence value, the image may be merged into the meaning of the entire image.

The harmful image blocking unit 5000 determines whether the image is final harmful.

FIG. 7 schematically illustrates a harmful meaning modeling process used in a process of determining a final harmfulness of an image by a harmful image blocking unit in a harmful image blocking apparatus according to an exemplary embodiment of the present invention.

The hazard modeling unit 7000 serves to model a hazard concept to be stored in the hazardous model storage unit 5200. The hazard modeling unit 7000 is trained in advance to generate hazard concepts. First, the harmful concept is expressed as Equation 12 below.

Here, M (g) means the g-th harmful concept, N _g represents the number of defined harmful concepts.

The learned SVM _malicious for each harmful concept is expressed as Equation 13 below.

Here, SVM _M represents the SVM 5200 learned for the harmful concept M, and the input of SVM _malicious includes F ' _{semantic, which} is a _semantic feature set extracted from the partitions and merged.

In determining the harmful meaning for the harmful image determination, for example, harmful meanings such as upper exposure, whole body exposure, backside exposure, and genital exposure may be used.

8 illustrates a configuration and a function of the harmful image blocking unit 5000 of the harmful image blocking apparatus according to the exemplary embodiment of the present invention.

The input of the harmful image blocking unit 5000 is a semantic feature vector 4300 obtained from the semantic feature generating unit 4000. As shown in FIG. 1, the harmful image blocking unit 5000 includes a harmful model storage unit 5200 and a harmful image blocking unit 5100. Using the harmful concept model stored in the harmful model storage unit 5200, the harmful content is classified using the semantic feature vector input from the harmful image blocking unit 5100. Using SVM _malicious (5200), the learned harmful concept model stored in the harmful model storage unit, the reliability value of whether the input image I has a harmful meaning is obtained through Equation 14 as follows.

Where V _M represents the confidence in the hazard concept M. The harmful image blocking unit 5100 discriminates and blocks the image 5300 having the harmful concept if the reliability obtained above is greater than or equal to a predetermined threshold, and otherwise determines the non-hazardous image 5400.

On the other hand, the technology for determining the harmfulness of the content using the semantic features according to the present invention is not limited to the image content can be applied to various types of digital content. In addition, the hazard determination technology using the semantic features of the present invention may be extended to applications such as IPTV and DMB.

The harmful image blocking method and apparatus according to the embodiment of the present invention may be embodied as computer readable codes on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, hard disk, floppy disk, flash memory, optical data storage device, and also carrier waves (for example, transmission over the Internet). It is also included to be implemented in the form of. The computer readable recording medium can also be distributed over computer systems connected over a computer network so that the computer readable code is stored and executed in a distributed fashion.

Although the present invention has been described above with reference to preferred embodiments, the harmful image blocking method and apparatus of the present invention are not necessarily limited to the above-described embodiments, and various modifications or variations may be made without departing from the spirit and scope of the present invention. It is possible. The appended claims will cover such modifications and variations as long as they fall within the spirit of the invention.

1 is a block diagram showing the configuration of a harmful image blocking device according to an embodiment of the present invention.

2 is a view showing the configuration and function of the preprocessor of the harmful image blocking device according to an embodiment of the present invention.

3 illustrates a region segmentation template of an image according to an embodiment of the present invention.

4 is a view showing the configuration and function of the visual feature generating unit of the harmful image blocking device according to an embodiment of the present invention.

5 illustrates a configuration and a process for modeling semantic features in a harmful image blocking device according to an embodiment of the present invention.

6 shows a process of extracting a harmful / harmful meaning feature vector for each region.

FIG. 7 illustrates a harmful meaning modeling process used in a process of determining whether a harmful image is finally harmful by an harmful image blocking unit in the harmful image blocking apparatus according to an exemplary embodiment of the present invention.

8 illustrates a configuration and a function of a harmful image blocking unit of a harmful image blocking apparatus according to an exemplary embodiment of the present invention.

Claims (20)

A preprocessing step of dividing the input image into a plurality of regions; A visual feature generation step of generating visual features of meaning for each divided region; A semantic feature modeling step of modeling a semantic feature using the generated visual feature; Generating a semantic feature of each of the divided regions based on the generated visual feature; A semantic feature merging step of merging the generated semantic features; A hazard modeling step of modeling hazards based on semantic features; A hazard determination step of determining whether the input image is harmful based on the semantic features merged; Harmful image blocking method comprising the harmful image blocking step of blocking the input image determined to be harmful in the harmfulness determination step. The method of claim 1, wherein in the pretreatment step, Receives a region template and divides the input image into a plurality of regions R ₁ , R ₂ , R ₃ , R ₄ , R ₅ , and R ₆ . The coordinates of each template are as shown in [Equation 15].

Hazardous image blocking method determined as follows. The method of claim 1, wherein the visual feature generating step, A visual feature extraction step of extracting visual features of meaning for each of the divided regions; And a feature normalization step of normalizing the extracted visual features. The method of claim 3, wherein in the visual feature extraction step, The harmful image blocking method of extracting a visual feature of meaning by using the color, texture, shape information of the input image. The method of claim 4, wherein in the visual feature extraction step, Hazardous image blocking methods, using MEPG-7 descriptors to further extract visual features of meaning. The method of claim 1, wherein in the semantic feature modeling step, A harmful image blocking method for modeling the meaning of each divided region using a support vector machine (SVM). The method of claim 6, wherein in the semantic feature modeling step, The meaning of each divided region is represented by Equation 16 below.

(R is each of the divided regions, X _R is the visual feature vector of the region R, and F _L (R) is the reliability of the semantic feature L of the region R) The harmful video blocking method determined by the. The method of claim 1, wherein in the merging of semantic features, The harmful image blocking method of merging the semantic features into the semantic features having the largest confidence value among the semantic features of the divided regions. The method of claim 1, wherein in the hazard modeling step, Harmful image blocking method that models harmful meanings using SVM (support vector machine). According to claim 1, In the hazard determination step, Equation 17

(SVM _M is SVM learned for harmful concept M, F ' _semantic is a set of semantic features extracted from each of the divided regions and merged) Harmful image blocking method to determine the hazard according to. According to claim 1, In the hazard determination step, Equation 18

(V _M is the reliability of harmful concept M) The harmful image blocking method determines that the image is harmful when the reliability is higher than or equal to a predetermined threshold. An image input unit for receiving an image, A preprocessor dividing the input image into a plurality of areas; A visual feature generator for generating a visual feature of meaning for each divided region; A semantic feature modeling unit for modeling semantic features using the generated visual features; A semantic feature generator for generating semantic features of each of the divided regions based on the generated visual features; A semantic feature merging unit for merging the generated semantic features; A hazard modeling unit for modeling hazards based on semantic features, A hazard determination unit that determines whether the input image is harmful based on the merged semantic features; Harmful image blocking device including a harmful image blocker for blocking the input image determined to be harmful in the hazard determination unit. The method of claim 12, wherein the preprocessing unit, An area template model storage unit for storing the area template model; And a region dividing unit which divides the input image into a plurality of regions by using the region template model. The method of claim 13, wherein the region template model comprises a plurality of region templates R ₁ , R ₂ , R ₃ , R ₄ , R ₅ , R ₆ , The coordinate of each template is represented by Equation 19 below.

Hazardous image blocking device determined as follows. The method of claim 12, wherein the visual feature generating unit, A visual feature extraction unit for extracting visual features of each of the divided regions; A harmful image blocking device comprising a feature normalization unit for normalizing the extracted visual feature. The method of claim 12, wherein the semantic feature modeling unit, A harmful image blocking device modeling the meaning of each divided region by using a support vector machine (SVM). The method of claim 16, wherein the semantic feature modeling unit, The meaning of each divided region is represented by Equation 20 below.

(R is each of the divided regions, X _R is the visual feature vector of the region R, and F _L (R) is the reliability of the semantic feature L of the region R) Hazardous video blocker determined by. The method of claim 12, wherein the semantic feature merger, And a harmful image blocking device incorporating the semantic feature into the semantic feature having the largest confidence value among the semantic features of the divided regions. The method of claim 12, wherein the hazard determination unit, Equation 21

(SVM _M is SVM learned for harmful concept M, F ' _semantic is a set of semantic features extracted from each of the divided regions and merged) Harmful image blocking method to determine the hazard according to. The method of claim 12, wherein the hazard determination unit, Equation 22

(V _M is the reliability of harmful concept M) Harmful image blocking device to determine the harmful image when the reliability according to the predetermined threshold value or more.

Images