JP2005141776A - Image extraction device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically extract a keyword related to more components from an image intended to be registered to register it as a keyword for retrieving the image. <P>SOLUTION: An image input part 1 quantitizes an inputted image to generate image data. A region extraction part 2 extracts regions from the image data. A region attribute extraction part 3 analyzes each region and provides it, as attributes, with features related to a color, fixture, size and shape of the region. An image structure extraction part 4 investigates a phase relationship among the respective regions to create a data structure expressing a phase relationship among the regions. An object extraction part 5 scans the data structure while referring to a dictionary to replace a part or the whole of the data structure with an object. A keyword extraction part 6 extracts a positional relationship between objects as a keyword of the object image. Image registration part 7 registers the keyword in an image database by associating it with the corresponding image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image extraction technique for extracting a region from an image, and is particularly suitable for extracting an image structure that expresses a region in a phase relationship by focusing on the region in the image.

  It is desirable to extract a region from an image. Here, image search is taken as an example. As a conventional technique related to image search, there is a search technique based on keyword assignment in a general database. This is a method in which an image registrant assigns an arbitrary keyword, or a word to be a keyword is determined in advance, and an appropriate keyword is assigned by referring to these keyword groups at the time of registration. This is a search method based on word matching, in which a search is performed by selecting an appropriate term from a keyword or a group of keywords according to a searcher's intention. This method has the advantage that the configuration of the search device is easy and search can be performed easily and quickly as long as the keyword intended by the searcher is registered, but the registrant assigns the keyword when registering the image. It takes a lot of work. Further, when the registrant and the searcher are different, different keywords may be assigned due to the difference in intention between the two, and a situation in which the searcher cannot search the target image occurs.

  In order to solve the above-described problem, a technique has been proposed in which an example image is used as a search key instead of a keyword as shown in Patent Document 1 and Patent Document 2. This method is a method for realizing a search based on visual characteristics unique to an image from an accumulated image. Basically, this is a technique for searching for an image that meets the conditions of the entire stored image using a schematic image describing the shape, size, position, color, texture, etc. of the image information. In order to improve the efficiency of retrieval from stored images, the representative color of the registered image is mapped, the representative color of the image that is the search key is mapped on this color space, and only for the image existing in the vicinity. By performing matching and evaluation, a certain degree of large classification is performed in the initial stage to increase the speed. However, it is necessary to create an example image for each search, and there are situations where it is difficult to draw an image to be searched for in the example image.

  In addition, the composition information of the material and its attribute information can be easily created / registered by using a drawing processor or the like at the outline image creation stage as shown in Patent Document 3, and the table of the composition information and its attribute information is also searched. There is also a method of using it. The constituent elements here are, for example, a desk and an apple, and their attribute information includes a position, a size, a posture, and the like, and a search with high accuracy is possible. However, it is necessary to add the configuration information / attribute information to the stored image in advance.

  Moreover, as a technique for efficiently searching for a natural image, for example, there is Patent Document 4. This is a technique in which a natural image is converted into a component and the shape / feature information of the component is used as a search key. However, it is premised on human intervention by a mouse / keyboard / digitizer or the like when converting an image into a component.

  The conventional image retrieval methods described above have required considerable labor at the time of image registration or retrieval. As a method for solving this problem, there is a method of extracting a keyword automatically or semi-automatically from an image and registering it together with the image.

  For example, in Patent Document 5, a sense word is automatically extracted from an image and registered together with the image, and this sense word is used as a keyword. This search based on sensory words is effective for a specific group of images such as paintings, but is not very effective for general images. This is due to the fact that it is not easy to extract sensory words from general images and that the sensations differ from person to person.

  Moreover, as a method for extracting a keyword related to a component from an image, for example, there is Patent Document 6. This sets the area of the image data, further extracts the physical information such as color information and frequency information contained in the area, and uses the physical quantity itself as a keyword or from the physical information of the area, for example, words such as sky and sea It is possible to bind to. However, there are limits to the words that can be linked only from the physical information of the area.

It is desirable to extract a region from an existing image and perform image search or the like based on the extracted region.
Japanese Patent Laid-Open No. 4-60770 JP-A-6-243178 Japanese Patent Laid-Open No. 1-130278 JP-A-4-267480 JP-A-1-73460 Japanese Patent Laid-Open No. 2-187864

  The present invention has been made in consideration of the above circumstances, and an object thereof is to provide an image extraction technique for extracting a region from an image.

  According to this invention, in order to achieve the above-mentioned object, the configuration as described in the claims is adopted. Here, prior to describing the invention in detail, supplementary explanations of the claims will be given.

  That is, according to one aspect of the present invention, in order to achieve the above-described object, the image extracting apparatus includes: an image input processing unit that quantizes an input image to generate image data; An area extracting means for extracting an area composed of pixels having similar characteristics from the generated image data is provided.

  For example, the region extracting unit extracts a pixel corresponding to an edge from the image data generated by the image input processing unit, and extracts a closed region constituted by the pixel as a region.

  According to this configuration, it is possible to easily extract a region from an image while paying attention to similar pixels.

  In this configuration, a region attribute extraction unit that extracts physical information of the region extracted by the region extraction unit as an attribute of the region may be provided.

  The area attribute extraction unit extracts, for example, at least one of a plurality of types of feature quantities respectively associated with the color, size, shape, and texture of the area as an attribute of the area.

  Further, there may be provided an image structure extraction unit that calculates a phase relationship between the regions extracted by the region extraction unit and expresses an image by a phase relationship of regions having attributes extracted by the region attribute extraction unit. .

  The image structure extracting unit extracts, for example, a relationship of coincidence, overlap, inclusion, joining, or exclusion as a phase relationship between regions. The image structure extraction unit may extract a distance and a direction between regions based on the coordinates of the center of gravity of the region as the phase relationship between the regions.

  The present invention can be realized not only as an apparatus or a system but also as a method. Of course, a part of the invention can be configured as software. Of course, software products used to cause a computer to execute such software are also included in the technical scope of the present invention. Further, when the present invention is realized by installing software in a computer, the above-described units are appropriately realized by cooperating computer hardware resources and software resources.

  These and other aspects of the invention are set forth in the appended claims and will be described in detail below with reference to examples.

  According to the present invention, a region can be extracted from an image.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of an image registration apparatus of the present invention. In the figure, 1 is an image input unit, 2 is a region extraction unit, 3 is a region attribute extraction unit, 4 is an image structure extraction unit, 5 is an object extraction unit, 6 is a keyword extraction unit, and 7 is an image registration unit.

  The image input unit 1 quantizes the input image and generates image data. The region extraction unit 2 extracts a region composed of pixels having similar characteristics from the image data generated by the image input unit 1. The region attribute extraction unit 3 analyzes each region extracted by the region extraction unit 2 and extracts features relating to the color, texture, size, and shape of the region as attributes. The image structure extraction unit 4 investigates the phase relationship between the regions extracted by the region extraction unit 2 and expresses the phase structure between the region having the attribute extracted by the region attribute extraction unit 3 and each region. Create The object extraction unit 5 scans the data structure created by the image structure extraction unit 4 while referring to the dictionary that associates the attribute condition and the phase relationship between the regions with the object, and part or all of the data structure is scanned. Replace with object. The keyword extraction unit 6 extracts the positional relationship between objects from the data structure updated by the object extraction unit 5 as a keyword of the target image. The image registration unit 7 registers the keyword extracted by the keyword extraction unit 6 in the image database in association with the corresponding image.

  FIG. 2 is a block diagram showing a configuration example for realizing an embodiment of the image registration apparatus of the present invention. In the figure, 101 is an image input processing unit, 102 is an image data storage memory, 103 is a region extraction processing unit, 104 is a region / region attribute storage memory, 105 is a region attribute extraction processing unit, 106 is an image structure extraction processing unit, 107 Is an image structure storage memory, 108 is an object extraction processing unit, 109 is an object dictionary, 110 is a keyword extraction processing unit, and 111 is an image registration processing unit.

  The image input processing unit 101 inputs a document to be input using a scanner or the like, quantizes it, and multi-tone RGB color image data (image data represented by three elements R, G, and B of the RGB color system) ) Is generated. Here, since the input object is a document, a scanner is used as an input means. However, when a landscape such as the outdoors is an input object, it may be input by a video camera or the like. Alternatively, an image drawn using a computer may be used. An input image can be stored once in a storage device and read when processing is performed. In this example, RGB color image data is generated as an input image. However, the present invention is not limited to this, and gray scale image data may be used, and other color specifications such as L * a * b * may be used. A system may be used.

  The image data storage memory 102 stores the image data generated by the image input processing unit 101.

  The region extraction processing unit 103 extracts a region composed of pixels having similar characteristics (color, density, texture, etc.) based on the image data stored in the image data storage memory 102. Conventionally, a method for extracting a large number of regions has been proposed. For example, a non-hierarchical clustering method, a so-called k-average region dividing method can be used. In this embodiment, a case is shown in which all pixels constituting an image are clustered using the k-average region division method based on the hue, saturation, and brightness of each pixel. Since regions in the present invention correspond to components in an image, for example, objects such as faces and eyes, even if extremely small clusters are obtained, it does not make much sense. Therefore, if necessary, the process of integrating the clusters may be performed last. Then, by extracting adjacent pixels belonging to the same cluster by labeling, it is possible to extract a region composed of pixels having similar characteristics. Here, it is not necessary to store all the areas extracted by labeling, and a threshold may be provided for the size and total number of areas to limit the area to be stored.

  In this embodiment, the region extraction processing is performed by k-average region division and labeling. However, the present invention is not limited to this, and any method can be used as long as it can extract a region composed of pixels having similar characteristics. Any technique can be used. As another region extraction method, there is a method of extracting a pixel corresponding to an edge from an image and extracting a closed region constituted by the pixel as a region.

  The region / region attribute storage memory 104 stores the region extracted by the region extraction processing unit 103 as region data. Various methods are conceivable for the configuration of the region data. In this example, the region data is configured with a labeling image as shown in FIG. 3 and stored in the region / region attribute storage memory 104. In the figure, one square corresponds to one pixel, and the numerical value in the square is a number unique to the area to which the corresponding pixel belongs (hereinafter referred to as an area number). The method of constructing region data in the present invention is not limited to a labeling image, and a unique number is assigned to each pixel in the image, and each region is represented by a set of unique numbers assigned to pixels belonging to that region. The region data may be configured by using other methods such as a method and a method of representing the region by a set of contour points.

  The region attribute extraction processing unit 105 extracts attributes corresponding to the respective regions stored in the region / region attribute storage memory 104. The extracted attribute is stored in the area / area attribute storage memory 104 in association with the corresponding area. The attributes in the present invention represent properties related to the color, size, shape, and texture of the region. For example, the average hue, saturation histogram, area, absolute maximum length, approximate shape parameter, roundness ( Features such as the degree of needle-like), average edge strength, and spatial frequency can be used. In the present invention, the attribute to be extracted is not limited to the above-described feature quantity, and any attribute quantity may be used as long as it represents a property related to the color, size, shape, and texture of the region. However, it is preferable to use at least one feature amount that represents properties relating to the color, size, and shape of the region. In this example, for the sake of simplicity, it is used as an attribute for extracting the average hue, average saturation, average brightness, number of constituent pixels, needle-like degree, and principal axis direction, excluding features related to texture. These features can be calculated from the area data by a known method and stored in the area / area attribute storage memory 104 as, for example, a table paired with area numbers (hereinafter referred to as attribute table) as shown in FIG. .

  Further, depending on the region extraction method used in the region extraction processing unit 103, a feature corresponding to the region attribute can be calculated in the region extraction process. In such a case, at the time of calculation, this feature can be stored in the region / region attribute storage memory 104 as an attribute in association with the corresponding region. In this example, the average hue, average saturation, and average brightness of each region can be calculated during the k-average region division, and the number of constituent pixels of each region can be calculated during labeling. Are stored in the region / region attribute storage memory 104. In addition, all the attributes (needle degree, main axis direction) calculated here can be calculated from the area data, but depending on the attributes used (for example, features related to texture), they are stored in the image data storage memory as necessary. An attribute can be calculated with reference to image data.

  The image structure extraction processing unit 106 uses the region data stored in the region / region attribute storage memory 104 to extract the phase relationship between the regions. The phase relationship between regions in the present invention refers to the relationship between regions, such as coincidence / inclusion / overlap / join / exclusion, and may have a distance and direction between regions as a detailed relationship. In this example, since each area is extracted by area division, any two areas do not overlap. Accordingly, the phase relationship between the two regions is either “included / included (included)”, “adjacent (joined)”, or “separated (exclusive)”. As for the phase relationship between the regions, the region data (labeled image) stored in the region / region attribute storage memory 104 is scanned in the horizontal direction and the vertical direction, and adjacent region numbers are extracted for each region. Can be calculated. However, when this method is used, it is necessary to create a region representing the outside of the image in the outer edge portion of the image as shown in FIG. Pixels labeled with a numerical value of “0” in the figure are areas that represent outside the image. The result of extracting the regions adjacent to each region is, for example, as shown in the section of the adjacent region number list in FIG. From this result, it can be seen that the regions of region numbers 2 and 3 that have only one adjacent region are included in the region of region number 1 that is adjacent. Then, by removing the region numbers 2 and 3 from the adjacent region number list of the region number 1, it is possible to obtain a list of regions that are joined but not included. Needless to say, the region representing the outside of the image (the region having the region number 0 in this example) must be excluded from the region list indicating the final phase relationship. The phase relationship extraction method in the present invention is not limited to the above-described method. In this example, the adjacency relationship is calculated in the vicinity of 4 but may be calculated in the vicinity of 8. The distance and direction between the regions used as the detailed relationship between the regions can be obtained by calculating the distance and the direction based on the coordinates of the center of gravity of the region.

  The image structure storage memory 107 stores the phase relationship between the regions created by the image structure extraction processing unit 106. In this example, as shown in FIG. 7, a graph structure (hereinafter referred to as an image structure graph) having an area number at a node and a phase relationship between areas at an edge is created and stored. When the distance and direction between the regions are calculated as a detailed relationship between the regions, these values can also be given to the edges. The image structure graph shown in FIG. 7 is created from the labeling image shown in FIG. 8 (one closed area represents one area). The area numbers 1 to 11 shown in FIG. It is assumed that the face, left eye, right eye, mouth, left eyebrow, right eyebrow, hair, left ear, right ear, neck, and background area are pointed in order. In the image structure graph in this example, only the relationship between the inclusion or adjacent regions is described. However, when the region extraction processing unit 103 extracts a region that allows overlap, the phase between regions such as coincidence and overlap Relationships can be added.

  The object extraction processing unit 108 extracts an object from the image structure graph stored in the image structure storage memory 107 while referring to the object dictionary 109. An object in the present invention refers to a name unique to things such as “person's face” or “tree” and an expression based on attributes of a region such as “red circle” or “white square”.

  The object dictionary 109 is a combination of a graph structure (hereinafter referred to as an object graph) describing object names, attributes of regions constituting the object, and topological relationships between the regions, and object conditions (hereinafter referred to as object items). Hold multiple. The object graph stored in the object dictionary 109 has the same format as the image structure graph. The difference is that an attribute condition corresponding to the number indicated by each node of the object graph is provided instead of the attribute. Hereinafter, the number of the node of the object graph is referred to as an attribute condition number. This attribute condition is a condition for examining whether or not the corresponding nodes of the image structure graph and the object graph match. The object condition is a rule relating to the relative relationship between attributes of the areas constituting the object. For example, the condition of the area A is 2 to 3 times the number of constituent pixels of the constituent pixel area B. It is. FIG. 9 shows an example of an object item corresponding to the object “human face” and an attribute condition of a portion corresponding to the object item. In the figure, the attribute condition number 301 corresponds to the face, 302 and 303 correspond to the eyes, and 304 corresponds to the hair.

  Hereinafter, an example of the object extraction method will be described with reference to the flowchart shown in FIG. First, the attribute table stored in the region / region attribute storage memory 104 is sorted in descending order by the number of constituent pixels (S201). This is based on an empirical rule that a region having a larger area is more likely to be a major component. In the subsequent processing, the sorted attribute table areas (area numbers) are sequentially processed (S202). The attribute conditions registered in the object dictionary 109 are searched to create a list of attribute condition numbers in which the attributes of the processing target area satisfy the conditions (S203). The object graph having the attribute condition number in this list is compared with the image structure graph (S204). This comparison is performed by matching the nodes of the image structure graph having the region number to be processed with the nodes of the object graph having the attribute condition number extracted in S203, and all the attributes corresponding to the nodes of the object graph. It is examined whether the phase relationship corresponding to the condition and the edge is suitable for part or all of the image structure graph. If this is met, the object condition corresponding to this attribute condition number is further applied to determine whether or not it is finally met (S205). If it is finally determined that it does not fit, the process of S208 is performed. If it is matched, the matched subgraph of the image structure graph is replaced with the corresponding object (S206). This replacement is based on a node having a number that can uniquely determine an object. For example, the number of a corresponding object item registered in the object dictionary 109 (referred to as an object number) can be used. Along with this replacement, the edge is deleted, the phase relationship corresponding to the edge, and the distance and direction between regions are corrected. The center of gravity of the object for calculating the distance and the direction can be calculated from an area obtained by integrating the areas included in the partial graph of the image structure graph replaced with the object. Then, the area number included in the partial graph of the image structure graph replaced with the object and the attribute corresponding thereto are deleted from the attribute table sorted in S201 (S207). In S208, it is checked whether or not processing has been performed for all areas. If there is any unprocessed area, the process returns to S202 and the process is repeated.

  As a result of the object extraction processing unit 108, some subgraphs included in the image structure graph are replaced with objects, and a graph having an object number or an area number at each node of the image structure graph is obtained. FIG. 11 shows a graph obtained as a result of performing the object extraction process on the image structure graph shown in FIG. 7 and adapting it to the object graph registered in the object dictionary 109 shown in FIG. It is assumed that the area numbers 1, 2, 3, and 7 in FIG. 7 match the attribute condition numbers 301, 302, 303, and 304 in FIG. 9 and the object conditions shown in FIG. 9 are also satisfied. The edge (topological relationship) between the nodes is matched, and the subgraph consisting of the region numbers 1, 2, 3, and 7 in FIG. 7 is replaced with the node having the object number 100 corresponding to the object “human face”. Has been.

  Here, an example is shown in which the unique name of an object is an object, but an object item is created in the object dictionary 109 such that an expression relating to an area attribute such as “red circle” or “white square” is the object name. Thus, objects such as “red circles” and “white squares” can be extracted.

  In this example, the object graph and the image structure graph are compared by extracting objects based on whether or not each node or edge is suitable. It is also possible to replace the subgraph of the structure graph with the corresponding object. For example, it can be realized by configuring the attribute condition held in the object dictionary 109 with a function such as a fuzzy function that returns the fitness by giving an attribute.

  The object dictionary 109 in the present invention needs to be created in advance. As an example of this creation method, an image structure graph as described above is created for a plurality of human face images, and a plurality of image structure graphs are obtained. Then, an object graph corresponding to the object “human face” can be created from the phase relationship of the edges common to the attributes of the nodes (regions) corresponding to the plurality of image structure graphs. In addition, among the relationships between attributes of the nodes of the obtained plurality of image structure graphs, a relationship that can be regarded as common can be set as an object condition.

  The keyword extraction processing unit 110 analyzes the image structure graph in which the partial graph is replaced with the object by the object extraction processing unit 108 and extracts the relationship between the objects as a keyword. In this example, the object (name unique to the object) determined from the object number existing in the image structure graph and the relative position in the image of the object, for example, the object “human face” and the position of this object The keyword “human face is the center of the image” can be extracted. Further, as a relationship between objects, a relative positional relationship between the extracted objects, for example, a keyword “traffic signal at the upper left of the car” can be extracted.

  The image registration processing unit 111 registers the keyword extracted by the keyword extraction processing unit 110 in the storage device (not shown) in association with the input image.

  As described above, by using the image registration apparatus described above, it is possible to configure a search apparatus that can easily and quickly search for a target image by designating a keyword, without the need for manually assigning a keyword when registering an image. . However, since the keywords extracted by the image registration apparatus according to the present invention and registered together with the images are mainly unique names, the image search is performed so that the synonym of the keyword specified as the search key at the time of search is also searched in addition to the search key. It is preferable to construct the apparatus.

It is a block diagram which shows the image registration apparatus of the Example of this invention as a whole. It is a block diagram which shows the structural example of the image registration apparatus of an Example. It is a figure which shows the example of the labeling image expressing the area | region data in an Example. It is a figure which shows the example of the attribute table of an Example. It is a figure which shows the example of the labeling image used when extracting the phase relationship between area | regions in an Example. It is a figure which shows the example which extracted the adjacent area | region for every area | region in the Example. It is a figure which shows the example of the image structure graph of an Example. It is a figure which shows the example of the labeling image corresponding to the image structure graph of FIG. It is a figure which shows the example of the object dictionary of an Example. It is a figure which shows the example of the processing flow of the object extraction processing method of an Example. It is a figure which shows the example of the image structure graph after the object extraction process of an Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image input part 2 Area extraction part 3 Area attribute extraction part 4 Image structure extraction part 5 Object extraction part 6 Keyword extraction part 7 Image registration part 101 Image input process part 102 Image data storage memory 103 Area extraction process part 104 Area / area attribute Storage memory 105 Region attribute extraction processing unit 106 Image structure extraction processing unit 107 Image structure storage memory 108 Object extraction processing unit 109 Object dictionary 110 Keyword extraction processing unit 111 Image registration processing unit

Claims (10)

Image input processing means for quantizing the input image to generate image data;
An image extracting apparatus comprising: an area extracting unit that extracts an area composed of pixels having similar characteristics from the image data generated by the image input processing unit.   The image extracting apparatus according to claim 1, wherein the region extracting unit extracts a pixel corresponding to an edge from the image data generated by the image input processing unit and extracts a closed region constituted by the pixel as a region.   The image extraction apparatus according to claim 1, further comprising region attribute extraction means for extracting physical information of the region extracted by the region extraction means as an attribute of the region.   The image extraction according to claim 3, wherein the region attribute extraction unit extracts at least one type of feature amount among a plurality of types of feature amounts respectively associated with the color, size, shape, and texture of the region as an attribute of the region. apparatus.   4. The image structure extracting unit according to claim 3, further comprising: an image structure extracting unit that calculates a phase relationship between the regions extracted by the region extracting unit and expresses an image by a phase relationship of regions having attributes extracted by the region attribute extracting unit. 4. The image extraction device according to 4.   The image extraction apparatus according to claim 5, wherein the image structure extraction unit extracts a relationship of coincidence, overlap, inclusion, joining, or exclusion as a phase relationship between regions.   The image extraction apparatus according to claim 5, wherein the image structure extraction unit extracts a distance and a direction between regions based on coordinates of a center of gravity of the region as a phase relationship between the regions. Image input processing means for quantizing the input image to generate image data;
An image extraction apparatus comprising: region attribute extraction means for calculating an attribute relating to texture with reference to the image data. An image input processing step of quantizing the input image and generating image data by the image input processing means;
A region attribute extraction step of calculating an attribute relating to texture by referring to the image data by a region attribute extraction means. An image input processing step of quantizing the input image and generating image data by the image input processing means;
An image extraction method comprising: an area extraction step of extracting an area composed of pixels having similar characteristics from the image data generated by the image input processing means by an area extraction means.

Images