JP6034970B2 - Dictionary generation system, dictionary generation method, and dictionary generation program - Google Patents

Description

  The present invention relates to a dictionary generation system, a dictionary generation method, and a dictionary generation program that generate a dictionary.

  Conventionally, there is an object detection method based on a similar image search that can be applied to a small amount of various objects. In this method, for the partial region of the input image, the nearest neighbor case is searched from a database in which cases to be detected are registered, and it is determined whether the object is an object based on the distance in the feature amount space.

Hiroki Watanabe, Hiroto Nagayoshi, Satoshi Tsunoike, "Case-Based General Object Detection Based on Similar Image Search", IEICE Technical Report, vol.111, no.353 (PRMU2011-124--PRMU2011-146), pp.101- 106,2011.

  In the above-described prior art, it is necessary to register an area including an object to be detected as a dictionary pattern. In order to improve the detection accuracy, it is necessary to register a large number of appropriate dictionary patterns, but there is a problem that the operation cost is increased in actual operation.

  An object of the present invention is to automatically generate a dictionary by automatically registering a highly reliable dictionary pattern.

A dictionary generation system, a dictionary generation method, and a dictionary generation program according to an aspect of the invention disclosed in the present application are a selection area selected from a first area group in a first image and the first area. A first distance in the feature amount space between the first region other than the selected region selected from the group and a second region group in the second region in the selected region and the second image. A second distance in the feature amount space between the selected second area and the selected area based on the ratio between the acquired first distance and the second distance. It is determined whether or not to be a dictionary pattern, and when it is determined to be a dictionary pattern, the selected region is registered in a dictionary in which a dictionary pattern group is stored.
Common attribute information is given to each image of the first image set including the first image, and each attribute of the second image set including the second image is assigned to the attribute. No information is given, the first region group is a region group obtained from each image in the first image set, and the second region group is in the second image set. This is a group of regions obtained from the respective images .
Further, no attribute information is given to any of each image of the first image set including the first image and each image of the second image set including the second image, The first area group is an area group obtained from each image in the first image set, and the second area group is an area group obtained from each image in the second image set. It may be.
Further, attribute information is not given to each image of the first image set including the first image, and common to each image of the second image set including the second image. The first region group is a region group obtained from each image in the first image set, and the second region group is the second image set. It may be a group of regions obtained from each of the images.
In addition, common first attribute information is given to each image of the first image set including the first image, and each image of the second image set including the second image is provided. Are given common second attribute information, and the first region group is a region group obtained from each image in the first image set, and the second region group is A region group obtained from each image in the second image set may be used.

  According to a typical embodiment of the present invention, a dictionary can be automatically generated by automatically registering a highly reliable dictionary pattern. Problems, configurations, and effects other than those described above will become apparent from the description of the following embodiments.

It is explanatory drawing which shows the example of a dictionary production | generation in the dictionary production | generation system concerning this invention. It is explanatory drawing which shows the specific example of the dictionary production | generation in a dictionary production | generation system. It is a block diagram which shows the hardware structural example of a dictionary production | generation system. It is a block diagram which shows the functional structural example of the dictionary production | generation system concerning Example 1. FIG. It is explanatory drawing which shows the example of an area | region production | generation from a 1st image set and a 2nd image set. It is explanatory drawing which shows the area | region production example 1 by a perturbation process. It is explanatory drawing which shows the area | region production example 2 by a perturbation process. It is explanatory drawing which shows the area | region production example 3 by a perturbation process. It is a flowchart which shows the example of a dictionary production | generation process procedure by a dictionary production | generation system. 10 is a flowchart illustrating a detailed processing procedure example of the generation processing (step S901) illustrated in FIG. 9. It is a flowchart which shows the example of a detailed process sequence of the area | region group production | generation process (step S1003) shown in FIG. 10 is a flowchart illustrating a detailed processing procedure example of the acquisition processing (step S902) illustrated in FIG. 9. It is a flowchart which shows the detailed process sequence example of the 1st average minimum distance acquisition process (step S1201) shown in FIG. It is a flowchart which shows the detailed process sequence example of the 1st average minimum distance calculation process (step S1305) shown in FIG. It is a flowchart which shows the detailed process sequence example of the minimum distance accumulation | storage process (step S1405) shown in FIG. It is a flowchart which shows the detailed process sequence example of the 2nd average minimum distance acquisition process (step S1202) shown in FIG. 17 is a flowchart showing a detailed processing procedure example of the second average minimum distance calculation processing (step S1605) shown in FIG. 16. It is a flowchart which shows the detailed process sequence example of the minimum distance accumulation | storage process (step S1703) shown in FIG. 10 is a flowchart illustrating a detailed processing procedure example of the determination processing (step S903) illustrated in FIG. 9. 10 is a flowchart illustrating a detailed processing procedure example of the perturbation processing (step S905) illustrated in FIG. 9. 12 is a flowchart illustrating a detailed processing procedure example of a minimum distance calculation process (step S1605) according to the second embodiment. It is a block diagram which shows the system structural example of the content cloud system concerning Example 5. FIG. It is a block diagram which shows the example of an operation system of a dictionary production | generation system. It is explanatory drawing which shows the list of the information which an image management server manages. It is explanatory drawing which shows the information which the pattern management server for a comparison manages. It is explanatory drawing which shows the list of the information which a dictionary pattern management server manages. It is explanatory drawing which shows an example of the screen used for dictionary production | generation. It is explanatory drawing which shows the example of a display of a confirmation screen.

  FIG. 1 is an explanatory diagram showing an example of dictionary generation in the dictionary generation system according to the present invention. A dictionary generation system is a system that generates a dictionary. The system may be a single device or a group of devices connected to a network such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. A dictionary is information that stores an object in an image including a pattern such as an object or wallpaper. The dictionary generation system automates dictionary generation by registering objects in the dictionary as dictionary patterns.

  The dictionary generation uses the first image set 101 and the second image set 102 which are two types of image groups. The first image set 101 is an image set that is an object detection target, and the second image set 102 is an image set for excluding object non-detection targets from the first image set 101. A region group obtained from each image of the first image set 101 is an object candidate. A region group obtained from each image in the second image set 102 is a comparison target with the object candidate.

  The dictionary generation system determines whether or not the regions (object candidates) of the region group (object candidate group) obtained from the image 110 of the first image set 101 are close to each other using their image feature amounts. The rectangles in the area group 111 indicate object candidates. For example, the distance in the feature amount space between the object candidates A and B obtained from the image 110 in the first image set 101 is set as the first distance. The dictionary generation system can evaluate the similarity between the object candidates A and B based on the first distance.

  In addition, the dictionary generation system divides the object candidates obtained from the image 110 of the first image set 101 and the comparison target areas in the area group 122 obtained from the image 120 of the second image set 102 into those regions. It is determined whether or not the image is close by using the image feature amount. For example, the distance in the feature amount space between the object candidate A and the region C to be compared obtained from the image 120 of the second image set 102 is set as the second distance. The dictionary generation system can evaluate the similarity between the object candidates A and C based on the second distance.

  Then, the dictionary generation system uses the ratio between the first distance and the second distance, so that the object candidate A is an area similar to the object candidate B or a candidate similar to the area C to be compared. It is determined whether. The dictionary generation system narrows down object candidates based on the determination result, and obtains a narrowing result 130. The dictionary generation system generates a new area candidate by performing a perturbation process to be described later on the narrowing result 130, and also uses the area group 122 to generate the first distance again for the generated area candidate. And determine the second distance. The above-described determination process is performed again on the original region candidate plus a new region candidate. By repeating such processing until convergence, it is possible to improve the reliability of object candidates to be registered in the dictionary.

  In the dictionary generation system described above, there are four combinations of the first image set 101 and the second image set 102. That is, (1) the first image set 101 is an image set with a tag and the second image set 102 is an image set without a tag, and (2) the first image set 101 is an image set without a tag and the second image set. The image set 102 is also an untagged image set, (3) the first image set 101 is an untagged image set, the second image set 102 is a tagged image set, and (4) the first image set 101 is a tagged. In the image set with the second image set 102, there are four types of image sets with the tag.

  A tag is information attached to an image such as text. For example, an arbitrary character string such as “car” or “summer vacation trip”, a time stamp indicating the acquisition time of the image, and position information indicating the acquisition position of the image. That is, the tag is information indicating some attribute about the assigned image. The image set having a tag is, for example, an image set searched using a certain tag as a search key. Therefore, an image set having a tag is an image set having the same or similar common tag.

  In the case of (1) above, the dictionary generation system excludes patterns such as wallpaper included in the region group 122 to be compared in the second image set 102 from the object candidates of the first image set 101, The object candidates in the first image set 101 are narrowed down. Thereby, the reliability of dictionary registration can be improved.

  In the case of (2) above, both the first image set 101 and the second image set 102 are image sets without tags. An image set without a tag is an image group to which no tag is assigned, but may be treated as having no tag in the dictionary generation system even if a tag is assigned. In the case of (2) above, both the first image set 101 and the second image set 102 have no tag, so the first image set 101 may be used as the second image set 102. Further, the first image set 101 and the second image set 102 may be integrated. In the case of (2) above, the dictionary generation system excludes similar object candidates in the same image in the first image set 101. Thereby, a simple repeating pattern in an image can be excluded, and object candidates similar between images can be extracted.

  In the case of (3) above, for example, it is assumed that a tag indicating a repetitive pattern such as wallpaper or background is assigned to each image of the second image set 102. In this case, the dictionary generation system excludes candidates that partially match the repetitive pattern from the object candidates of the first image set 101 to which no tag has been assigned, and sets an area corresponding to an object such as a person or article as an object. Can be narrowed down as candidates.

  In the case of (4) above, for example, each image in the first image set 101 is given a specific tag X, and each image in the second image set 102 is a tag that characterizes the area to be excluded. It is assumed that Y is given. A tag Y may be assigned to each image in the first image set 101. In this case, the dictionary generation system can exclude the object candidates included in the image to which the tag Y is added from the object candidates included in the image to which the tag X is added, thereby improving the accuracy of narrowing down the object candidates. be able to.

  In the above dictionary generation, the second image set 102 is used, but the second image set 102 may not be used. In this case, the similarity between the object candidate, which is an area in the first image set 101, and the area in the second image set 102 is not evaluated. Can be planned.

  FIG. 2 is an explanatory diagram showing a specific example of dictionary generation in the dictionary generation system. FIG. 2 shows a dictionary generation example in the case of the combination (1) described above. A tag “car” is assigned to the first image set 101. The object candidate includes an area including a car image and an area not including a car image. The second image set 102 is an image set to which no tag is assigned. A region including various images to be compared is extracted from the second image set 102. The dictionary generation system excludes, from the object candidates, an area evaluated as similar between the object candidate and the comparison target area by the above-described determination process (the bold rectangle at the right end in FIG. 2). Thereby, the reliability of dictionary registration can be improved.

<Hardware configuration example>
FIG. 3 is a block diagram illustrating a hardware configuration example of the dictionary generation system. The dictionary generation system 300 includes a processor 301, a storage device 302, an input device 303, an output device 304, and a communication interface (communication IF 305). The processor 301, the storage device 302, the input device 303, the output device 304, and the communication IF 305 are connected by a bus. The processor 301 controls the dictionary generation system 300. The storage device 302 serves as a work area for the processor 301. The storage device 302 stores various programs and data. Examples of the storage device 302 include a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory.

  The input device 303 inputs data. Examples of the input device 303 include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 304 outputs data. Examples of the output device 304 include a display and a printer. The communication IF 305 is connected to a network and transmits / receives data. Examples according to the present invention will be described below.

Example 1
In the first embodiment, the case where (1) the first image set 101 is an image set with a tag and the second image set 102 is an image set without a tag will be described as an example.

<Functional configuration example>
FIG. 4 is a block diagram of a functional configuration example of the dictionary generation system 300 according to the first embodiment. 4, the dictionary generation system 300 includes a dictionary 400, a generation unit 401, an acquisition unit 402, a determination unit 403, a perturbation processing unit 404, a determination unit 405, a display unit 406, a registration unit 407, Have The dictionary 400 stores a dictionary pattern group. Specifically, the dictionary 400 realizes its function by, for example, the storage device 302 shown in FIG. Specifically, for example, the generation unit 401 to the registration unit 407 realize their functions by the processor 301 executing a program stored in the storage device 302 illustrated in FIG. 3.

  The generation unit 401 generates a region from the first image and the second image that are generation targets. The first image is, for example, the image 110 in the first image set 101 described above. The second image is, for example, the image 120 in the second image set 102 described above. Specifically, for example, the generation unit 401 performs multi-resolution processing on the generation target image and generates a multi-stage multi-resolution image. Then, the generation unit 401 performs grid-like scanning for each of the multi-resolution images using a plurality of quantized scanning windows having a plurality of aspect ratios. Thereby, the generation unit 401 generates a region from the first image and the second image. A specific example of generation by the generation unit 401 will be described with reference to FIG.

  The acquisition unit 402 includes a feature amount between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Obtain a first distance in space. The first region group is a region group 111 cut out from each image 110 of the first image set 101. Specifically, for example, grid-like scanning is performed on each image 110 of the first image set 101. This is a group of regions obtained by execution.

  The selection area is an area selected from the first area group and is an area to be registered as a dictionary pattern in the dictionary 400. The selection area corresponds to the object candidate A described above. The first area is an area different from the selected area in the first area group, and corresponds to, for example, the object candidate B. The selected area and the first area are areas in the first image set 101.

  Similarly, the acquisition unit 402 acquires the second distance in the feature amount space between the selected region and the second region selected from the second region group in the second image. . The second area group is an area group 122 cut out from each image of the second image set 102. Specifically, for example, grid-like scanning is performed on each image of the second image set 102. It is the area group obtained by this. Therefore, the region in the second region group corresponds to the region C described above.

  The acquisition unit 402 acquires a first minimum distance that is the minimum among a first distance group between a selected region in a plurality of first regions and a first region included in the image of interest. To do. Similarly, the acquisition unit 402 acquires the second minimum distance that is the minimum among the second distance group between the selected region and the second region included in the image of interest.

  In addition, since there are a plurality of images, there are many first minimum distances for each selected region. The acquisition unit 402 acquires the first average minimum distance for each selected region by averaging the first minimum distance for the image. Similarly, the acquisition unit 402 acquires the second average minimum distance for each selected region by averaging the second minimum distance with respect to the image.

  The determination unit 403 determines a specific selection region to be subjected to perturbation processing from a plurality of selection regions based on the ratio between the first average minimum distance and the second average minimum distance. Specifically, for example, the determination unit sorts the ratio between the first average minimum distance and the second average minimum distance in each selection region in ascending order. Then, the determination unit 403 determines, for example, a selection region corresponding to the ratio up to the top M (M is an integer of 1 or more) as a specific selection region to be subjected to perturbation processing. Thereby, it is possible to narrow down selection areas suitable for registration as a dictionary pattern.

  The perturbation processing unit 404 generates a new region by executing perturbation processing for perturbing the selected region based on the ratio between the first distance and the second distance. Specifically, for example, the perturbation processing unit 404 performs the perturbation process for the specific selection region determined by the determination unit 403. The perturbation process is a process for generating a new area by shifting the position of the selected area. For example, the perturbation processing unit 404 performs perturbation processing on the object candidates that are the narrowing result 130 shown in FIG.

  The determination unit 405 determines whether or not the selected area should be a dictionary pattern based on the ratio between the first distance and the second distance. The ratio of the first distance to the second distance is a value obtained by dividing the first distance by the second distance. The smaller the ratio value, the more suitable the selected area is as a dictionary pattern.

  For example, if the first distance is small, the selected area and the first area are similar in the first image set 101. However, whether or not both areas correspond to the target dictionary pattern is determined by the second distance. For example, when the second image set 102 is an image set that is not related to the first image set 101, the fact that the second distance is small is evaluated that the selected area is similar to the second area. . Therefore, if the second distance is small, the ratio of the distances for the selected region is large, which is not suitable as a dictionary pattern.

  When the perturbation process is executed, the determination unit 405 determines whether or not the selected region should be a dictionary pattern based on the processing result by the perturbation processing unit 404. Since the selected region is a discrete digital image, a new region is not generated due to minute fluctuations by repeatedly executing the perturbation process. Therefore, if no new area is generated in the perturbation process, the determination unit 405 determines that the area has converged and can determine that the remaining selection area should be registered as a dictionary pattern.

  The display unit 406 displays the selection area determined to be a dictionary pattern by the determination unit 405 so that it can be specified whether or not it can be registered in the dictionary 400. Specifically, for example, the display unit 406 displays on the display screen of the output device 304 a selection area determined to be a dictionary pattern. As a result, the user can specify whether or not to register using the input device 303.

  The registration unit 407 registers the selected area in the dictionary 400 when the determination unit 405 determines that the dictionary pattern should be used. When the attribute information is given to the selection area, the registration unit 407 registers the selection area in the dictionary 400 in association with the attribute information. The attribute information is the tag described above. Thereby, when searching the dictionary 400 using attribute information, the dictionary pattern which is a desired object can be extracted. In addition, the registration unit 407 may register the selection area designated on the display unit 406 to be registered by the user using the input device 303.

<Example of area generation>
FIG. 5 is an explanatory diagram showing an example of region generation from the first image set 101 and the second image set 102. The dictionary generation system 300 generates a plurality of quantized scanning windows having different aspect ratios. In the example of FIG. 5, five types of aspect ratio scanning windows w1 to w5 are generated.

  Further, the dictionary generation system 300 executes multi-resolution processing for each of the images 110 and 120. The dictionary generation system 300 generates, for example, a multi-resolution image group obtained by reducing the resolution of the images 110 and 120 by 1/2 by multi-resolution processing. In the example of FIG. 5, four-stage multi-resolution images 110, 110a, 110b, and 110c are generated for an image 110.

  The dictionary generation system 300 performs a grid-like scan on a multi-resolution image obtained by multi-resolution processing using a scan window obtained by quantization of an aspect ratio. In the example of FIG. 5, the dictionary generation system 300 scans a four-stage multi-resolution image group in a grid pattern using five types of scanning windows w. Thereby, a region is extracted from the image 110. The extracted area is defined by the aspect ratio of the scanning window to be applied, the resolution of the applied multi-resolution image, and the scanning position of the scanning window by grid scanning.

<Example of region generation by perturbation processing>
6-8 is explanatory drawing which shows the example of area | region generation by a perturbation process. The perturbation process is a process for generating a region whose position is shifted with respect to the object candidate that becomes the narrowing-down result 130 shown in FIG. 6 to 8, a dotted rectangle is a region that is a candidate object in the narrowing result 130, and a white solid rectangle is a region that has been changed by the perturbation process. FIG. 6 shows a region in which an object candidate in the narrowing-down result 130 is changed vertically and horizontally on the image 110 including the object candidate.

  FIG. 7 shows a region in which a certain object candidate in the narrowing-down result 130 is changed to the upper right, lower right, upper left, and lower left on the image 110 including the object candidate. FIG. 8 shows a region in which an object candidate in the narrowing-down result 130 is changed by enlargement / reduction on the image 110 including the object candidate.

  An example of the fluctuation amount due to the perturbation process will be described. The horizontal step width of the grid scan is gx, the vertical step width is gy, and the number of repetitions of the perturbation process is q. The fluctuation amount dx in the horizontal direction, the fluctuation amount dy in the vertical direction, and the enlargement ratio dz in the q-th perturbation process are as follows. The reduction ratio is 1 / dz.

dx = gx / 2 ^q (1)
dy = gy / 2 ^q (2)
dz = 2 ^ (1/2 ^q ) (3)

  According to the variation amount, the enlargement rate, and the reduction rate according to the above formulas (1) to (3), the variation amount and the enlargement rate become smaller and the reduction rate as the number of repetitions q of the perturbation process increases. Becomes larger. That is, the blurring of the region due to the perturbation process is suppressed, and the region is easily converged to the original region. Equations (1) to (3) are examples, and as the number of repetitions q of the perturbation process increases, the variation amount and the enlargement rate become smaller than before the increase, and the reduction rate becomes larger. Other expressions may be used. Also, a fixed amount of variation may be used regardless of the number of repetitions q of the perturbation process. In this case, the calculations of the expressions (1) to (3) are not necessary, so that the area generation by the perturbation process can be speeded up.

<Dictionary generation process>
FIG. 9 is a flowchart illustrating an example of a dictionary generation processing procedure by the dictionary generation system 300. The dictionary generation system 300 includes a generation process (step S901) by the generation unit 401 shown in FIG. The processes are executed in the order (step S904).

  Thereafter, the dictionary generation system 300 performs a convergence determination process by the determination unit (step S904). In the convergence determination process (step S904), since the target image is a discrete digital image, a new region is not generated due to slight variations by repeatedly executing the image. Therefore, if no new region candidate is generated in the perturbation process (step S904), the dictionary generation system 300 determines that the convergence has been completed (step S905: Yes), and proceeds to the display process (step S906). .

  In addition, an upper limit of the number of repetitions may be set for efficiency of calculation, and it may be determined that convergence has been reached when the upper limit is reached. In addition, when it determines with not having converged (step S905: No), it transfers to an acquisition process (step S902). That is, steps S902 to S904 are repeated until convergence. On the other hand, when it has converged (step S905: Yes), display processing (step S906) by the display unit and registration processing (step S907) by the registration unit are executed. As a result, the series of processes is completed.

<Generation process>
FIG. 10 is a flowchart illustrating a detailed processing procedure example of the generation processing (step S901) illustrated in FIG. Here, the first image set 101 is described as an example, but the first image set 101 is also applied to the second image set 102.

  The dictionary generation system 300 determines whether there is an unselected image in the first image set 101 (step S1001). When there is an unselected image (step S1001: Yes), the dictionary generation system 300 selects one unselected image (step S1002). Next, the dictionary generation system 300 executes region group generation processing for the selected image (step S1003). The details of the area group generation process (step S1003) will be described with reference to FIG. 11, but a plurality of areas are extracted from the selected image by the area group generation process (step S1003).

  Thereafter, the dictionary generation system 300 determines whether or not there is an unselected area in the area group extracted from the selected image (step S1004). When there is an unselected area (step S1004: Yes), the dictionary generation system 300 selects one unselected area (step S1005), and extracts the image feature amount of the selected area (step S1006). A detailed description of the image feature extraction method is described in Non-Patent Document 1 described above. By using the extracted image feature amount, the similarity between regions in the same image set and the similarity between regions between different image sets can be evaluated.

  After extracting the image feature amount (step S1006), the process returns to step S1004. In step S1004, when there is no unselected area (step S1004: No), the process returns to step S1001. In step S1001, when there is no unselected image (step S1001: No), the generation process is terminated (step S901), and the process proceeds to the acquisition process (step S902) in FIG.

<Region group generation processing>
FIG. 11 is a flowchart showing a detailed processing procedure example of the region group generation processing (step S1003) shown in FIG. First, as illustrated in FIG. 5, the dictionary generation system 300 performs multi-resolution processing on the selected image selected in step S1002 for each quantized aspect ratio (step S1101). Next, the dictionary generation system 300 determines whether there is an unselected aspect ratio (step S1102). If there is an unselected aspect ratio (step S1102: Yes), the dictionary generation system 300 selects an unselected aspect ratio (step S1103), and determines whether there is an unselected multi-resolution image (step S1103). S1104).

  When there is an unselected multiresolution (step S1104: Yes), the dictionary generation system 300 selects an unselected multiresolution image (step S1105). Then, the dictionary generation system 300 generates a region group having the same shape and size as the scanning window by scanning the selected multi-resolution image in a grid pattern with the scanning window having the selected aspect ratio (step S1106). Thereafter, returning to step S1104, the dictionary generation system 300 determines whether there is an unselected multi-resolution image (step S1104). If there is no unselected multi-resolution image (step S1104: No), the process returns to step S1102, and the dictionary generation system 300 determines whether there is an unselected aspect ratio (step S1102). If there is no unselected aspect ratio (step S1102: No), the series of processing is terminated, and the process proceeds to step S1004 in FIG.

<Acquisition processing>
FIG. 12 is a flowchart illustrating a detailed processing procedure example of the acquisition process (step S902) illustrated in FIG. The acquisition process (step S901) is a process of acquiring a minimum distance based on an image feature amount between regions and averaging these to acquire an average minimum distance. The dictionary generation system 300 first executes a first average minimum distance acquisition process (step S1201). In the first average minimum distance acquisition process (step S1201), a minimum distance between regions extracted from images in the first image set 101 is acquired, and these are averaged to acquire a first average minimum distance. It is processing to do. Details of the first average minimum distance acquisition process (step S1201) will be described with reference to FIG.

  Next, the dictionary generation system 300 executes a second average minimum distance acquisition process (step S1202). In the second average minimum distance acquisition process (step S1202), the minimum distance between the region extracted from the image in the first image set 101 and the region extracted from the image in the second image set 102 is displayed. Is obtained, and these are averaged to obtain the second average minimum distance. Details of the second average minimum distance acquisition process (step S1202) will be described with reference to FIG. Thereby, when the average minimum distance acquisition process (step S1201) is completed, the process proceeds to the determination process (step S903).

  FIG. 13 is a flowchart showing a detailed processing procedure example of the first average minimum distance acquisition processing (step S1201) shown in FIG. The dictionary generation system 300 first initializes variables (step S1301). Here, the index i of the image group in the first image set 101 is set to i = 1. An image with index i in the first image set 101 is set as an image Ai. i is an integer taking 1 ≦ i ≦ Na. Na is the number of images in the first image set 101.

  Also, let j be the index of the area group extracted from the image Ai. The j-th area extracted from the image Ai is assumed to be aij. j is an integer taking 1 ≦ j ≦ nai. nai is the number of areas in the image Ai.

  Next, the dictionary generation system 300 determines whether i> Na is satisfied (step S1302). That is, the dictionary generation system 300 determines whether all the images in the first image set 101 have been processed. If i> Na is not satisfied (step S1302: No), j = 1 is set (step S1303), and the dictionary generation system 300 determines whether j> nai is satisfied (step S1304). That is, the dictionary generation system 300 determines whether or not all the areas aij in the image Ai have been processed.

  If j> nai is not satisfied (step S1304: NO), the dictionary generation system 300 extracts the first feature amount that is the image feature amount from the area aij (step S1305). Thereafter, the dictionary generation system 300 executes a first average minimum distance calculation process (step S1306). In the first average minimum distance calculation process (step S1306), a second feature amount that is an image feature amount is extracted from a region akl in an image Ak different from the image Ai in the first image set 101, and a region aij is extracted. And a first average minimum distance that is an average value of the minimum distances between the area akl and the area akl. The index k is an integer taking 1 ≦ k ≦ Na, and k ≠ i. The index l is an integer that takes 1 ≦ l ≦ nak. nak is the number of regions in the image Ak. Details of the first average minimum distance calculation process (step S1306) will be described with reference to FIG.

  Thereafter, the dictionary generation system 300 increments the index j (step S1307) and returns to step S1304. In step S1303, when j> nai (step S1304: Yes), the dictionary generation system 300 increments i (step S1308), and returns to step S1302. In step S1302, if i> Na (step S1302: Yes), the process proceeds to the second average minimum distance acquisition process (step S1202) in FIG. Thereby, the first average minimum distance acquisition process (step S1201) ends.

  FIG. 14 is a flowchart showing a detailed processing procedure example of the first average minimum distance calculation processing (step S1306) shown in FIG. The dictionary generation system 300 first initializes variables (step S1401). Here, k = 1 and l = 1. Further, the dictionary generation system 300 sets the variable s to s = 0. The variable s is a cumulative value of the minimum distance from each area akl for a certain area aij.

  Next, the dictionary generation system 300 determines whether or not k = i (step S1402). That is, the dictionary generation system 300 is for excluding the region akl where k = i from the processing target.

  When k = i (step S1402: Yes), k is incremented (step S1403), and the process proceeds to step S1404. On the other hand, if k ≠ i (step S1402: No), the process proceeds to step S1404.

  Next, the dictionary generation system 300 determines whether or not k> Na (step S1404). That is, the dictionary generation system 300 determines whether all the images in the first image set 101 have been processed. If k> Na is not satisfied (step S1404: No), the dictionary generation system 300 executes minimum distance accumulation processing (step S1405). The minimum distance accumulation process (step S1405) is a process of accumulating the minimum distance between each area akl and a certain area aij. In the minimum distance accumulation process (step S1405), the accumulated value (variable s) of the minimum distance from each area akl with respect to a certain area aij is obtained. Details of the minimum distance accumulation process (step S1405) will be described with reference to FIG.

  Thereafter, the dictionary generation system 300 increments k (step S1406), and proceeds to step S1402. In step S1404, if k> Na (step S1404: Yes), the dictionary generation system 300 calculates the first average minimum distance (step S1407), and proceeds to step S1307 in FIG. The first average minimum distance Dij is calculated by the following equation (4). Thereby, the first average minimum distance calculation process (step S1306) is completed.

Dij = s / (Na-1) (4)

  FIG. 15 is a flowchart illustrating a detailed processing procedure example of the minimum distance accumulation processing (step S1405) illustrated in FIG. The dictionary generation system 300 sets t = dmax and l = 1 (step S1501). t is a variable used for calculating the distance between the regions, and dmax is the maximum value of the distance. Next, the dictionary generation system 300 determines whether or not l> nak (step S1502). nak is the number of regions in the image Ak. That is, the dictionary generation system 300 determines whether or not all the regions akl in the image Ak different from the image Ai in the first image set 101 have been processed.

  If l> nak is not satisfied (step S1502: NO), the dictionary generation system 300 extracts the second feature amount that is the image feature amount from the region akl (step S1503). Then, the dictionary generation system 300 calculates the distance d between the area aij and the area akl (step S1601). For the calculation of the distance d, the first feature value of the region aij and the second feature value of the region akl are used. The following formula (5) is a formula showing an example of calculating the distance d.

Wherein (5), v _r is the r th component in the vector indicating the first feature amount of a region aij, u _r is the r th component in the vector showing the second characteristic amount region akl. R is the number of components. Then, the dictionary generation system 300 determines whether or not the calculated distance d is d <t (step S1505). Here, t is a value added to the variable s in step S1508, and an initial value of t is a distance evaluation range in the feature amount space, that is, a maximum distance dmax.

  If d <t (step S1505: Yes), the dictionary generation system 300 sets t = d (step S1506), and proceeds to step S1507. On the other hand, if d <t is not satisfied (step S1505: NO), the process proceeds to step S1507. That is, the initial value of t is t = dmax, but every time d <t, the value of t decreases.

  In step S1507, the dictionary generation system 300 increments l (step S1507) and returns to step S1502. In step S1502, if l> nak (step S1502: Yes), the dictionary generation system 300 updates the variable s (step S1508), and proceeds to step S1406. Therefore, in step S1508, the minimum value of the distance d when the region akl is tried from l = 1 to l = nak, that is, the minimum distance is added to the variable s as t.

  FIG. 16 is a flowchart showing a detailed processing procedure example of the second average minimum distance acquisition processing (step S1202) shown in FIG. First, the dictionary generation system 300 initializes variables (step S1601). Here, i = 1.

  Next, the dictionary generation system 300 determines whether i> Na is satisfied (step S1602). That is, the dictionary generation system 300 determines whether all the images in the first image set 101 have been processed. If i> Na is not satisfied (step S1602: NO), j = 1 is set (step S1603), and the dictionary generation system 300 determines whether j> nai is satisfied (step S1604). That is, the dictionary generation system 300 determines whether or not all the areas aij in the image Ai have been processed.

  If j> nai is not satisfied (step S1604: NO), the dictionary generation system 300 extracts the first feature amount that is the image feature amount from the area aij (step S1605). Thereafter, the dictionary generation system 300 executes a second average minimum distance calculation process (step S1606). In the second average minimum distance calculation process (step S1606), the second feature quantity that is the image feature quantity is extracted from the area bkl in the image Bk in the second image set 102, and the area aij and the area akl are extracted. This is a process of calculating a second average minimum distance that is an average value of the minimum distances. Details of the second average minimum distance calculation process (step S1606) will be described with reference to FIG.

  Thereafter, the dictionary generation system 300 increments the index j (step S1607) and returns to step S1604. If j> nai is satisfied in step S1604 (step S1604: YES), the dictionary generation system 300 increments i (step S1608) and returns to step S1602. In step S1602, if i> Na (step S1602: Yes), the process proceeds to the determination process of FIG. 9 (step S903). Thereby, the second average minimum distance acquisition process (step S1202) ends.

  FIG. 17 is a flowchart showing a detailed processing procedure example of the second average minimum distance calculation processing (step S1606) shown in FIG. The dictionary generation system 300 first initializes variables (step S1701). Here, k = 1 and l = 1. In addition, the variable s is set to s = 0. The variable s is a cumulative value of the minimum value of the distance from each area bkl to a certain area aij. Details of the calculation will be described with reference to FIG. The index k is an integer that takes 1 ≦ k ≦ Nb. Nb is the number of images in the second image set 102. The index l is an integer that takes 1 ≦ j ≦ nbk. nbk is the number of regions in the image Bk.

  Next, the dictionary generation system 300 determines whether or not k> Nb (step S1702). That is, the dictionary generation system 300 determines whether all the images in the second image set 102 have been processed. If k> Nb is not satisfied (step S1702: NO), the dictionary generation system 300 executes a minimum distance accumulation process (step S1703). The minimum distance accumulation process (step S1703) is a process of accumulating the minimum distance between each area bij and a certain area aij, similar to S1405. In the minimum distance accumulation process (step S1703), the accumulated value (variable s) of the minimum distance from each area bkl to a certain area aij is obtained. Details of the minimum distance accumulation process (step S1703) will be described with reference to FIG.

  Thereafter, the dictionary generation system 300 increments k (step S1704), and proceeds to step S1702. In step S1702, if k> Nb (step S1702: Yes), the dictionary generation system 300 calculates the second average minimum distance (step S1705), and proceeds to step S1607 in FIG. The second average minimum distance Eij is calculated by the following equation (6). Thereby, the second average minimum distance calculation process (step S1606) ends.

Eij = s / Nb (6)

  FIG. 18 is a flowchart showing a detailed processing procedure example of the minimum distance accumulation processing (step S1703) shown in FIG. The dictionary generation system 300 sets t = dmax and l = 1 (step S1801). t is a variable used for calculating the distance between the regions, and dmax is the maximum value of the distance. Next, the dictionary generation system 300 determines whether or not l> nbk (step S1802). nbk is the number of regions in the image Bk. That is, the dictionary generation system 300 determines whether or not all the regions bkl in the image Bk different from the image Ai in the first image set 101 have been processed.

  If l> nbk is not satisfied (step S1802: NO), the dictionary generation system 300 extracts the third feature amount that is the image feature amount from the region bkl (step S1803). Thereafter, the dictionary generation system 300 calculates a distance e between the area aij and the area bkl (step S1804). For the calculation of the distance e, the first feature quantity of the area aij and the third feature quantity of the area bkl are used. The following formula (7) is a formula showing an example of calculating the distance e.

Wherein (7), v _r is the r th component in the vector indicating the first feature amount of a region aij, u _r is the r th component in the vector showing a third feature amount of a region bkl. R is the number of components. Then, the dictionary generation system 300 determines whether or not the calculated distance e is e <t (step S1805). Here, t is a value added to the variable s in step S1808, and an initial value of t is a distance evaluation range in the feature amount space, that is, a maximum distance dmax.

  If e <t (step S1805: YES), the dictionary generation system 300 sets t = e (step S1806), and proceeds to step S1807. On the other hand, if e <t is not satisfied (step S1805: NO), the process proceeds to step S1807. That is, the initial value of t is t = dmax, but every time e <t, the value of t decreases.

  In step S1807, the dictionary generation system 300 increments l (step S1807) and returns to step S1802. In step S1802, if l> nbk (step S1802: Yes), the dictionary generation system 300 updates the variable s (step S1808), and proceeds to step S1704. Therefore, in step S1808, the minimum value of the distance e when trying from 1 = 1 to l = nbk for the region bkl is added to the variable s as t.

<Decision process>
FIG. 19 is a flowchart illustrating a detailed processing procedure example of the determination processing (step S903) illustrated in FIG. In the determination process (step S903), the dictionary generation system 300 determines an area to be determined in the convergence determination process (step S904). The dictionary generation system 300 first initializes variables (step S1901). Here, i = 1. Next, the dictionary generation system 300 determines whether i> Na is satisfied (step S1902). That is, the dictionary generation system 300 determines whether all the images in the first image set 101 have been processed. If i> Na is not satisfied (step S1902: NO), j = 1 is set (step S1903), and the dictionary generation system 300 determines whether j> nai is satisfied (step S1904). That is, the dictionary generation system 300 determines whether or not all the areas aij in the image Ai have been processed.

  If j> nai is not satisfied (step S1904: NO), the dictionary generation system 300 calculates the normalized average minimum distance Fij by dividing the first average minimum distance Dij by the second average minimum distance Eij. (Step S1905). The normalized average minimum distance Fij is a numerical value serving as an index as to whether or not the region aij should be registered in the dictionary 400. A smaller value of the normalized average minimum distance Fij indicates that registration in the dictionary 400 is more appropriate.

  After calculating the normalized average minimum distance Fij, the dictionary generation system 300 increments j (step S1906) and returns to step S1903. In step S2003, if j> nai (step S1904: Yes), i is incremented (step S1907), and the process returns to step S1902. Then, in step S1902, if i> Na (step S1902: Yes), the dictionary generation system 300 sorts the normalized average minimum distance Fij in ascending order of the values (step S1908).

  Then, the dictionary generation system 300 acquires a region aij in which the value of the normalized average minimum distance Fij is the upper Mth. Thereby, the area aij is narrowed down to the top M. Therefore, the dictionary generation system 300 updates the index (i, j, nai) for the acquired M areas aij (step S1909). For example, M = 10, and the top ten regions aij are {a12, a24, a15, a26, a61, a31, a47, a63, a48, a69}.

  In this case, the content of {a11, a12} is updated to the content of {a12, a15}. The contents of {a21, a22} are updated to the contents of {a24, a26}. {A31} remains as it is. The contents of {a41, a42} are updated to the contents of {a47, a48}. The contents of {a61, a62, a63} are updated to the contents of {a61, a63, a69}. Further, the number of areas nai in the image Ai is updated to nai = 0, na2 = 2, na3 = 1, na4 = 2, na6 = 3, and otherwise. Thereby, the determination process ends (step S903), and the process proceeds to the convergence determination process (step S904). In the convergence determination process (step S904), the convergence determination is executed for each of the M areas aij.

<Perturbation processing>
FIG. 20 is a flowchart showing a detailed processing procedure example of the perturbation processing (step S905) shown in FIG. First, the dictionary generation system 300 increments the number of repetitions q of the perturbation process (initial value is q = 0) (step S2001), and changes the horizontal fluctuation amount dx, the vertical fluctuation amount dy, the enlargement ratio dz, and the reduction ratio. 1 / dz is calculated (step S2002). Next, the dictionary generation system 300 determines whether there is an unselected area aij (step S2003). When there is an unselected area (step S2003: Yes), the dictionary generation system 300 selects the unselected area aij (step S2004), and the horizontal direction fluctuation amount dx, the vertical direction fluctuation amount dy, the enlargement ratio dz, and the reduction. A new area based on the selected area aij is generated using the rate 1 / dz (step S2005), and the process returns to step S2003.

  In step S2003, when there is no unselected candidate (step S2003: No), the dictionary generation system 300 updates the number of areas nai included in each image because the area is added in step S2005 (step S2006). For example, in the above normalization example, the dictionary generation system 300 includes the contents aij of the array storing the area candidates in each image, so that the additional area is included after being narrowed down to the top M. The number of area candidates nai is updated. Thereby, the perturbation process (step S904) is terminated, and the process proceeds to the convergence determination process (step S905).

  As described above, according to the first embodiment, by removing patterns such as wallpaper included in the region group to be compared in the second image set 102 from the object candidates of the first image set 101, The object candidates in the image set 101 can be narrowed down. Thereby, the reliability of dictionary registration can be improved. In addition, an element in the second image set 102 may be mixed with an image to be given an attribute of interest, that is, an image to be included in the first image set 101. If the number of elements of the second image set 102 is sufficient, in the process of calculating the average minimum value, it is possible to sufficiently reduce the effect due to such inadequate attribute assignment.

(Example 2)
Next, Example 2 will be described. The second embodiment is an example in which (2) the first image set 101 described above is an untagged image set and the second image set 102 is also an untagged image set. In this case, the second image set 102 may be replaced with the first image set 101 in the flowcharts of FIGS. In the case of the second embodiment, the second average minimum distance calculation process (step S1605) shown in FIGS. 17 and 18 differs from the first embodiment and is replaced with FIG. In the second embodiment, the minimum distance from other regions of the image including the region of interest is adopted as an equivalent to Eij of the first embodiment. Hereinafter, the second minimum distance calculation process (step S1605) according to the second embodiment will be described.

  FIG. 21 is a flowchart of a detailed process procedure example of the minimum distance calculation process (step S1605) according to the second embodiment. After extracting the first feature quantity that is the image feature quantity of the area aij in step S1604 in FIG. 16, the dictionary generation system 300 first initializes variables (step S2101). Here, k = 1. Next, the dictionary generation system 300 determines whether or not k = j (step S2102). That is, the dictionary generation system 300 is for excluding the region aik where k = j from the processing target.

  If k = j (step S2102: YES), k is incremented (step S2103), and the process proceeds to step S2104. On the other hand, if k ≠ j (step S2102: No), the process proceeds to step S2104.

  Next, the dictionary generation system 300 determines whether k> nai is satisfied (step S2104). That is, the dictionary generation system 300 determines whether or not all areas in the image have been processed. If k> nai is not satisfied (step S2104: NO), the dictionary generation system 300 sets t = dmax (step S2105). t is a variable used for calculating the distance between the regions, and dmax is the maximum value of the distance. Next, the dictionary generation system 300 extracts a third feature amount that is an image feature amount from the region aik (step S2106). Then, the dictionary generation system 300 calculates the distance e between the area aij and the area aik (step S2107). The above formula (7) is used to calculate the distance e.

  If e <t (step S2108: YES), the dictionary generation system 300 sets t = e (step S2109) and proceeds to step S2110. On the other hand, if e <t is not satisfied (step S2108: NO), the process proceeds to step S2110. That is, the initial value of t is t = dmax, but every time e <t, the value of t decreases.

  In step S2110, the dictionary generation system 300 increments k (step S2110), and proceeds to step S2102. In step S2104, if k> Na (step S2104: Yes), the second minimum distance Eij is determined to be t (step S2111), and the process proceeds to step S1606 in FIG. Therefore, the minimum value t of the distance e when the region aik is tried from k = 1 to k = nai is determined as the minimum distance Eij with respect to the region aij. As a result, the minimum distance calculation process (step S1706) ends, and the process proceeds to step S1606 in FIG.

  According to the second embodiment, the dictionary generation system 300 excludes similar object candidates in the same image in the first image set 101. Thereby, a simple repeating pattern in an image can be excluded, and object candidates similar between images can be extracted.

Example 3
Next, Example 3 will be described. In the third embodiment, (3) the first image set 101 is an image set with no tag and the second image set 102 is an image set with a tag. In the third embodiment, the processing contents are the same as those in the flowcharts of FIGS. 9 to 20 in the first embodiment except that the first image set 101 has no tag and the second image set 102 has a tag. is there. Thus, according to the third embodiment, for example, by using a tag corresponding to a simple repeating pattern such as a background as a tag for designating the second image set, the first image to which no tag is attached. Candidates that partially match the repetitive pattern are excluded from the object candidates of the set 101, and areas corresponding to objects such as people and articles can be narrowed down as object candidates.

Example 4
Next, Example 4 will be described. In the fourth embodiment, (4) the first image set 101 is an image set with a tag and the second image set 102 is also an example of an image set with a tag. In the fourth embodiment, the tag X given to the first image set 101 and the tag Y given to the second image set 102 are different tags. The fourth embodiment is the same as the flowchart of FIGS. 9 to 20 in the first embodiment except that tags are assigned to the first image set 101 and the second image set 102. As described above, the object candidates included in the image to which the tag Y is added can be excluded from the object candidates included in the image to which the tag X is added, and the refinement accuracy of the object candidates can be improved.

(Example 5)
Next, Example 5 will be described. The fifth embodiment is an example in which the dictionary generation system 300 according to the first to fourth embodiments is incorporated into a content cloud system.

  FIG. 22 is a block diagram of a system configuration example of the content cloud system according to the fifth embodiment. The content cloud system 2200 includes an Extract Transform Load (ETL) module 2203, a content storage 2204, a search engine 2205, a metadata server 2206, and a multimedia server 2207. The content cloud system 2200 operates on a computer (eg, see FIG. 3) including one or more processors 301 and a storage device 302, and is composed of various modules. In addition, each module may be executed by an independent computer. In this case, each storage is connected to the module via a network or the like, and is realized by distributed processing in which data communication is performed via them.

  The application program 2208 sends a request to the content cloud system 2200 via a network or the like, and the content cloud system 2200 transmits information corresponding to the request to the application program 2208.

  The content cloud system 2200 receives data 2201 in an arbitrary format such as video data, image data, document data, and audio data as an input. The data 2201 is, for example, a graphic trademark and its publicity document, a website image and HTML document, closed caption or video data with audio, etc., and may be structured data or unstructured data. Data input to the content cloud system 2200 is temporarily stored in the storage 2202.

  The ETL module 2203 monitors the storage 2202, and when data 2201 is stored in the storage 2202, information (metadata) obtained from the file system is archived and stored in the content storage 2204.

  The content storage 2204 stores the information extracted by the ETL module 2203 and the pre-processing data 2201 temporarily stored in the storage 2202.

  When there is a request from the application program 2208, the search engine 2205 performs a text search based on the index created by the ETL module 2203, for example, if it is a text search, and transmits the search result to the application program 2208. A known technique can be applied to the algorithm of the search engine 2205. The search engine 2205 can include a module for searching not only text but also data such as images and sounds.

  The metadata server 2206 manages metadata stored in an RDB (Relational DataBase). For example, it is assumed that the file name, data registration date, original data type, metadata text information, etc. extracted by the ETL module 2203 are registered in the RDB. When there is a request from the application program 2208, the metadata server 2206 transmits information in the RDB to the application program 2208 in accordance with the request.

  The multimedia server 2207 operates the information extraction processing module 2209 according to the data archived in the content storage 2204, and extracts metadata related to the contents of the data.

  The information extraction processing module 2209 includes, for example, a text index module, an image recognition module, and the like. Examples of metadata include time, an N-gram index, an image recognition result (object name, region coordinates in the image), an image feature amount and related words, a speech recognition result, and the like. As the information extraction processing module 2209, any program for extracting some information (metadata) can be used, and a known technique can be adopted. Therefore, the description of the information extraction processing module 2209 is omitted here.

  Metadata extracted from each media data is stored in a graph DB 2311 that is associated with each other and structured in a graph format. As an example of association mapping, the correspondence relationship of the original audio file, image data, related words, and the like can be expressed in a network format with respect to the speech recognition result “apple” stored in the content storage 2204. When there is a request from the application program 2208, the multimedia server 2207 transmits meta information corresponding to the request to the application program 2208. For example, when there is a request of “apple”, meta information associated on a network graph such as an image including an apple, an average market price, and an artist's song name is provided based on the constructed graph structure.

  The information extraction processing module 2209 executes object detection processing from an image. In the dictionary 400, a dictionary pattern for object detection generated by the dictionary generation system 300 shown in the first to fourth embodiments is registered. For each dictionary pattern in the dictionary 400, metadata indicating what kind of thing each dictionary pattern is is defined along with the image feature amount of each dictionary pattern.

  When the information extraction processing module 2209 detects a desired object from a given image by collation with the dictionary 400, the graph DB 2310 displays information related to detection such as the detection position and the size of the region and metadata of the collated dictionary pattern. Register with. When a plurality of information extraction processing modules 2209 are incorporated in the multimedia server 2207, the resources of one computer may be shared, or an independent computer may be used for each module.

  In this way, by using the object detection dictionary pattern generated by the dictionary generation system 300 shown in the first to fourth embodiments, the content cloud system 2200 can use metadata that can be commonly used for each media data. Can be generated. Therefore, information can be integrated across media, and information with higher added value can be provided to the user.

(Example 6)
Next, Example 6 will be described. The sixth embodiment is an example of an operation method of the dictionary generation system 300 in the first to fifth embodiments.

  FIG. 23 is a block diagram illustrating an example of an operation method of the dictionary generation system 300. In the dictionary generation system 300, the image management server 2310, the word management server 2320, the dictionary generation service 2330, the comparison pattern management server 2340, the dictionary pattern management server 2350, and the terminal device 2360 are coupled to each other via the network 2300. System.

  The image management server 2310 manages image data. The word management server 2320 manages the linguistic information given to the image data. The dictionary generation service 2330 generates a dictionary pattern. The dictionary generation service 2330 is a computer having the generation unit 401 to the determination unit 405 shown in FIG.

  The comparison pattern management server 2340 manages the image feature amount of the area obtained from the second image set 102 to be compared with the area obtained from the first image set 101 when the dictionary pattern is generated. The dictionary pattern management server 2350 manages dictionary patterns. The dictionary pattern management server 2350 is a computer having the registration unit 407 shown in FIG. The terminal device 2360 issues various requests to the server, confirms the generated dictionary pattern, and the like. The terminal device 2360 is a computer having the display unit 406 shown in FIG.

  FIG. 24 is an explanatory diagram showing a list of information managed by the image management server 2310. An item 2401 “image” is image data expressed in a byte string. The image data itself is often managed separately by a file server or the like. In this case, information necessary for acquiring an image such as a URL in which an image file exists is stored as information corresponding to the item 2401. An item 2402 “keyword” is a set of words related to the image data. Each word may be managed as a character string. In the sixth embodiment, each word is managed by the word management server 2420 in order to save memory consumption and improve data processing efficiency, and the item 2402 includes a management number of each word on the word management server 2420. Stores a column of integer values. Hereinafter, the item 2402 is referred to as “word”.

  On the other hand, “used_keyword” of the item 2403 is a word ID of a word used as an attribute when the image is used by the dictionary pattern generation service 2430. The word ID is information prepared in order to avoid using the same image repeatedly for dictionary generation. As described above, the image management server 2310 manages the image data defined by the items 2401 to 2403.

  FIG. 25 is an explanatory diagram showing information managed by the comparison pattern management server 2340. The item “image” of the item 2501 is a management number on the image management server 2310 of the original image expressed by an integer value. The item 2502 “rect” is a two-dimensional coordinate value representing a rectangular area of the comparison pattern, and is a four-dimensional integer array. An item 2503 “feature” is an image feature amount of a rectangular area of a comparison pattern defined by an item 2502 “rect”. Thus, the comparison pattern management server 2340 manages the comparison patterns defined by the items 2501 to 2503.

  FIG. 26 is an explanatory diagram showing a list of information managed by the dictionary pattern management server 2350. An item 2601 “image” is a management number of the original image on the image management server 2410. The item 2602 “rect” is a coordinate value of two points representing a rectangular area of the dictionary pattern. An item 2603 “feature” is an image feature amount of a rectangular area of the dictionary pattern. An item 2604 “keyword” is a list of word IDs used as attributes when the dictionary is generated. Thus, the dictionary pattern management server 2350 manages the dictionary pattern defined by the items 2601 to 2604.

  Next, dictionary generation processing according to the sixth embodiment will be described. As the image managed by the image management server 2310, for example, an image collected from the Web by a Web crawler can be used. The word 2402 given to the image may be given by the operator, but when collected by the Web crawler, the text existing before and after the image citation location in the HTML document including the original image, or It is also possible to automatically extract from the title of the HTML document.

  The dictionary generation service samples an appropriate number of images from images managed by the image management server 2310 as pre-diction processing for generating a dictionary, and generates a group of regions as comparison patterns from the second image set 102. The generated region group is registered in the comparison pattern management server 2340.

  FIG. 27 is an explanatory diagram showing an example of a screen used for dictionary generation. A screen 2710 is a screen presented to the user when the user of the dictionary generation system 300 issues a dictionary generation request from the terminal device 2460 to the dictionary generation service 2330. First, the user enumerates words used as attributes in the input field 2711 on the screen 2710 of the application program running on the terminal device 2360, and transmits it to the dictionary generation service 2330.

  The dictionary generation service 2330 obtains, for the image management server 2310, the management number of the image including the word on the designated word string in the list of the word 2402, and configures information for displaying the list of those images. To the application program on the terminal device 2360. As a result, the screen presented to the user transitions from the screen 2710 to the screen 2720, and a list 2721 of the corresponding images is displayed. The image of the list 2721 is the original image for dictionary generation, but the user can also specify on the screen an image that is determined to be inappropriate for dictionary generation.

  In general, a large number of original image candidates exist. If necessary, the user can check all cases by pressing a page switching button 2722. However, since the sixth embodiment automatically acquires an appropriate dictionary pattern, there is no operational problem even if such a check by the user is not performed.

  If the number of images is large, it takes a long time to generate a dictionary pattern. Therefore, it is more convenient in terms of operation to use a certain number of images instead of using all the corresponding images as original image candidates. It is possible to determine whether or not an image has been used for dictionary generation based on a list of word IDs (hereinafter referred to as word ID list 2403) indicated by an item 2403 of the image management server 2310. As a result, an image that has not been used as a dictionary pattern in the past can be selected and used as an original image.

  When a start button 2723 on the screen 2720 is pressed, a dictionary pattern generation request using an image confirmed by the user as an original image is transmitted to the dictionary generation service 2330. The dictionary generation service 2330 executes a dictionary generation process by using the set of images confirmed by the user as the first image set 101. On the other hand, the second image set 102 is managed on the comparison pattern management server 2340. At this time, the word string designated by the user by referring to the management number of the image indicated by the item 2501. Exclude images included in the word 2502 of the word ID list 2403.

  The dictionary generation service 2330 makes an inquiry to the image management server 2310 when the desired number of images in the image registered in the comparison pattern management server 2340 does not reach the desired number. Then, the dictionary generation service acquires images in the second image set 102, that is, images in which the designated word string is not included in the word 2402 of the word ID list 2403, and region candidates for comparison are acquired from these images. Is generated. The generated rectangular area is additionally registered in the comparison pattern management server 2340 as an area group to be compared.

  When the dictionary generation service 2330 finishes the dictionary generation process, the result is registered in the dictionary pattern management server 2350. At the same time, the ID string of the designated word string is stored in the word ID list of item 2604. Also, the word ID list 2403 of the image management server 2310 is updated. The user can check the registered dictionary pattern on the terminal device 2360.

  FIG. 28 is an explanatory diagram illustrating a display example of a confirmation screen. A list of generated dictionary patterns is displayed in a display area 2801 on the confirmation screen 2800. When an unsuitable dictionary pattern such as the icon 2802 is included, the user can delete it from the registered data of the dictionary pattern management server 2350 by specifying the dictionary pattern on this screen. it can. Furthermore, if the user desires, data that is not appropriate as the dictionary pattern can be registered in the comparison pattern management server 2340. A pattern similar to the pattern registered in the comparison pattern management server 2340 is more likely to be excluded in subsequent generation of a dictionary pattern.

  As described above, according to this embodiment, a dictionary can be automatically generated by automatically registering a highly reliable dictionary pattern.

  Although the present invention has been described in detail with reference to the accompanying drawings, the present invention is not limited to such specific configurations, and various modifications and equivalents within the spirit of the appended claims Includes configuration.

Claims (14)

In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. And
A determination unit that determines whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition unit;
A registration unit for registering the selection area in a dictionary in which a dictionary pattern group is stored, when the determination unit determines that the dictionary pattern should be used ;
Common attribute information is given to each image of the first image set including the first image, and each attribute of the second image set including the second image is assigned to the attribute. No information is given, the first region group is a region group obtained from each image in the first image set, and the second region group is in the second image set. A dictionary generation system characterized by being a group of regions obtained from each of the images . In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. And
A determination unit that determines whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition unit;
A registration unit for registering the selection area in a dictionary in which a dictionary pattern group is stored, when the determination unit determines that the dictionary pattern should be used;
Attribute information is not given to any of each image of the first image set including the first image and each image of the second image set including the second image, and the first image set includes the first image set. The area group is an area group obtained from each image in the first image set, and the second area group is an area group obtained from each image in the second image set. A dictionary generation system characterized by that. In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. And
A determination unit that determines whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition unit;
A registration unit for registering the selection area in a dictionary in which a dictionary pattern group is stored, when the determination unit determines that the dictionary pattern should be used;
No attribute information is given to each image of the first image set including the first image, and a common attribute is assigned to each image of the second image set including the second image. Information is provided, the first region group is a region group obtained from each image in the first image set, and the second region group is in the second image set. A dictionary generation system characterized by being a group of regions obtained from each image. In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. And
A determination unit that determines whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition unit;
A registration unit for registering the selection area in a dictionary in which a dictionary pattern group is stored, when the determination unit determines that the dictionary pattern should be used;
Common first attribute information is given to each image of the first image set including the first image, and each image of the second image set including the second image is assigned to each image of the first image set including the first image. , Common second attribute information is given, and the first area group is an area group obtained from each image in the first image set, and the second area group is A dictionary generation system, which is a group of regions obtained from each image in the second image set. In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. A feature amount between a first minimum distance that is the minimum of the plurality of first distances, and a second area selected from the selected area and a second area group in the second image. An acquisition unit that acquires a second minimum distance that is the minimum of the plurality of second distances in the space;
A determination unit that determines whether or not the selection area should be a dictionary pattern based on a ratio between the first minimum distance and the second minimum distance acquired by the acquisition unit;
A registration unit for registering the selection area in a dictionary in which a dictionary pattern group is stored;
A dictionary generation system comprising: The dictionary generation system according to claim 5,
The acquisition unit acquires a first average minimum distance obtained by averaging the first minimum distance for each selected region, and acquires the second minimum distance for each selected region and calculates the average. The second average minimum distance
The determination unit determines whether or not the selection area should be a dictionary pattern based on a ratio between the first average minimum distance and the second average minimum distance. The dictionary generation system according to any one of claims 1 to 4,
A perturbation processing unit that generates a new region by performing a perturbation process that perturbs the selected region based on a ratio between the first distance and the second distance;
The determination unit, on the basis of the processing result of the perturbation processor, dictionary generation system that is characterized in that determining whether to said selection area on the dictionary pattern. The dictionary generation system according to claim 7,
The dictionary generation system, wherein the perturbation processing unit executes the perturbation process by reducing a perturbation amount that perturbs the selected region according to an increase in the number of times the perturbation process is executed. The dictionary generation system according to claim 7,
The acquisition unit acquires a first average minimum distance obtained by averaging the first minimum distance that is the minimum of the plurality of the first distances for each of the selected regions, and a plurality of the first distances. Obtaining a second average minimum distance obtained by averaging the second minimum distance, which is the smallest of the two distances, for each selected region;
A determination unit configured to determine a specific selection region to be subjected to the perturbation process from a plurality of the selection regions based on a ratio between the first average minimum distance and the second average minimum distance;
The dictionary generation system, wherein the perturbation processing unit generates a new region by executing the perturbation process for the specific selection region determined by the determination unit. The dictionary generation system according to claim 1,
The registration unit registers the selected area determined to be a dictionary pattern by the determination unit in the dictionary in association with the attribute information. The dictionary generation system according to claim 3,
The registration unit registers the selected area determined to be a dictionary pattern by the determination unit in the dictionary in association with the attribute information. The dictionary generation system according to claim 1,
A display unit configured to display the selection area determined to be a dictionary pattern by the determination unit so as to be able to designate whether or not to be registered in the dictionary;
The said registration part registers the said selection area | region where registration to the said dictionary of the said selection area | region displayed on the said display part was designated to the said dictionary, The dictionary generation system characterized by the above-mentioned. A dictionary generation method executed by a dictionary generation system comprising: a processor that executes a program; and a memory that stores a program executed by the processor,
The processor is
In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. Procedure and
  A determination procedure for determining whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition procedure;
  A registration procedure for registering the selected area in a dictionary in which a dictionary pattern group is stored when it is determined that the determination procedure should be a dictionary pattern;
Common attribute information is given to each image of the first image set including the first image, and each attribute of the second image set including the second image is assigned to the attribute. No information is given, the first region group is a region group obtained from each image in the first image set, and the second region group is in the second image set. A dictionary generation method characterized by being a group of regions obtained from each of the images. In a dictionary generation system comprising a processor that executes a program and a memory that stores a program executed by the processor,
In a feature amount space between a selection region selected from the first region group in the first image and a first region other than the selection region selected from the first region group. Acquisition of acquiring a first distance and a second distance in the feature amount space between the selected area and a second area selected from the second area group in the second image. Procedure and
A determination procedure for determining whether or not the selection area should be a dictionary pattern based on a ratio between the first distance and the second distance acquired by the acquisition procedure;
A registration procedure for registering the selected region in a dictionary in which a dictionary pattern group is stored;
Common attribute information is given to each image of the first image set including the first image, and each attribute of the second image set including the second image is assigned to the attribute. No information is given, the first region group is a region group obtained from each image in the first image set, and the second region group is in the second image set. A dictionary generation program characterized by being a group of regions obtained from each of the images.

Images