JP6725310B2 - Image processing device and program - Google Patents

Description

The present invention relates to an image processing device and a program for calculating a feature amount of an image.

Conventionally, a method of searching an image including a subject is known. For example, when performing an image search considering the arrangement of colors, patterns, etc. on the image (hereinafter referred to as “layout”), the feature amount is calculated from each block area obtained by dividing the image in a grid pattern, and the block corresponding to the position is calculated. A method of comparing feature amounts between regions is generally used (for example, see Patent Documents 1 and 2).

However, when this method is used, the search result is affected by the similarity of the background, and the similarity of the subject is not reflected so much. Therefore, in this method, the search result intended by the user may not be obtained.

In order to solve this problem, after moving the block area at the center of the image onto the subject area, when calculating the similarity for the block area, a method of applying a high weighting coefficient to the subject area is proposed. (For example, see Patent Document 3).

Here, the block area after the central block area of the image is moved onto the object area is referred to as “object block area”. With this method, it is possible to obtain search results that emphasize the similarity of subjects.

JP, 10-260983, A JP 2001-319232 A JP, 2013-225180, A

FIG. 12 is a diagram showing an example of an image in which the background other than the subject is included in the subject block area. In FIG. 12, a region surrounded by a rectangular frame is a subject block region. These subject block areas include the background (the location of the arrow) in addition to the subject boat. Since this background is an extra area from the viewpoint of calculating the image feature amount of the subject itself, it can be said that the subject block area includes an extra area other than the subject.

As described above, since the background is included in the subject block area, the method according to Patent Document 3 described above calculates the search result affected by the background, and as a result, the search accuracy cannot be improved. There was a problem.

Therefore, the present invention has been made in order to solve the above-mentioned problems, and an object thereof is an image that enables high-accuracy image search that attaches great importance to the similarity of subjects without being affected by the background. To provide a processing device and a program.

In order to solve the above problems, the image processing apparatus according to claim 1 is an image processing apparatus that calculates a feature amount of an image including a subject, and selects an area in which the visual attention in which the subject included in the image is present is guided. Using a saliency map that represents a saliency value for visualization for each pixel, a subject rectangular area setting unit that sets a rectangular subject rectangular area including the subject, and the subject rectangular area setting unit that sets the subject rectangular area setting unit A predetermined number of control points are set on the rectangular frame of the subject rectangular area, an active contour model of the contour connecting the predetermined number of control points with a straight line is set, and the positions of the predetermined number of control points are set. A subject region setting unit that calculates the energy of the dynamic contour model each time it is moved, sets the contour of the subject as a subject contour based on the energy, and sets a region within the subject contour as a subject region And a subject feature calculation unit that calculates an image feature amount of the subject region set by the subject region setting unit, wherein the subject region setting unit sets the subject rectangle set by the subject rectangular region setting unit. A predetermined number of control points are set on the rectangular frame of the area, an active contour model of a contour connecting the predetermined number of control points is set, and the positions of the predetermined number of control points are moved. For each time, the added value of the energy related to the arc length of the contour of the active contour model, the energy related to the curvature of the active contour model, and the energy related to the gradient of the image of the active contour model, It is characterized in that it is calculated as energy, a contour connecting the predetermined number of control points when the energy is minimum is set as a subject contour, and an area within the subject contour is set as a subject area.
An image processing apparatus according to a second aspect is the image processing apparatus according to the first aspect, wherein a block setting unit that sets a plurality of grid-like block areas from the image, and a plurality of block setting units set by the block setting unit. A block feature calculation unit that calculates a feature amount for each of the block regions by calculating the entire image feature amount, and the image feature amount of the subject region calculated by the subject feature calculation unit; and the block It is characterized in that the overall image feature amount calculated by the feature calculation unit is output.

The image processing apparatus according to claim 3 is an image processing apparatus that searches for an image similar to a request image from a plurality of search target images, and a preprocessing unit that calculates an image feature amount for each of the plurality of search target images. A feature calculation unit that calculates an image feature amount for the requested image, an image feature amount for each of the plurality of search target images calculated by the preprocessing unit, and the image calculation amount calculated by the feature calculation unit. Based on the image feature amount of the requested image, the similarity calculation unit that calculates the similarity between each of the plurality of search target images and the requested image, and the similarity calculated by the similarity calculation unit. A similar image selection unit that selects an image similar to the request image from the plurality of search target images based on the plurality of search target images, and the preprocessing unit, for each of the plurality of search target images, a plurality of grid-like images. For each of the first block setting unit that sets a block region and each of the plurality of search target images, the feature amount of the plurality of block regions set by the first block setting unit is calculated by using the overall feature amount. For each of the first block feature calculation unit and each of the plurality of search target images, a saliency value for visualizing an area in which a subject included in the search target image is present to which visual attention is directed is provided for each pixel. A first subject rectangular area setting unit that sets a rectangular subject rectangular area including the subject using the represented saliency map, and the first subject rectangular area setting unit for each of the plurality of search target images By setting a predetermined number of control points on the rectangular frame of the subject rectangular area set by, a dynamic contour model of a contour connecting the predetermined number of control points with a straight line is set, and the predetermined number of control points is set. Each time the position of the control point is moved, an added value of energy related to the arc length of the contour of the active contour model, energy related to the curvature of the active contour model, and energy related to the gradient of the image of the active contour model is calculated. A first contour that is calculated as energy of the dynamic contour model, a contour connecting the predetermined number of control points when the energy is minimum is set as a subject contour, and an area within the subject contour is set as a subject area; First subject feature calculation for calculating the image feature amount of the subject region set by the first subject region setting unit for each of the subject region setting unit and the plurality of search target images. And a plurality of grid-shaped blocks for the requested image. A second block setting unit that sets a lock area; and a second block characteristic calculation unit that calculates the feature amounts of the plurality of block regions set by the second block setting unit by using the overall feature amount. A rectangular subject rectangular area including the subject is set using a saliency map that represents, for each pixel, a saliency value for visualizing a region in which the subject included in the requested image is guided. A predetermined number of control points are set on the second subject rectangular area setting unit and the rectangular frame of the subject rectangular area set by the second subject rectangular area setting unit, and the predetermined number of control points are set. Is set by a straight line, and each time the positions of the predetermined number of control points are moved, the energy related to the arc length of the contour of the active contour model and the curvature of the active contour model are set. Energy of the image of the active contour model and the energy of the gradient of the image of the active contour model are calculated as the energy of the active contour model, and the contour connecting the predetermined number of control points when the energy is the minimum is taken as a subject. A second subject area setting unit that sets the subject contour as an outline and sets an area within the subject contour as the subject area, and the image feature amount of the subject area set by the second subject area setting unit as the subject feature amount. And a second subject feature calculating unit that calculates as a whole, for each of the plurality of search target images calculated by the first block feature calculating unit of the preprocessing unit by the similarity calculating unit. Based on the feature amount and the overall feature amount of the request image calculated by the second block feature calculation unit of the feature calculation unit, overall similarity between each of the plurality of search target images and the request image Degree, the subject feature amount for each of the plurality of search target images calculated by the first subject feature calculation unit of the preprocessing unit, and the second subject feature calculation unit of the feature calculation unit. Based on the subject feature amount of the request image calculated by, the subject similarity between each of the plurality of search target images and the request image is calculated, based on the overall similarity and the subject similarity The similarity between each of the plurality of search target images and the requested image is calculated.

An image processing apparatus according to a fourth aspect is the image processing apparatus according to the third aspect, wherein the feature calculation unit is further set by the second subject area setting unit using the saliency map. A weighting factor calculation unit that averages the saliency values of all the control points of the subject contour and calculates a weighting factor of the subject region based on the average value is provided. A similarity between each of the plurality of search target images and the requested image is calculated based on the weighting factor calculated by the weighting factor calculator, the overall similarity and the subject similarity. And

The image processing device according to claim 5 is the image processing device according to any one of claims 1 to 4 , wherein energy relating to a gradient of the image of the active contour model is represented in the saliency map. And a gradient value of the pixel position of the control point obtained from the saliency value.

Further, a program according to claim 6 causes a computer to function as the image processing device according to any one of claims 1 to 5 .

As described above, according to the present invention, it is possible to realize a highly accurate image search that attaches great importance to the similarity of subjects without being significantly affected by the background.

3 is a block diagram showing a configuration example of an image processing apparatus of Example 1. FIG. It is a flow chart which shows an example of processing of a photographic subject feature processing part. 6 is a block diagram showing a configuration example of an image processing apparatus of Example 2. FIG. It is a flow chart which shows the example of processing of a preprocessing part. It is a flow chart which shows the example of processing of a search processing part. It is a figure which shows the image example for every block area|region in case of M=4 and N=4. It is a figure which shows the saliency map of the image I, and the example of a to-be-photographed object rectangular area. It is a figure which shows the example of the initial contour of an active contour model, and a control point. It is a figure which shows the example of the outline of a to-be-photographed object when the energy E is the minimum in the saliency map of the image I. It is a figure which shows the example of the outline of the to-be-photographed object in the image I when the energy E is the minimum. It is a diagram illustrating an example of an image feature vector V _I. It is a figure which shows the example image which contains the background other than a to-be-photographed object in a to-be-photographed block area|region.

Hereinafter, modes for carrying out the present invention will be described in detail with reference to the drawings. The present invention sets a rectangular subject rectangular area including a subject using a saliency map for an image, extracts the subject area from the subject rectangular area so as to minimize the energy of the active contour model, and extracts the subject area. It is characterized in that the image feature amount of is calculated.

Further, according to the present invention, the similarity between each of the plurality of search target images and the requested image is calculated using the image feature amount of the block region and the image feature amount of the subject region, and the similarity is calculated from the plurality of search target images. It is characterized by selecting an image with a high degree.

As a result, the degree of inclusion of the background in the subject area is lower than that in the subject rectangular area, so the image feature amount of the subject area becomes a value that reflects the characteristics of the subject itself. Therefore, by using the image feature amount of the subject area for the image search, it is possible to realize a high-precision image search that attaches great importance to the similarity of the subject without being affected by the background.

[Example 1]
First, the first embodiment will be described. The first embodiment is an example in which a conventional image feature amount for each block region is calculated, a subject region in which the background is excluded from the image is set, and the image feature amount of the subject region is calculated.

FIG. 1 is a block diagram illustrating a configuration example of the image processing apparatus according to the first embodiment. The image processing apparatus 1 includes a block setting unit 10, a block feature calculation unit 11, a subject feature processing unit 12, and an image feature vector generation unit 13.

The image processing apparatus 1 inputs one image I, calculates an overall image feature vector V _GI consisting of image feature amounts for each block region, and also calculates a subject feature vector V _OI and an object feature amount V _OI which are image feature amounts of a subject region. The weighting coefficient W _{I of the} area is calculated. Then, the image processing apparatus 1 outputs the image feature vector V _I ={V _GI , V _OI } of the image I and the weighting factor W _I of the subject region.

Here, the overall image feature vector V _GI indicates the overall feature amount of the image based on the block division of the image I, and the subject feature vector V _OI indicates the feature amount of the subject. The weight coefficient W _I of the subject region indicates the degree to which the subject stands out in the image I, and is used when calculating the similarity between images including the subject.

The block setting unit 10 inputs the image I, sets M1×M2 block regions in the image I in a grid pattern, and cuts out an image for each block region. M1 and M2 are positive integers. Then, the block setting unit 10 outputs the image for each block area to the block feature calculation unit 11.

FIG. 6 is a diagram showing an example of an image for each block area when M1=4 and M2=4. As shown in FIG. 6, 16 images for each block area (m1, m2) are cut out from the image I. m1=1,..., 4, m2=1,.

Returning to FIG. 1, the block feature calculation unit 11 inputs an image for each block region from the block setting unit 10, calculates an image feature amount V _GI (m1, m2) for each block region (m1, m2), An image overall feature vector V _GI consisting of the image feature amount V _GI (m1, m2) for each block area (m1, m2) is generated. Then, the block feature calculation unit 11 outputs the entire image feature vector V _GI to the image feature vector generation unit 13.

When m1=1,..., M1, m2=1,..., M2, the overall image feature vector V _GI is expressed by the following equation.
[Equation 1]
V _GI ={V _GI (1,1),..., V _GI (M1,M2)} (1)

The subject feature processing unit 12 inputs the image I, sets the subject region from the image _I , calculates the subject feature vector V _OI and the weighting coefficient W _I, and outputs the subject feature vector V _OI to the image feature vector generation unit 13. Then, the weight coefficient W _I is output to the outside.

The subject feature processing unit 12 includes a subject rectangular region setting unit 14, a subject region setting unit 15, a subject feature calculation unit 16, and a weighting factor calculation unit 17. FIG. 2 is a flowchart showing a processing example of the subject feature processing unit 12.

The subject rectangular area setting unit 14 inputs the image I (step S201), generates a saliency map of the image I, and uses the saliency map of the image I to set a rectangular subject rectangular area including the subject ( Step S202). Then, the subject rectangular area setting unit 14 outputs information for identifying the image I and the subject rectangular area to the subject area setting unit 15.

The processing for generating the saliency map and the processing for setting the subject rectangular area are known, and for details, refer to Patent Document 3 mentioned above.

The saliency map of an image is a map that visualizes a region where an object (subject) in the image is guided, in consideration of color, brightness, direction, etc., and the saliency value of the position of each pixel. It is represented by. Image saliency maps are known, see the following references for details.
[Non-Patent Document] L. Itti, C. Koch, and E. Niebur: “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, IEEE Transactions on Pattern Analysis and Machine Intelligence, VOl. 20, No11, 1998. November [Patent Document] Japanese Patent Publication No. 2007-51509

FIG. 7 is a diagram showing an example of the saliency map of the image I and the subject rectangular area. As shown in FIG. 7, the saliency map of the image I is represented by the saliency value at each pixel position, and the saliency value is represented by the brightness level. The area surrounded by a square is the subject rectangular area.

Returning to FIG. 1 and FIG. 2, the subject area setting unit 15 inputs information for specifying the image I and the subject rectangular area from the subject rectangular area setting unit 14. Then, the subject region setting unit 15, on the rectangular frame of the object rectangular region, N _c pieces of control points {CP (1), ···, CP (N c)} set arbitrarily. Subject region setting unit 15, N _c pieces of control points {CP (1), ···, CP (N c)} of connecting it contour with a straight line, the initial contour of the active contour model with these control points (Step S203). N _c is preset.

FIG. 8 is a diagram showing an example of initial contours and control points of the active contour model. In FIG. 8, circles indicate control points, and the contour connecting the control points with a straight line is the initial contour of the active contour model.

Returning to FIG. 1 and FIG. 2, the subject region setting unit 15 calculates the energies E of all the control points of the active contour model shown below, moves the control points in the direction in which the energy E is the minimum, and outputs the energy E. Stop each control point at the position where is minimum.

As a method for minimizing the energy of the active contour model, for example, the snake method is used. For details, refer to the following non-patent documents.
[Non-Patent Document] M. Kass, A. Witkin, and D. Terzopoulos: “Snakes: Active contour models,” International Journal of Computer Vision, 1(4): 321-331, January 1998.

Specifically, the subject area setting unit 15 calculates the energy E of the active contour model (step S204) and determines whether the energy E is the minimum (step S205).

When it is determined in step S205 that the energy E is not the minimum (step S205: N), the subject area setting unit 15 performs arbitrary control in a direction in which the contour is contracted so that the area of the active contour model is reduced. The point is arbitrarily moved (step S206). Then, the processing of steps S204 to S206 is repeated. When the subject area setting unit 15 determines in step S205 that the energy E is the minimum (step S205: Y), the process proceeds to step S207.

The subject area setting unit 15 may perform the minimum determination of the energy E in step S205 after performing the processes of steps S204 and S206 a predetermined number of times. Further, the subject area setting unit 15 may determine that the minimum point of the energy E is the minimum point of the energy E, when the minimum point of the energy E is detected.

In step S204, the energy E of the active contour model is calculated by the following formula.
[Equation 2]
E = E _LEN + E _CRV + E _IMG ... (2)

The energy E of the active contour model adds the energy E _LEN for the arc length of the entire contour, the energy E _{CRV for} the curvature at each control point (curvature of the entire contour), and the energy E _IMG for the gradient of the image at each control point. Required by that.

When the energy E _LEN is small, it indicates that the arc length of the entire contour is short, and when the energy E _CRV related to the curvature of each control point is small, the contour does not bend sharply (small and smooth). Is shown. Further, when the energy E _IMG related to the gradient of the image at each control point is small, it is indicated that each control point has a high proportion at the boundary between the subject and the background. Therefore, the case where the energy E of the active contour model is minimum means that the arc length of the entire contour is short, the bending of each control point is not tight (small and smooth), and each control point is Indicates that there is a high ratio of being present at the boundary between the subject and the background.

Here, at the control point CP(i) (i=1,..., N _c ), the x coordinate of the control point CP(i) is x _i , and the y coordinate of the control point CP(i) is y _i . .. In addition, x ₀ =x _Nc . Energy E _LEN related to the arc length of the entire contour is calculated by the following formula.
[Equation 3]

The energy E _CRV regarding the curvature at each control point is calculated by the following formula.
[Equation 4]

The energy E _IMG regarding the image at each control point is calculated by the following formula.
[Equation 5]
I′(i) is image energy at the control point CP(i), and gradient (edge) information (gradient value) obtained from the pixel value at the pixel position of the control point in the image I is used.

Here, when the background has a complicated texture, in the process of moving the control point in the direction in which the energy E becomes the minimum, the control point stops at the edge in the background, not the edge between the subject and the background. It can happen. In this case, in step S207 described later, a subject contour that is significantly different from the original contour of the subject may be set.

Therefore, the subject area setting unit 15 uses the saliency map of the image I generated in step S202 as the image energy I′(i) when calculating the energy E _IMG regarding the image at each control point. That is, the subject region setting unit 15 uses, as the image energy I′(i), the gradient value of the pixel position of the control point obtained from the saliency value in the saliency map of the image I.

The saliency map of the image I is composed of saliency values that visualize a region (a conspicuous region in the entire image I) in which a subject to which visual attention is directed exists in the image I. Therefore, by using this saliency map, it is possible to set a subject contour close to the original contour of the subject and to set a highly accurate subject area in step S207 described later. In this case, a rough subject area can be set even if the boundary between the background and the subject is ambiguous.

The subject area setting unit 15 shifts from step S205, and sets the contour connecting all the control points when the energy E is the minimum as a contour of the subject (subject contour), and the region within the subject contour is set as the subject. The area is set (step S207). Then, the subject region setting unit 15 outputs information for identifying the image I and the subject region to the subject feature calculation unit 16 and the weighting factor calculation unit 17.

FIG. 9 is a diagram showing an example of the contour of the subject when the energy E is the minimum in the saliency map of the image I, and FIG. 10 is an example of the contour of the subject when the energy E is the minimum in the image I. FIG. As shown in FIGS. 9 and 10, the object contour is set by connecting all the control points when the energy E is the minimum with a straight line.

It can be seen from FIGS. 9 and 10 that the subject contour is closer to the shape of the subject than the initial contour of the active contour model shown in FIG. That is, the subject area, which is an area within the subject contour in FIGS. 9 and 10, is an area (background area) in which a background other than the subject is present rather than the subject rectangular area (conventional subject block area) shown in FIG. The area is getting smaller.

Returning to FIG. 1 and FIG. 2, the subject feature calculation unit 16 inputs information for specifying the image I and the subject region from the subject region setting unit 15. Then, the subject feature calculation unit 16 calculates the image feature amount only from the pixels of the subject region in the image I, and outputs this as the subject feature vector V _OI to the image feature vector generation unit 13 (step S208).

The weighting factor calculation unit 17 inputs information for specifying the image I and the subject region from the subject region setting unit 15. Then, using the saliency map of the image I, the weighting factor calculation unit 17 averages the saliency values of all the control points of the subject contour set by the subject area setting unit 15 in step S207. The weighting factor calculation unit 17 obtains the weighting factor W _I of the subject area by dividing the average value K _ave by a predetermined value (255) in the following formula, and outputs the weighting factor W _I to the outside (step S209).
[Equation 6]
W _I =K _ave /255 (6)

The image feature vector generation unit 13 inputs the entire image feature vector V _GI from the block feature calculation unit 11 and the subject feature vector V _OI from the subject feature calculation unit 16 of the subject feature processing unit 12. Then, the image feature vector generation unit 13 generates an image feature vector V _I ={V _GI , V _OI } of the image I composed of the entire image feature vector V _GI and the subject feature vector V _OI, and the image feature vector of the image I is generated. Output V _I ={V _GI , V _OI } to the outside.

FIG. 11 is a diagram showing an example of the image feature vector V _I. As shown in FIG. 11, the image feature vector V _I ={V _GI , V _OI } of the image I is the entire image feature vector consisting of the image feature amount V _GI (m1, m2) for each block area (m1, m2). V _GI and a subject feature vector V _OI which is an image feature amount of the subject region.

As described above, according to the image processing apparatus 1 of the first embodiment, the subject rectangular area setting unit 14 of the subject feature processing unit 12 sets the subject rectangular area of the image I using the saliency map of the image I. Then, the subject area setting unit 15 sets N _c control points on the rectangular frame of the subject rectangular area, and sets a contour connecting the N _c control points with a straight line as the initial contour of the active contour model. .. Then, the subject region setting unit 15 adds the energy E _LEN regarding the arc length of the entire contour, the energy E _CRV regarding the curvature at each control point, and the energy E _IMG regarding the image at each control point to the _active contour model. Ask for.

The subject area setting unit 15 moves the control point in the direction in which the energy E is minimized, stops each control point at the position where the energy E is minimized, sets the subject contour at that position, and sets the region within the subject contour. Is set as the subject area.

The subject feature calculation unit 16 calculates the image feature amount of the subject region in the image I as a subject feature vector V _OI .

As a result, the subject region is less likely to include the background than the subject rectangular region, so that the subject feature vector V _OI does not reflect the background feature so much and the subject feature vector V _OI has an image feature amount that reflects the feature of the subject itself. Become.

Therefore, by using the subject feature vector V _OI for the image search, it is possible to realize a highly accurate image search that emphasizes the similarity of the subjects without being affected by the background. In addition, the subject feature vector V _OI can be used for a process of classifying a large number of images, and a highly accurate image classifying process that emphasizes the similarity of the subjects can be realized.

Furthermore, in addition to the object feature vector V _OI , the conventional image overall feature vector V _GI consisting of the image feature amount for each block area is used, so that the background of the background can be maintained while maintaining the similarity of the layout such as color and pattern in the image I. It is possible to realize a highly accurate image search that attaches great importance to the similarity of subjects without being affected by the above. It is also possible to use the subject feature vector V _OI and the entire image feature vector V _GI in the process of classifying a large number of images. For example, it is possible to implement high-precision image classification processing that emphasizes the similarity of the subjects after classifying the entire images.

[Example 2]
Next, a second embodiment will be described. The second embodiment is an example in which an image similar to the request image is selected from a plurality of search target images using the image feature amount of each block region and the image feature amount of the subject region calculated in the first embodiment. In the second embodiment, between each of the plurality of search target images and the request image, the similarity reflecting the image feature amount of each block region and the image feature amount of the subject region is calculated, and the similarity is calculated from the plurality of search target images. Select a high-quality image.

FIG. 3 is a block diagram illustrating a configuration example of the image processing apparatus according to the second embodiment. The image processing apparatus 2 includes a search target image DB (database) 20, a preprocessing unit 21, an image feature vector DB 22 and a search processing unit 23.

The image processing apparatus 2 selects an image similar to the requested image Q from the plurality of search target images I[1],..., I[N _D ] and outputs it. Specifically, the image processing apparatus 2 uses the image feature vectors V _I [1],..., V _I [N _D ] of the plurality of search target images I[1],..., I[N _D ]. Is calculated and one requested image Q is input, and the image feature vector V _Q and the weighting coefficient W _Q of the requested image _Q are calculated. Then, the image processing apparatus 2 uses the similarities S(1),... S(N _D ) between the request image Q and the plurality of search target images I[1],..., I[N _D ]. Are calculated respectively. The image processing device 2, a plurality of retrieval target image I [1], · · ·, and selects and outputs the image of the predetermined high number of similarity S of the I [N _D].

The search target image DB 20 stores N _D search target images I[1],..., I[N _D ]. N _D is the number of images to be searched and is an integer of 2 or more.

The preprocessing unit 21 performs the same processing as that of the image processing apparatus 1 shown in FIG. First, the preprocessing unit 21 sequentially reads the search target images I[1],..., I[N _D ] from the search target image DB 20. Then, the preprocessing unit 21 calculates image feature vectors V _I [1],..., V _I [N _D ] for each of the search target images I[1],..., I[N _D ]. Then, these are stored in the image feature vector DB 22. The image feature vector is V _I [1]={V _GI [1], V _OI [1]},..., V _I [N _D ]={V _GI [N _D ], V _OI [N _D ] }.

When n=1,..., N _D , the image feature vector V I [n] of the image _I [n] is an image including the image feature amount V _GI (m1, m2) for each block area (m1, m2). It is composed of the overall feature vector V _GI [n] and the subject feature vector V _OI [n] which is the image feature amount of the subject region.

The image feature vector DB 22 has image feature vectors V _I [1]={V _GI [1], V _OI [1] of N _D search target images I[1],..., I[N _D ]. },..., V _I [N _D ]={V _{G I} [N _D ], V _OI [N _D ]} are stored.

FIG. 4 is a flowchart showing a processing example of the preprocessing unit 21. The preprocessing unit 21 sets the parameter n=1 (step S401) and reads the search target image I[n] from the search target image DB 20 (step S402).

The pre-processing unit 21 calculates an image feature vector V _I [n] consisting of the entire image feature vector V _GI [n] and the subject feature vector V _OI [n] for the search target image I[n], and calculates this as an image. It is stored in the feature vector DB 22 (step S403). The method of calculating the image feature vector V _I [n] is the same as the processing of the image processing apparatus 1 shown in FIG.

The preprocessing unit 21 adds 1 to the parameter n (step S404), and determines whether or not the parameter n is larger than the predetermined number of search target images N _D (step S405). When the pre-processing unit 21 determines in step S405 that the parameter n is not larger than the search target image number N _D (step S405: N), the pre-processing unit 21 proceeds to step S402. On the other hand, when the pre-processing unit 21 determines in step S405 that the parameter n is larger than the number N _{D of} search target images (step S405: Y), the process ends.

As a result, the image feature vector DB 22 stores the image feature vector V _I [n]={V _GI [n], V _OI [n]} of the search target image I[n] (n=1, _... .., N _D ).

Returning to FIG. 3, the search processing unit 23 inputs the request image Q, calculates the image feature vector V _Q and the weighting factor W _Q of the request image Q, and from the image feature vector DB 22 the image feature vector V _I [1]. ,..., V _I [N _D ] is read. The search processing unit 23 uses the weighting factor W _Q, requested image Q and the search target image I [1], · · ·, the similarity S (1) between the I [N _D], ·· , S(N _D ) are calculated, and R images I[n ₁ ],..., I[n _R ] having a high degree of similarity are output as search results.

The search processor 23 includes a feature calculator 24, a similarity calculator 25, and a similar image selector 26. FIG. 5 is a flowchart showing a processing example of the search processing unit 23. The feature calculation unit 24 of the search processing unit 23 performs the same processing as that of the image processing device 1 shown in FIG. The feature calculation unit 24 inputs the requested image Q (step S501), calculates an image feature vector V _Q ={V _GQ , V _OQ } composed of the entire image feature vector V _GQ and the subject feature vector V _OQ , and weights it. The coefficient W _Q is calculated (step S502). Then, the feature calculation unit 24 outputs the image feature vector V _Q ={V _GQ , V _OQ } of the requested image Q and the weighting factor W _Q to the similarity calculation unit 25.

The similarity calculation unit 25 inputs the image feature vector V _Q ={V _GQ , V _OQ } of the requested image Q and the weighting coefficient W _Q from the feature calculation unit 24. Then, the similarity calculation unit 25 sets the parameter n=1 (step S503), and the image feature vector V _I [n]={V _GI [n], of the search target image I[n] is set from the image feature vector DB 22. V _OI [n]} is read (step S504).

The similarity calculation unit 25 determines the similarity S _G (n) of the entire image between the image entire feature vector V _GQ of the requested image Q and the image entire feature vector V _GI [n] of the search target image I[n]. Along with the calculation, the similarity S _O (n) of the subject between the subject feature vector V _OQ of the request image Q and the subject feature vector V _OI [n] of the search target image I[n] is calculated. These similarities are obtained by a general calculation method such as inner product of vectors or normalization of Euclidean distance to 0 to 1.

The similarity calculating unit 25 uses the similarity S _G (n) of the entire image, the similarity S _O (n) of the subject, and the weighting factor W _Q of the request image _Q to calculate the required image Q as An overall similarity S(n) with the search target image I[n] is calculated (step S505). Then, the similarity calculation unit 25 outputs the similarity S(n) to the similar image selection unit 26.
[Equation 7]
_{S (n) = (S G} (n) + W Q S O (n)) / (1 + W Q) ··· (7)

In Expression (7), when the subject is conspicuous (when the weighting coefficient W _Q is large), the ratio of the subject similarity S _O (n) reflected in the similarity S(n) is high. On the contrary, when the subject is not conspicuous (when the weighting coefficient W _Q is small), the ratio of the subject similarity S _O (n) reflected in the similarity S(n) is low. As a result, the similarity S(n) according to the degree of conspicuousness of the subject can be obtained.

The similarity calculation unit 25 may calculate the similarity S(n) by adding the similarity S _G (n) and S _O (n) without using the weighting factor W _Q. ..

The similarity calculation unit 25 adds 1 to the parameter n (step S506), and determines whether the parameter n is larger than the predetermined number N _D of search target images (step S507). When the similarity calculation unit 25 determines in step S507 that the parameter n is not larger than the search target image number N _D (step S507: N), the similarity calculation unit 25 proceeds to step S504.

On the other hand, when the similarity calculation unit 25 determines in step S507 that the parameter n is larger than the search target image number N _D (step S507: Y), the similarity calculation unit 25 proceeds to step S508.

As a result, the similarity calculation unit 25 obtains the similarity S(n) between the request image Q and all the search target images I[n] (n=1,..., N _D ).

The similar image selection unit 26 inputs all the similarity S(n) from the similarity calculation unit 25, and the N _D search target images I[1],..., I stored in the search target image DB 20. from [n _D], the image I [n _1] values of high-level R sheets of similarity S (n), ···, selects the I [n _R]. R is an integer of 1 or more. Then, the similar image selection unit 26 outputs the selected R images I[n ₁ ],..., I[n _R ] or information for specifying these images as a search result (step S508). ..

As described above, according to the image processing apparatus 2 of the second embodiment, the preprocessing unit 21 performs N _D sheets (n=1,..., And) by the same processing as the image processing apparatus 1 of the first embodiment. For each of the search target images I[n] of N _D ), an image feature vector V _I [n]={V _GI [n] consisting of the entire image feature vector V _GI [n] and the subject feature vector V _OI [n]. , V _OI [n]} is calculated.

The feature calculation unit 24 of the search processing unit 23 calculates an image feature vector V _Q ={V _GQ , V _OQ } composed of the entire image feature vector V _GQ and the subject feature vector V _OQ for the requested image Q, and a weighting factor. Calculate W _Q.

The similarity calculation unit 25 calculates the similarity S _G (n) between the overall image feature vector V _GQ and the overall image feature vector V _GI [n], and the subject feature vector V _OQ and the subject feature vector V _OI [ calculating a similarity S _O (n) between the n], the similarity S _G (n), using S _O (n) and the weighting factor W _Q, requested image Q and the search target image I [n] And a similarity S(n) between and.

The similar image selection unit 26 selects the upper R images I[n ₁ ],..., I[n _R ] having a high value of the similarity S(n), and outputs this as a search result.

As a result, it is possible to realize a highly accurate image search that attaches importance to the similarity of the subject without being affected by the background so much while maintaining the similarity of the layout such as colors and patterns in the image.

A normal computer can be used as the hardware configuration of the image processing apparatuses 1 and 2 according to the first and second embodiments of the present invention. The image processing devices 1 and 2 are configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, and an interface.

The functions of the block setting unit 10, the block feature calculation unit 11, the subject feature processing unit 12, and the image feature vector generation unit 13 included in the image processing apparatus 1 are respectively executed by causing the CPU to execute a program that describes these functions. Will be realized. Further, each function of the preprocessing unit 21 and the search processing unit 23 included in the image processing apparatus 2 is also realized by causing the CPU to execute a program describing these functions.

Further, these programs can be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc., and distributed via a network. You can also send and receive.

1, 2 image processing device 10 block setting unit 11 block feature calculation unit 12 subject feature processing unit 13 image feature vector generation unit 14 subject rectangular region setting unit 15 subject region setting unit 16 subject feature calculation unit 17 weighting factor calculation unit 20 search target Image DB
21 preprocessing unit 22 image feature vector DB
23 Search Processing Section 24 Feature Calculation Section 25 Similarity Calculation Section 26 Similar Image Selection Section

Claims (6)

In an image processing device that calculates a feature amount of an image including a subject,
A rectangular subject rectangular area including the subject is set using a saliency map that represents, for each pixel, a saliency value for visualizing a region in the image in which the subject is present and in which visual attention is guided. Subject rectangular area setting unit,
A predetermined number of control points are set on the rectangular frame of the subject rectangular area set by the subject rectangular area setting unit, and an active contour model of a contour connecting the predetermined number of control points with a straight line is set. Then
Each time the positions of the predetermined number of control points are moved, the energy of the active contour model is calculated,
A subject area setting unit that sets the contour of the subject as a subject contour based on the energy, and sets a region within the subject contour as a subject area;
A subject feature calculation unit that calculates an image feature amount of the subject region set by the subject region setting unit,
The subject area setting unit,
A predetermined number of control points are set on the rectangular frame of the subject rectangular area set by the subject rectangular area setting unit, and an active contour model of a contour connecting the predetermined number of control points with a straight line is set. Then
Every time the positions of the predetermined number of control points are moved, energy related to the arc length of the contour of the active contour model, energy related to the curvature of the active contour model, and energy related to the gradient of the image of the active contour model. An added value of is calculated as the energy of the active contour model,
An image processing apparatus, wherein a contour connecting the predetermined number of control points when the energy is minimum is set as a subject contour, and an area within the subject contour is set as a subject area. The image processing apparatus according to claim 1,
A block setting unit for setting a plurality of grid-like block regions from the image,
A block feature calculation unit that calculates the feature amount for each of the plurality of block areas set by the block setting unit by calculating the entire image feature amount,
An image processing apparatus, which outputs the image feature amount of the subject region calculated by the subject feature calculation unit and the overall image feature amount calculated by the block feature calculation unit. In an image processing device for searching an image similar to a requested image from a plurality of search target images,
For each of the plurality of search target images, a pre-processing unit that calculates the image feature amount,
A feature calculation unit that calculates an image feature amount for the requested image;
Based on the image feature amount of each of the plurality of search target images calculated by the preprocessing unit, and the image feature amount of the request image calculated by the feature calculation unit, each of the plurality of search target images And a similarity calculation unit that calculates a similarity between the requested image and
A similar image selecting unit that selects an image similar to the requested image from the plurality of search target images based on the similarity calculated by the similarity calculating unit,
The pre-processing unit,
A first block setting unit that sets a plurality of grid-shaped block areas for each of the plurality of search target images;
A first block feature calculation unit that calculates a feature amount of a plurality of block areas set by the first block setting unit as an overall feature amount for each of the plurality of search target images;
For each of the plurality of search target images, using a saliency map that represents, for each pixel, a saliency value for visualizing a region in which a subject included in the search target image is present where visual attention is guided, A first subject rectangular area setting unit that sets a rectangular subject rectangular area including a subject;
For each of the plurality of search target images, a predetermined number of control points are set on the rectangular frame of the subject rectangular area set by the first subject rectangular area setting unit, and the predetermined number of control points are set. Set the active contour model of the contour that connects
Every time the positions of the predetermined number of control points are moved, energy related to the arc length of the contour of the active contour model, energy related to the curvature of the active contour model, and energy related to the gradient of the image of the active contour model. An added value of is calculated as the energy of the active contour model,
A first subject area setting unit that sets a contour connecting the predetermined number of control points when the energy is minimum as a subject contour, and sets an area within the subject contour as a subject area;
A first subject feature calculation unit that calculates an image feature amount of the subject region set by the first subject region setting unit as a subject feature amount for each of the plurality of search target images;
The feature calculation unit,
A second block setting unit that sets a plurality of grid-shaped block areas for the requested image;
A second block feature calculation unit that calculates the feature amount of the plurality of block areas set by the second block setting unit as the overall feature amount;
A rectangular subject rectangular area including the subject is set using a saliency map that represents, for each pixel, a saliency value for visualizing a region in which the subject included in the requested image is guided. A second subject rectangular area setting unit,
A dynamic contour having a predetermined number of control points set on the rectangular frame of the subject rectangular area set by the second subject rectangular area setting unit and connecting the predetermined number of control points with a straight line Set the model,
Every time the positions of the predetermined number of control points are moved, energy related to the arc length of the contour of the active contour model, energy related to the curvature of the active contour model, and energy related to the gradient of the image of the active contour model. An added value of is calculated as the energy of the active contour model,
A second subject area setting unit that sets a contour connecting the predetermined number of control points when the energy is minimum as a subject contour, and sets a region within the subject contour as a subject area;
A second subject feature calculation unit that calculates an image feature amount of the subject region set by the second subject region setting unit as a subject feature amount,
The similarity calculation unit,
The overall feature amount for each of the plurality of search target images calculated by the first block feature calculation unit of the preprocessing unit, and the total feature amount calculated by the second block feature calculation unit of the feature calculation unit Based on the overall feature amount of the requested image, calculate the overall similarity between each of the plurality of search target images and the requested image,
The subject feature amount for each of the plurality of search target images calculated by the first subject feature calculation unit of the preprocessing unit, and the subject feature amount calculated by the second subject feature calculation unit of the feature calculation unit Based on the subject feature amount of the request image, the subject similarity between each of the plurality of search target images and the request image is calculated,
An image processing apparatus, wherein the similarity between each of the plurality of search target images and the requested image is calculated based on the overall similarity and the subject similarity. The image processing apparatus according to claim 3 ,
The feature calculation unit further includes
Using the saliency map, the saliency values of all the control points of the subject contour set by the second subject area setting unit are averaged, and the weighting factor of the subject area is calculated based on the average value. A weighting factor calculator,
The similarity calculation unit,
The similarity between each of the plurality of search target images and the requested image is calculated based on the weighting factor calculated by the weighting factor calculator of the feature calculator, the overall similarity and the subject similarity. An image processing apparatus, characterized by: The image processing apparatus according to any one of claims 1 to 4 ,
An image processing apparatus, wherein energy relating to a gradient of an image of the active contour model is set as a gradient value of a pixel position of the control point obtained from a saliency value represented in the saliency map. A program for causing a computer to function as the image processing device according to any one of claims 1 to 5 .