JP6717049B2 - Image analysis apparatus, image analysis method and program - Google Patents

Description

The present invention relates to an image analysis device, an image analysis method and a program.

Conventionally, a technique for extracting a user's region of interest from an image has been widely used for automatic cropping/thumbnail generation of an image, preprocessing of annotation generation in image understanding/image search, and the like. Methods using object recognition and saliency maps are known.

As a region-of-interest extraction technique based on object recognition, Patent Document 1 discloses a technique for detecting a face region in an image and extracting an image of the face region, and Patent Document 2 discloses a person region in the image by human detection. A technique for extracting is disclosed. When extracting the region of interest based on object recognition, it is necessary to prepare a model for each object.

On the other hand, in the interest area extraction using the saliency map, a more general interest area extraction can be performed by using low-order feature quantities such as colors and edges. In this regard, Non-Patent Document 1 discloses a method of generating a saliency map in a bottom-up manner from a local feature of an image using a human visual model studied in neuroscience. Further, Patent Document 3 discloses a technique of accurately obtaining a saliency map by multiplying the map of the edge amount calculated for each pixel by the attention area weighting map. Furthermore, Patent Documents 4 and 5 disclose techniques for calculating saliency by combining depth information with image feature amounts.

Furthermore, in recent years, an approach has been attempted in which a region of interest is extracted by using higher-order semantic information for lower-order features (color, edge, depth, etc.) of an image. In this regard, Non-Patent Documents 2 and 3 disclose a method of estimating a region of interest by extracting higher-order features from an image using a neural network.

Furthermore, in recent years, super wide-angle cameras such as fish-eye cameras having an angle of view of more than 180 degrees and omnidirectional cameras capable of shooting 360 degrees in all directions have been widely used. There is a request to estimate the area.

The present invention has been made in view of the above, and an object thereof is to provide an image analysis apparatus capable of accurately estimating a region of interest (point of interest) from an ultra-wide-angle image.

As a result of earnest studies on an image analysis apparatus capable of accurately estimating a region of interest (point of interest) from an ultra-wide-angle image, the present inventor conceived the following configuration and arrived at the present invention.

That is, according to the present invention, there is provided an image analysis device for extracting a point of interest from an input image, the element feature extraction unit extracting the element feature at each position of the input image, and dividing the input image into a plurality of regions. Then, an area feature calculation unit that calculates the area feature by integrating the element features for each divided area, and an attention point that calculates the attention point of the input image based on a predetermined regression model from the calculated area feature An image analysis apparatus including a regression unit is provided.

As described above, according to the present invention, there is provided an image analysis device capable of accurately estimating a region of interest (point of interest) from an ultra wide-angle image.

A conceptual diagram for explaining an image in the Equirectangular format (equidistant cylinder projection). The functional block diagram of the image analysis apparatus of 1st Embodiment. 3 is a flowchart showing a process executed by the image analysis device of the first embodiment. FIG. 4 is a conceptual diagram for explaining a process executed by an element feature extraction unit. FIG. 4 is a conceptual diagram for explaining a process executed by an element feature extraction unit. FIG. 6 is a conceptual diagram for explaining a process executed by a region feature calculation unit. FIG. 4 is a conceptual diagram for explaining a process executed by an element feature extraction unit. The functional block diagram of the image analysis apparatus of 2nd Embodiment. The flowchart which shows the process which the image analysis apparatus of 2nd Embodiment performs. The functional block diagram of the image analysis apparatus of 3rd Embodiment. The flowchart which shows the process which the image analysis apparatus of 3rd Embodiment performs. The functional block diagram of the image analysis apparatus of 4th Embodiment. The flowchart which shows the process which the image analysis apparatus of 4th Embodiment performs. The hardware block diagram of the image analysis apparatus of this embodiment.

Hereinafter, the present invention will be described with reference to embodiments, but the present invention is not limited to the embodiments described below. In each drawing referred to below, common elements are denoted by the same reference numerals, and description thereof will be appropriately omitted.

An image analysis apparatus according to an embodiment of the present invention has a function of extracting a region of interest from an input image, and more specifically, estimates a point of interest (a point within the region of interest or a center of gravity of the region of interest). It has a function to do. Here, before entering the description of the image analysis apparatus of the present embodiment, if the conventional ROI extraction technique is applied to a super wide-angle image (image captured by a fisheye camera or an omnidirectional camera), the ROI The reason why can not be extracted accurately will be described.

First of all, a method of converting the super wide-angle image into an image of the Equirectangular format (equidistant cylinder projection) shown in FIG. 1 and extracting the region of interest from the converted image can be considered. Here, the Equirectangular format is an image representation format mainly used for panoramic photography, and as shown in FIG. 1, the three-dimensional directions of pixels are decomposed into latitude and longitude, and the pixel values corresponding to a square lattice are calculated. It is a side-by-side image format. From the image in the Equirectangular format, it is possible to obtain pixel values in arbitrary three-dimensional directions from the coordinate values of longitude and latitude, and conceptually, it can be considered that the pixel values are plotted on the unit sphere.

However, when extracting the region of interest directly from the image in the Equirectangular format, there is a problem that the region near the zenith/nadir where the distortion becomes extremely large or the region of interest existing at the image boundary cannot be extracted.

Secondly, a method of dividing the super wide-angle image into a plurality of images and extracting the region of interest from each divided image can be considered. However, in this case, it is not clear how to integrate the saliency maps obtained from the divided images.

Further, in the case of an ultra wide-angle image, it is assumed that a plurality of highly prominent objects are included in one image, but the prior art does not have a mechanism for determining the priority order among a plurality of objects. ..

Although the problem of the conventional region-of-interest extraction technology has been described above, the image analysis device of the present embodiment has a large distortion, and an ultra-wide-angle image including a plurality of objects accurately addresses the user's interest. It is characterized by having a function of extracting a region. Hereinafter, a specific configuration of the image analysis apparatus of this embodiment will be described.

(First embodiment)
The image analysis apparatus 100A according to the first embodiment of the present invention has a function of dividing an image to be processed into a plurality of areas and estimating a point of interest of the image to be processed from the characteristics of each divided area. Hereinafter, the functional configuration of the image analysis apparatus 100A of the present embodiment will be described based on the functional block diagram shown in FIG.

As shown in FIG. 2, the image analysis device 100A includes an image input unit 101, an element feature extraction unit 102, a region feature calculation unit 103, an attention point regression unit 104, and an attention point output unit 105. To be done.

The image input unit 101 is means for inputting an image to be processed.

The element feature extraction unit 102 is means for extracting the element feature at each position of the image to be processed.

The area feature calculation unit 103 is a unit that divides an image to be processed into a plurality of areas and integrates the element features for each of the divided areas to calculate the area features.

The attention point regression unit 104 is means for calculating the attention point of the image to be processed from the calculated region features based on a predetermined regression model.

The attention point output unit 105 is means for outputting the calculated attention point.

In the present embodiment, the computer configuring the image analysis apparatus 100A executes a predetermined program, so that the image analysis apparatus 100A functions as each of the above-mentioned units.

The functional configuration of the image analysis apparatus 100A according to the present embodiment has been described above. Next, the content of processing executed by the image analysis apparatus 100A will be described based on the flowchart shown in FIG.

First, in step 101, the image input unit 101 reads and inputs an omnidirectional image in the Equirectangular format to be processed from an arbitrary storage means. Hereinafter, the input image will be referred to as an “input image”.

In the following step 102, the element feature extraction unit 102 extracts the element feature from each position of the input image read in the previous step 101. The element feature may be extracted for each pixel of the input image or may be extracted from a specific sampling position.

In the present embodiment, color, edge, saliency, object position/label, etc. can be used as element features.

As the color feature, a value in a specific color space (RGB, L*a*b*, etc.), a Euclidean distance from a specific color (for example, skin color), a Mahalanobis distance, or the like can be used.

As the edge feature, the direction and strength of the pixel value gradient extracted by the Sobel filter or Gabor filter can be used.

As the saliency, a saliency value extracted by an existing saliency extraction algorithm can be used. Examples of the saliency extraction algorithm here include the algorithms disclosed in the above-mentioned Patent Documents 3 to 5 and Non-Patent Documents 1 to 3.

As the object position/label feature, use the position of the object detected by a known object detection algorithm (usually represented by the coordinates of the four corners of the detection rectangle) and the object type (face, person, car, etc.). You can Here, as an example of the object detection algorithm, the algorithms disclosed in the above-mentioned Patent Documents 1 and 2 can be cited.

Note that the element features that can be adopted in the present embodiment are not limited to the above, and other feature amounts conventionally used in the field of image recognition (LBP, Haar like feature, HOG, SIFT, It goes without saying that it is acceptable to adopt (for example).

Here, in the present embodiment, from the viewpoint of feature extraction accuracy, element features are extracted by the following method.

As shown in FIG. 1, from an image in Equirectangular format, a pixel value in an arbitrary three-dimensional direction can be obtained from coordinate values of longitude and latitude, and an image in Equirectangular format conceptually has pixel values in a unit sphere. It can be considered as plotted. Therefore, in the present embodiment, as shown in FIG. 4, a predetermined projection plane is defined, the center of the unit sphere is set as the projection center O, and the pixel value of the omnidirectional image in the Equirectangular format (θ , Φ) is associated with the pixel value (x, y) on the defined projection surface, and the element feature is extracted from the image subjected to the perspective projection conversion. In the following formula (1), P indicates a perspective projection matrix, and the equal sign indicates equality with a scalar multiple other than 0.

Specifically, a regular polyhedron having a common center with the unit sphere is defined as the projection surface of the omnidirectional image of the Equirectangular format, and perspective projection conversion is performed with the normal direction of each surface as the line-of-sight direction. FIG. 5A shows an example in which a regular octahedron is defined as the projection surface of the omnidirectional image, and FIG. 5B shows an example in which a regular dodecahedron is defined as the projection surface of the omnidirectional image.

Returning to FIG. 3 again, the description will be continued.

In the following step 103, the region feature calculation unit 103 divides the input image (omnidirectional image) into a plurality of regions by spatially equally dividing the shooting direction of the input image, and then extracts the divided regions from the respective divided regions. The obtained element features are integrated, and the integrated value for each region is calculated as the region feature. For example, as shown in FIG. 5, when the spherical surface of the omnidirectional image is approximated by a regular polyhedron, the integrated value of the element features extracted from the perspective projection conversion image in which each face of the regular polyhedron is the projection surface is the regional feature. Becomes When the color feature composed of RGB is used as the element feature, the values of R, G, and B are integrated in each divided area.

FIG. 6 exemplarily shows a case where the area feature is calculated by using three types of element features of edge strength, saliency, and object position (face distribution). As described above, when the area feature is calculated using two or more types of element features, the number of calculated area features=the number of divided areas×the number of types of element features.

In the following step 104, the attention point regression unit 104 calculates the position of the attention point from the area feature calculated in the previous step 103 using a predetermined regression model prepared in advance. Here, the position y of the point of interest can be expressed by the following equation (2).

In the above formula (2), x represents a region feature vector, f represents a regression model, and α represents a regression parameter. The regression parameter α is identified in advance by machine learning using training data (a plurality of sets of x and y). For regression, known regression methods such as linear regression, logistic regression, support vector regression, random forest regression, and neural network can be used.

Hereinafter, the case of using the support vector regression will be described as an example.

In this case, the regression parameter α is the support vector {s _i }, the weight of the support vector {w _i }, and the offset h (actually, in addition to this, the kernel type and kernel parameters exist as hyperparameters). .. Position y of the target point is represented by a unit direction in 3-dimensional space _{(e x, e y, e} z), a regression model from the region feature vector x with respect to e _x, e y, _{e z} respectively To do. In this case, the regression model f can be expressed by the following equation (3). In the formula (3) below, K represents a kernel.

Finally, in step 105, the attention point output unit 105 outputs the position of the attention point calculated in the previous step 104, and the process ends.

When the present embodiment is applied to cropping or thumbnail generation, an area of interest is defined by setting a specific angle of view centered on the point of interest obtained in the procedure described above, and the image of the defined area of interest is used as it is. Use as cropping images or thumbnail images. In this case, the angle of view to be set is preferably the angle of view of the region of interest including the point of interest in the training data given to the regression model. Further, when the present embodiment is applied to the image recognition/image search system, the object region including the point of interest is set as the recognition target object and the search target object.

As described above, in the present embodiment, since the element feature is calculated after decomposing the image into partial images (divided regions) with less distortion, it is possible to robustly process an ultra wide-angle image exceeding 180 degrees. It will be possible.

Further, in the present embodiment, the saliency map and the object distribution obtained from each partial image are not simply integrated, but the attention point is estimated based on the regression model from the area features aggregated for each divided area. , A region crossing rule exists such that an X object exists in the area A and a Y object exists in the area B, and the C point is the point of interest in the regression model in machine learning. By doing so, it becomes possible to estimate the point of interest in consideration of the interaction of features between regions.

In addition, in the above-described first embodiment, the following design changes can be made.

For example, for the area division of the input image at the time of calculating the area features in the previous step 103, an arbitrary division method can be adopted other than the method of approximating the spherical surface of the omnidirectional image with a regular polyhedron. For example, the sphere of the omnidirectional image may be approximated and divided by a quasi-regular polyhedron, or may be divided by Voronoi division based on a generating point randomly developed on the sphere of the omnidirectional image. It should be noted that the division method in the perspective perspective transformation for performing the element feature and the division method in the region division for the region feature calculation do not necessarily have to match, but from the viewpoint of calculation cost reduction, they do not. Preferably.

Further, the target image of the element feature extraction in the previous step 102 is not limited to the image obtained by perspective projection conversion of the omnidirectional image, and may be an image projected by another projection method. For example, it may be an orthographically projected image, or may be an image obtained by performing perspective projection conversion by shifting the projection center O from the center of the unit sphere as shown in FIGS. 7A and 7B. good. According to the projection method shown in FIGS. 7A and 7B, it is possible to reduce the projective distortion at the image end and to project the image with a field angle of 180 degrees or more. It becomes possible to extract features.

Further, when processing an image captured by a camera whose angle of view does not reach 360 degrees, the image obtained by converting the image of the angle of view within the range to the Equirectangluar format (partially missing image) is described above. The procedure may be the same as that described above.

Furthermore, even if the processing target is not an image in Equirectangular format, as long as the camera that captured the image is calibrated (that is, the direction of the ray in the three-dimensional space corresponding to the position of the camera imaging plane is known). , Can be handled in the same manner as described above. If the processing target is an image taken by an uncalibrated camera, the method of approximating and dividing the image by a regular polyhedron cannot be applied, but in that case, another applicable division method (for example, the above-mentioned Area division).

The first embodiment of the present invention has been described above, and then the second embodiment of the present invention will be described. It should be noted that in the following, description of the parts common to the contents of the first embodiment will be omitted, and only the differences from the first embodiment will be explained.

(Second embodiment)
The image analysis apparatus 100 according to the second embodiment has a function of integrating different types of element features within a region and estimating a point of interest of an input image from the integrated region features.

FIG. 8 shows a functional block diagram of the image analysis apparatus 100B. As shown in FIG. 8, the functional configuration of the image analysis apparatus 100B is the same as that of the image analysis apparatus 100A of the first embodiment, except that the area feature integration unit 110 is additionally provided.

Here, the area feature integration unit 110 is means for mapping the area features to lower-dimensional features to obtain integrated area features.

The contents of the processing executed by the image analysis apparatus 100B will be described below with reference to the flowchart shown in FIG.

Since the contents of steps 101 to 103 are the same as those of steps 101 to 103 described above with reference to FIG. 3, the description thereof will be omitted. Here, step 110 will be described.

In step 110, the area feature integration unit 110 integrates the area features calculated in the previous step into lower dimensional features. Here, the area feature integration unit 110 obtains a low-dimensional integrated area feature vector x _i ′ by the mapping g with respect to the area feature vector x _i of the area i as shown in the following expression (4). In the present embodiment, the mapping g is designed in advance or identified by machine learning.

In the following step 104, the attention point regression unit 104 calculates the position of the attention point from the integrated area feature vector x _i ′ obtained in the previous step 110, using a predetermined regression model prepared in advance. Here, the position y of the point of interest can be expressed by the following equation (5).

In addition, {x _i '} in the above formula (5) indicates that if there are S areas, the following formula (6) is given.

Here, the mapping g will be described.

The simplest mapping g is a mapping that adds all the elements of the region feature vector x _i . In this case, the region feature vector x _i is aggregated up to one dimension.

As another example, a linear conversion W from R ⁿ to R ^m (m<n) can be adopted as the mapping g as shown in the following formula (7).

The linear transformation W can be acquired by machine learning when a set of a region feature vector x and a position y of a point of interest is given as training data. That is, when the mapping f from the integrated region feature x _i ′ to the position y of the target point is determined, {x _i ′} that satisfies the above equation (5) is obtained for y of the training data, and {x _i The mapping g (that is, the matrix W) can be obtained by learning from the set of'} and x. When the mapping f is not determined, f and g can be obtained by repeating the process of learning g for the temporarily determined f and learning f for the learned g. Here, when both f and g are linear transformations and W is R ⁿ →R, f can be expressed by the matrix V as shown in the following expression (8).

If the above equations (8) and (7) are arranged, the whole can be expressed by the following equations (9) and (10).

Here, in the above equation (9), a process of fixing V and seeing W as a linear regression from VX ^T to y, and obtaining W by fixing W and seeing V as a linear regression from WX to y By repeating the above, V and W, that is, f and g can be obtained.

It is also possible to consider something other than linear transformation as the mapping g. After all, finding g is to solve a regression problem, and similarly to f, a known regression method such as support vector regression, random forest regression, or neural network can be used.

Finally, in step 105, the attention point output unit 105 outputs the position of the attention point calculated in the previous step 104, and the process ends.

As described above, according to the present embodiment, the parameters of the regression model can be reduced by aggregating the region features into a smaller number of integrated region features. Taking linear regression as an example, in the first embodiment, the number of parameters proportional to “(number of element features)×(number of area divisions)” was required, whereas in the first embodiment, “(element features) The number of parameters can be reduced to a number proportional to (number)+(area division number)”. In the case of nonlinear regression, the same parameter reduction effect can be obtained. As a result, it is possible to suppress overfitting that occurs when a regression model is obtained, and it becomes possible to accurately estimate the point of interest from a small amount of training data.

The second embodiment of the present invention has been described above, and then the third embodiment of the present invention will be described. It should be noted that in the following, description of the parts common to the contents of the first embodiment will be omitted, and only the differences from the first embodiment will be explained.

(Third Embodiment)
The image analysis apparatus 100C of the third embodiment has a function of calculating a region feature for a soft segmented region and estimating a point of interest in the input image.

FIG. 10 shows a functional block diagram of the image analysis apparatus 100C. As shown in FIG. 10, the functional configuration of the image analysis apparatus 100C is the same as that of the image analysis apparatus 100A of the first embodiment except that the area feature calculation unit 120 is provided instead of the region feature calculation unit 103.

Here, the area feature calculation unit 120 is means for calculating the area feature by performing weighted addition of the weighting function and the element feature according to the position for each area.

The contents of the processing executed by the image analysis device 100C will be described below with reference to the flowchart shown in FIG.

The contents of Steps 101 to 102 are the same as those of Steps 101 to 102 described above with reference to FIG. 3, and therefore a description thereof will be omitted. Here, Step 120 will be described.

In step 120, the area feature calculation unit 120 calculates the area feature by integrating the element features extracted in the previous step 102 for each area. In the present embodiment, there is an overlap between adjacent regions, and the probability of belonging P(i|q) to the region i is defined for the position q=(XYZ) ^T on the unit spherical surface. Here, the center coordinates of the region can be set by the surface center of the polyhedron or by random generation as in the first embodiment. The belonging probability P(i|q) can be set, for example, as shown in the following formula (11) with the center coordinate of the region i as c _i (unit vector).

In the above equation (11), β is a parameter, and the smaller β is, the softer the segmentation becomes. However, the above formula (11) is an example, and the belonging probability P is not limited to this form and can be freely designed.

In the present embodiment, based on the above settings, the region feature calculation unit 120 calculates the region feature x _i by performing weighted addition of the weighting function and the element feature according to the position for each region. Specifically, the region feature x _i is obtained by weighting the element features at the position q for each region with the belonging probabilities and integrating them. More specifically, with respect to the element feature vector a(q) at the position q, the area feature x _i in the area i is obtained by the following equation (12).

Here, it can be seen that the above formula (12) is a generalization of the first embodiment. That is, the first embodiment can be regarded as a special example (hard segmentation) in which the belonging probability P(i|q) takes only 0 or 1 in the above formula (12).

Further generalizing away from the probability, the area feature x _i can be obtained by the following formula (13) using an arbitrary function h _i (q).

In the present embodiment, a spherical harmonic function can be used as h _i (q) in the above equation (13).

In the following step 104, the attention point regression unit 104 calculates the position of the attention point from the region feature calculated in the previous step 103 by using a predetermined regression model prepared in advance, and finally, in step 105, The point output unit 105 outputs the position of the point of interest calculated in the previous step 104, and the process ends.

As described above, according to the present embodiment, it is possible to reduce the error due to the discretization of the region and estimate the target point with higher accuracy by soft segmenting the region.

The third embodiment of the present invention has been described above, and then the fourth embodiment of the present invention will be described. It should be noted that in the following, description of the parts common to the contents of the first embodiment will be omitted, and only the differences from the first embodiment will be explained.

(Fourth Embodiment)
The image analysis apparatus 100D of the fourth embodiment has a function of estimating a plurality of points of interest from an input image.

FIG. 12 shows a functional block diagram of the image analysis device 100D. As shown in FIG. 12, the functional configuration of the image analysis device 100D is the element feature integration unit 130 and the attention point search instead of the area feature calculation unit 103 and the attention point regression unit 104 of the image analysis device 100A of the first embodiment. It is the same except that the unit 140 is provided.

Here, the element feature integrating unit 130 is a unit that obtains an integrated element feature by integrating the element features at each position of the input image into one value, and the attention point searching unit 140 uses the integrated element feature and a predetermined window function. It is a means for calculating one or more points of interest as a local solution of the evaluation function consisting of the sum of products.

The contents of the processing executed by the image analysis device 100D will be described below with reference to the flowchart shown in FIG.

The contents of Steps 101 to 102 are the same as those of Steps 101 to 102 described above with reference to FIG. 3, and thus the description thereof will be omitted. Here, Step 130 will be described.

In step 130, the element feature integration unit 130 combines the element features. In the present embodiment, element feature vectors obtained for each position q are integrated into a one-dimensional value by the same method as in the second embodiment. That is, in the second embodiment, the element feature is integrated for each area, but in the present embodiment, it is integrated for each position. The integration method is determined in advance by using the learning method described in the second embodiment.

In the following step 140, the attention point search unit 140 searches the position of the attention point. Specifically, using the window function ψ for the one-dimensional value b(q) obtained by collecting the element feature vectors for each position q obtained in the previous step 130, the evaluation shown in the following equation (14) Construct the function J(p).

In the present embodiment, one or more local solutions p in which the value of the evaluation function J(p) is equal to or greater than the threshold value are obtained, and this is set as a point of interest. As the window function, a δ function or a Gaussian function can be used.

Finally, in step 105, the attention point output unit 105 outputs the position of the attention point calculated in the previous step 104, and the process ends.

As described above, according to this embodiment, a plurality of points of interest can be estimated from the input image.

Finally, the hardware configuration of the computer configuring the image analysis apparatus 100 of this embodiment will be described based on FIG.

As shown in FIG. 14, the computer configuring the image analysis apparatus 100 of the present embodiment provides a processor 10 that controls the operation of the entire apparatus, a ROM 12 that stores a boot program, a firmware program, and the like, and a program execution space. RAM 14 for storage, an auxiliary storage device 15 for storing a program or an operating system (OS) for causing the image analysis device 100 to function as each of the above-described means, and an input/output interface 16 for connecting an external input/output device. And a network interface 18 for connecting to a network.

Each function of the above-described embodiment can be realized by a program described in C, C++, C#, Java (registered trademark), etc., and the program of the present embodiment is a hard disk device, CD-ROM, MO, DVD. It can be stored in a recording medium such as a flexible disk, an EEPROM, or an EPROM for distribution, and can be transmitted via a network in a format that can be used by another device.

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments, and within the scope of the embodiments conceivable to those skilled in the art, as long as the operations and effects of the present invention are exhibited. Are included in the scope of the present invention.

10... Processor 12... ROM
14... RAM
15... Auxiliary storage device 16... Input/output interface 18... Network interface 100... Image analysis device 101... Image input unit 102... Element feature extraction unit 103... Region feature calculation unit 104... Attention point regression unit 105... Attention point output unit 110 Area feature integration unit 120 Area feature calculation unit 130 Element feature integration unit 140 Attention point search unit

Japanese Patent No. 4538008 Japanese Patent No. 3411971 Japanese Patent No. 5158974 Japanese Patent No. 5766620 Japanese Patent No. 5865078

L. Itti, et al., "A model of saliency-based visual attention for rapid scene analysis," IEEE Transactions on Pattern Analysis & Machine Intelligence 11 pp. 1254-1259, 1998. R. Zhao, et al., "Saliency detection by multi-context deep learning," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. X. Huang, et al., "SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks," Proceedings of the IEEE International Conference on Computer Vision. 2015.

Claims (13)

An image analysis device for extracting a point of interest from an input image,
An element feature extraction unit that extracts element features at each position of the input image,
An area feature calculation unit that divides the input image into a plurality of areas and integrates the element features for each of the divided areas to calculate area features,
An attention point regression unit that calculates the attention point of the input image based on a predetermined regression model from the calculated region features,
Only including,
The area feature calculation unit divides the input image by spatially dividing the shooting direction of the input image so as to correspond to each surface of a polyhedron that approximates the spherical surface of the input image,
Image analysis device. The image analysis device according to claim 1, wherein the element feature extraction unit extracts the element feature based on a projection conversion image obtained by projecting the input image onto each surface of the polyhedron. The image analysis apparatus according to claim 1, wherein the polyhedron is a regular polyhedron. Further comprising a region feature integration unit that maps the region feature to a lower dimensional feature to obtain an integrated region feature,
The point of interest regression section,
Calculating the point of interest based on the regression model from the integrated region feature,
The image analysis apparatus according to claim 1. The area feature calculation unit,
Calculating the region feature by weighted addition of the weighting function and the element feature according to the position for each region,
The image analysis apparatus according to any one of claims 1 to 4 . The regression model is selected from the group consisting of linear regression, logistic regression, support vector regression, random forest regression and neural networks,
The image analysis apparatus according to any one of claims 1-5. 7. The image analysis apparatus according to claim 1, wherein the element feature is at least one element feature selected from the group consisting of color, edge, saliency, and object position/label. A method of extracting a point of interest from an input image,
Extracting the element features at each position of the input image,
Dividing the input image into a plurality of regions, and integrating the element features for each divided region to calculate a region feature,
Calculating a point of interest of the input image from the calculated region features based on a predetermined regression model;
Only including,
The step of calculating the region feature includes the step of dividing the input image by spatially dividing the shooting direction of the input image so as to correspond to each surface of a polyhedron that approximates the spherical surface of the input image. ,
Method. Further comprising mapping the region features to lower dimensional features to obtain integrated region features,
The step of calculating the point of interest includes
9. The method of claim 8 including the step of calculating the point of interest from the integrated region features based on the regression model. The step of calculating the region feature includes
A step of calculating the area feature by performing weighted addition of a weighting function and element features according to the position for each area,
The method of claim 8 . The regression model is selected from the group consisting of linear regression, logistic regression, support vector regression, random forest regression and neural networks,
The method according to any one of claims 8 to 10 . The method according to any one of claims 8 to 12 , wherein the element feature is at least one element feature selected from the group consisting of color, edge, saliency, object position/label. The computer program for executing the steps of the method according to any one of claims 8-12.