JP2009003615A - Attention region extraction method, attention region extraction device, computer program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract an attention region under the consideration of the temporal variation of the degree of attention. <P>SOLUTION: This attention region extraction method includes: a process for extracting a basic attention degree image displaying a space region having remarkable characteristics in the certain frame of an input video; a process for suppressing the maximum basic attention degree region as a region whose basic attention degree is the highest of the basic attention degree image extracted from a frame previous to a current frame in the basic attention degree image extracted from the current frame, and for extracting a post-instantaneous suppression attention degree image or a process for extracting a region having a remarkable value in a time-axis direction about the basic attention degree image calculated from several frames before the current frame, and for emphasizing the extracted region, and for suppressing the other region to extract a post-gradual suppression attention degree image in the basic attention degree image or an instantaneous transition attention degree image; or both of those processes; and a process for extracting the post-gradual suppression attention degree image as the time series of the post-instantaneous suppression attention degree image or the post-gradual suppression attention degree video, and for outputting this as an attention degree video. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an attention area extraction method, an attention area extraction device, a computer program, and a recording medium that extract an attention area in a video signal.

  With the spread of digital cameras, the increase in recording media capacity, the price reduction, and the proliferation of large-capacity networks, a technique for storing and retrieving and using various images / videos has become necessary. Above all, technology that quickly extracts similar images and videos from colors, shapes, compositions, and patterns that are difficult to express in words requires development in a wide range of fields such as shopping sites and video sharing sites. ing.

In the implementation of the search technology as described above, in order to reflect the search intention of the user, the degree of importance that a person feels important in the image / video is extracted as the attention level, and the similarity of the images based on the extracted attention level A promising method for gender determination is promising. As a technique for calculating the degree of attention, a method described in Non-Patent Document 1 and Non-Patent Document 2 is known, and as a technique for determining image similarity based on the degree of attention, a method described in Patent Document 1 is known. Yes.
JP 2006-338313 A L. Itti et al., “A model of saliency-based visual attention for rapid scene analysis”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 20, Number 11, pp. 1254-1259, November 1998. L. Itti et a1, “Realistic avatar eye and head animation using a neurobiological model of visual attention”, Proceedings of SPIE International Symposium on Optical Science and Technology, Volume 5200, pp. 64-78, August 2003.

  However, in the method described in the above document, the aspect in which the degree of attention varies with time is hardly taken into consideration. The present invention has been made in view of such circumstances, and the purpose thereof is an attention area extraction method, an attention area extraction device, a computer program, and a new attention degree calculation method that takes into account temporal variations. And providing a recording medium.

  That is, in the present invention, the attention level calculation method considering the temporal variation of the attention level is realized by the following two points.

(1) Introduction of instantaneous movement of an area where the degree of attention is maximum.
In order to grasp the information of the entire visual field as soon as possible in human early vision, a phenomenon called “visual search” in which the point of interest moves at high speed is known (RM Klein: “Inhibition of return,” Trends in Cognitive Sciences , Vol.4, No.4, pp.138-147, April 2000 .; In the present invention, this “visual search” is introduced in the calculation of the degree of attention.

(2) Introduction of suppression of attention to visual stimuli with small fluctuations.
In early human vision, a phenomenon called “visual adaptation” is known in which the degree of attention to an area with a small visual stimulus decreases with time in order to grasp information of the entire visual field with as little gaze point movement as possible ( S. Martinez-Conde, SL Macknik and DH Hubel: “The role of fixational eye movements in visual perception,” Nature Reviews, Vol. 5, pp. 229-240, March 2004 .; In the present invention, this “visual adaptation” is introduced in the calculation of the degree of attention.

  In order to solve the above-mentioned problems, the present invention provides a region-of-interest extraction method for extracting a degree-of-interest video, which is a video displaying a spatio-temporal region having remarkable characteristics in the input video, from a target input video. A basic attention level image extraction process for extracting a basic attention level image, which is an image displaying a spatial region having a remarkable characteristic in the frame from a certain frame constituting the input video, and the basic attention level image In the basic attention level image extracted from the previous frame of the input video by the extraction process, the maximum basic attention level area that is the area where the basic attention level that is the value of each pixel is the largest is the basic attention level area. An attention level instantaneous suppression process of extracting an attention level image after instantaneous suppression by suppressing in a basic attention level image extracted from the current frame of the input video by an image extraction process, or An area having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video of the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Accordingly, one or both of the attention degree gradual suppression processes for extracting the attention degree image after gradual suppression, the basic attention degree image extraction process, one of the attention degree instantaneous suppression process or the attention degree gradual suppression process, or By repeating both steps in order for each frame of the input video, Proceed to extract the series progressive suppression after prominence image is a time of suppression after prominence image, a region of interest extraction method and having a degree of attention video output process of outputting this as attention image.

  In the above invention, an instantaneous movement of a region where the degree of attention is maximized is introduced by the process of instantaneous attention degree suppression, and the “visual search” phenomenon is realized by signal processing. In addition, the attention degree gradual suppression process is used to introduce attention degree suppression for visual stimuli with small fluctuations, and the “visual adaptation” phenomenon is realized by signal processing.

  Further, the present invention is the attention area extraction method described above, wherein the basic attention image extraction process includes a basic feature image extraction process for extracting a plurality of types of basic feature images from a frame of the input video, and the basic For each type of basic feature image extracted by the feature image extraction process, a multi-resolution image extraction process for extracting a multi-resolution image, which is a multi-resolution expression, and each of the multi-resolution images extracted by the multi-resolution image extraction process A resolution difference image extraction process for extracting a plurality of resolution difference images, which are differences between images having different resolutions, and a resolution difference having a different resolution for each type of resolution difference image extracted by the resolution difference image extraction process. By integrating the images, the saliency image extraction process for extracting the saliency image and the saliency image extraction process A saliency image integration process for extracting a basic attention level image by integrating a plurality of types of saliency images with respect to the issued saliency image, and the attention degree gradual suppression process is replaced with a basic attention level image. The saliency image is extracted using the progressive suppression image.

  Further, the present invention is the above-described attention area extraction method, further extracting the maximum basic attention degree area from the basic attention degree image extracted by the basic attention degree image extraction process, and a plurality of types of saliency images. A saliency image integration ratio calculating step of calculating a value in an area corresponding to the maximum basic attention degree area and determining a saliency image integration ratio that is a weight of the corresponding saliency image from the magnitude of the value In the basic attention level image extraction process, the saliency image is weighted and integrated by the saliency image integration ratio calculated by the saliency image integration ratio calculation process for the previous frame of the input video. Thus, the basic attention degree image is extracted.

  Further, the present invention is the attention area extraction method described above, wherein the saliency image integration ratio calculation step uses the saliency image integration ratio calculated for one previous frame of the input video as an initial value, A new saliency image integration ratio is updated using a value in the maximum basic attention level calculated for each saliency image as a difference value with respect to the initial value.

  Further, the present invention is the above-described attention area extracting method, wherein the attention degree instantaneous suppression process includes a basic attention degree extracted from the previous frame of the input video by the basic attention degree image extraction process. Maximum basic attention area detection process for extracting the maximum basic attention area from the image, and maximum basic attention area shielding that is an image for shielding the maximum basic attention area extracted by the maximum basic attention area detection process Reduction of occlusion in the area of the maximum basic attention area occlusion image extraction process for extracting an image and the area occluded by the maximum basic attention area occlusion image extracted by the maximum basic attention area occlusion image extraction process Extracted by the attention degree progressive recovery image extraction process of extracting the attention degree progressive recovery image that is the image to be captured and the maximum basic attention degree region occlusion image extraction process. The attention level instantaneous suppression image generation process for generating the attention level instantaneous suppression image by integrating the maximum basic attention level region occlusion image and the attention level progressive recovery image extracted by the attention level progressive recovery image extraction process, After instantaneous suppression by integrating the instantaneous attention suppressed image generated by the attention instantaneous suppression image generation process and the basic attention image of the current frame of the input video extracted by the basic attention image extraction process It is characterized by comprising an attention degree image generation process after instantaneous suppression for generating an attention degree image.

  Further, the present invention is the attention area extraction method described above, wherein the attention degree gradual suppression process is an attention degree that is an image that gradually blocks the basic attention degree image extracted by the basic attention degree image extraction process. Attention level progressive occlusion image generation process for generating a progressive occlusion image, and a basic attention level image or saliency image calculated from several frames before the current frame of the input video, and a remarkable value in the time axis direction Attention instant recovery that extracts a region of interest and generates an attention degree instantaneous recovery image that is an image for releasing the suppression of the basic attention degree of the region in the basic attention degree image or the instantaneous attention degree attention image corresponding to the region Image generation process, attention degree progressive occlusion image generated by the attention degree progressive occlusion image generation process, and attention generated by the attention degree instantaneous recovery image generation process The attention degree gradual suppression image generation process for generating the attention degree gradual suppression image by integrating the instantaneous recovery images, the basic attention degree image extracted by the basic attention degree image extraction process, or the attention degree instantaneous suppression process Generating attention degree image after progressive suppression by integrating the extracted attention degree image after instantaneous suppression and the attention degree progressive suppression image generated in the attention degree progressive suppression image generation process. It consists of a process.

  The present invention is also a region-of-interest extraction apparatus that extracts a video of attention level, which is a video displaying a spatio-temporal region having remarkable characteristics in the input video, from the target input video, A basic attention level image extraction unit that extracts a basic attention level image that is an image displaying a spatial region having a remarkable characteristic in the frame from a certain frame that constitutes the frame, and the input video by the basic attention level image extraction unit In the basic attention level image extracted from the previous frame from the current frame, the maximum basic attention level area, which is the area having the highest basic attention level as the value of each pixel, is input by the basic attention level image extraction unit. An attention level instantaneous suppression unit that extracts an attention level image after instantaneous suppression by suppressing the basic attention level image extracted from the current frame of the video, or the current frame of the input video For the basic attention degree image calculated from several previous frames, a region having a remarkable value in the time axis direction is extracted, and the basic attention extracted from the frame of the input video by the basic attention degree image extraction unit In the instantaneous image or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression unit, the extracted region is emphasized and the other regions are suppressed, thereby attracting attention after progressive suppression. The attention degree gradual suppression unit for extracting the degree image and the instantaneous degree of attention suppression image extracted by the attention degree instantaneous suppression unit or the attention degree gradual extraction for each frame of the input video A degree-of-interest video output unit that extracts a degree-of-intermediate attention level image that is a time series of the degree-of-gradual attention level image extracted by the suppression unit, and outputs this Is a region of interest extraction device, characterized in that it comprises.

  The present invention also provides an input to a computer used as an attention area extraction device that extracts an attention degree video that is a video displaying a spatio-temporal area having remarkable characteristics in the input video from the target input video. A basic attention degree image extraction process for extracting a basic attention degree image, which is an image displaying a spatial region having a remarkable characteristic in the frame, from a certain frame constituting the video, and the basic attention degree image extraction process. In the basic attention degree image extracted from the frame immediately before the current input video, the maximum basic attention degree area, which is the area having the largest basic attention degree as the value of each pixel, is obtained by the basic attention degree image extraction process. Attention level instantaneous suppression process of extracting an attention level image after instantaneous suppression by suppressing the basic attention level image extracted from the current frame of the input video, or A region having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the attention degree gradual suppression process for extracting the attention degree image after the gradual suppression, the basic attention degree image extraction process, the attention degree instantaneous suppression process or the attention degree gradual suppression process. Alternatively, by repeating the both processes in order for each frame of the input video, Extract the series progressive suppression after prominence image is a time of gradual suppression after prominence image, a computer program, characterized in that to execute the attention image output step of outputting this as attention image.

  The present invention also provides an input to a computer used as an attention area extraction device that extracts an attention degree video that is a video displaying a spatio-temporal area having remarkable characteristics in the input video from the target input video. A basic attention degree image extraction process for extracting a basic attention degree image, which is an image displaying a spatial region having a remarkable characteristic in the frame, from a certain frame constituting the video, and the basic attention degree image extraction process. In the basic attention degree image extracted from the frame immediately before the current input video, the maximum basic attention degree area, which is the area having the largest basic attention degree as the value of each pixel, is obtained by the basic attention degree image extraction process. Attention level instantaneous suppression process of extracting an attention level image after instantaneous suppression by suppressing the basic attention level image extracted from the current frame of the input video, or A region having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the attention degree gradual suppression process for extracting the attention degree image after the gradual suppression, the basic attention degree image extraction process, the attention degree instantaneous suppression process or the attention degree gradual suppression process. Alternatively, by repeating the both processes in order for each frame of the input video, A computer-readable recording medium that records a computer program that extracts a degree-of-interest attention level video that is a time series of the degree-of-advance attention degree image and outputs the attention degree video as a degree-of-interest video. is there.

  According to the present invention, when extracting a region of interest from an image, the attention of the image is obtained by introducing instantaneous movement of the region where the degree of attention is maximum and suppressing the degree of attention to a visual stimulus with small fluctuation. It is possible to extract a region of interest close to a portion that is felt important by humans in consideration of the temporal change of the degree.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a functional block diagram of a region of interest extraction apparatus according to the first embodiment of the present invention.
The attention area extracting apparatus shown in the figure is composed of a basic attention degree image extraction unit 1, an attention degree instantaneous suppression unit 3, and an attention degree video output unit 5, and is video data that is a target of attention degree extraction. An input video is input, and an attention level video which is a video displaying a spatio-temporal region having a remarkable characteristic with a high level of attention in the input image is output. A video consists of images of a plurality of frames.

The basic attention level image extracting unit 1 inputs an input image that is a frame with an input video, extracts a basic attention level image that is an image displaying a spatial region having a remarkable characteristic in the frame, Output attention level image.
The basic attention level image extraction method is not particularly limited, but in this embodiment, the basic attention level image extraction unit 1 includes a basic feature image extraction unit 11, a multi-resolution image extraction unit 12, and a resolution difference. A feature extraction method in the case where the image extraction unit 13, the saliency image extraction unit 14, and the saliency image integration unit 15 are configured will be described.

The basic feature image extraction unit 11 inputs an input image, extracts basic feature images from the input image by a plurality of feature extraction methods, and outputs a set of the basic feature images.
Although the basic feature image extraction method is not particularly limited, in this embodiment, the basic feature image extraction unit 11 includes a luminance feature image extraction unit 111, a color feature image extraction unit 112, and a direction feature image extraction. A feature extraction method in the case of being configured by the unit 113, the blinking feature image extraction unit 114, and the motion feature image extraction unit 115 will be described.

  The luminance feature image extraction unit 111 receives an input image and outputs a luminance feature image that represents the luminance component of the input image. The luminance feature image is obtained as the following (Equation 1) as an average of RGB (Red, Green, Blue) components of the input image.

  Here, r (i), g (i), and b (i) are R (red), G (green), and B (blue) in the i-th input image (= i-th frame of the input video), respectively. The pixel values are represented by real values from 0 to 1, respectively. I (i) is a luminance feature image calculated from the i-th input image.

  The color feature image extraction unit 112 inputs an input image and outputs a color feature image that represents a color component of the input image. The color feature image extraction unit 112 extracts four types of color feature images corresponding to red, green, blue, and yellow, respectively. Each of these color feature images is obtained as in the following (Expression 2) to (Expression 4). Here, max indicates the maximum value among the values in parentheses {}.

Here, R (i), G (i), B (i), and Y (i) are color feature images calculated from the i-th input image corresponding to red, green, blue, and yellow, respectively. . R (i) _{(x, y)} is a pixel value of the color feature image R (i) at the coordinates (x, y). In the following description, the subscript (x, y) is omitted unless necessary.

  The direction feature image extraction unit 113 inputs an input image and outputs a direction feature image representing the direction component of the input image. The direction feature image is obtained as shown in the following (formula 6) by applying a Gabor filter to the luminance feature image I (i) obtained by the luminance feature image extraction unit 111. The Gabor filter is for extracting local grayscale information of an image.

Here, g _φ is a Gabor filter having a rotation angle φ, and * is an operator that expresses convolution (the other function is added together while moving the function in parallel). O _φ (i) is a direction feature image corresponding to the rotation angle φ calculated from the i-th input image. Direction feature image is extracted for the rotation angle of the street n _phi. At this time, the rotation angle φ is selected as shown in the following (Equation 7) so that π = 180 ° is divided into n _φ evenly.

  The blinking feature image extraction unit 114 inputs an input image and outputs a blinking feature image that represents the blinking component of the input image. The blinking feature image is calculated by the brightness feature image extraction unit 111 using the brightness feature image I (i) calculated from the current and previous input images as shown in (Equation 8) below. .

Here, n _F flashing number of luminance feature image to be referred to when extracting feature image (where luminance feature image extracted from the current input image except), F (i) is the i-th and earlier input It is a blinking feature image calculated from the image. When n _F = 1, F (i) = | I (i) −I (i−1) |, which is consistent with the method described in Non-Patent Document 2.

  The motion feature image extraction unit 115 inputs an input image and outputs a motion feature image representing a motion component of the input image. The method for extracting the motion feature image is not particularly limited, but in the present embodiment, the motion feature image is extracted by obtaining an optical flow at each point of the luminance feature image. The method for deriving the optical flow is not particularly limited. In the present embodiment, for example, a method based on an image gradient generally called the Lucas-Kanade method is used, and the following (formula 9) to (formula 10) are used. Calculate as follows.

Here, A (x, y) is a neighborhood region of coordinates (x, y), and M _x (i) and M _y (i) are motion feature images corresponding to the horizontal and vertical components of motion, respectively.

  As described above, the basic feature image extraction unit 11 sets the luminance feature image, the color feature image, the direction feature image, the blinking feature image, and the motion feature image as basic feature images, respectively. Output a set.

The multi-resolution image extraction unit 12 receives the set of basic feature images, extracts a multi-resolution image that is a multi-resolution representation of each basic feature image, and outputs a set of multi-resolution images.
In the present embodiment, since the same processing is performed for any basic feature image, the processing will be described using a luminance feature image as an example. A luminance multi-resolution image that is a multi-resolution representation of the luminance feature image is extracted as shown in the following (Equation 11) by repeatedly applying a Gaussian filter to the luminance feature image. The Gaussian filter is one of smoothing filters for removing noise from an image, and uses a Gaussian function for weighting based on the distance from the target pixel.

Here, G _σ is a Gaussian filter having variance σ, I (i; l) is the l-th level luminance multi-resolution image extracted from the luminance feature image I (i), and n _l is the number of levels of the multi-resolution image. . The brightness multi-resolution image at the 0th level is the brightness feature image itself, that is, I (i; 0) = I (i).

Multiresolution images can be extracted in the same manner for other basic feature images. At this time, n _l luminance multi-resolution images are extracted, whereas color multi-resolution images when color feature images R (i), G (i), B (i), and Y (i) are used. R (i; l), G (i; l), B (i; l), Y (i; l) are 4n _l in total, and a direction multi-resolution image when the direction feature image _Oφ (i) is used. O _{φ (i;} l) total n _phi n _l Like, blinking multiple resolution images F in the case of using a blinking feature image F (i) (i; l ) is _{n l} Like, the movement feature image _M x (i) , M _y (i), the motion multi-resolution images M _x (i; l) and M _y (i; l) are extracted in total 2n ₁ respectively.

  As described above, the multi-resolution image extraction unit 12 sets the luminance multi-resolution image, the color multi-resolution image, the direction multi-resolution image, the blinking multi-resolution image, and the motion multi-resolution image as multi-resolution images, respectively. Output a set of multi-resolution images.

The resolution difference image extraction unit 13 receives the set of the multi-resolution images output from the multi-resolution image extraction unit 12 and, for each type of multi-resolution image (luminance, color, etc.), the difference between images having different resolution levels. A resolution difference image that is an image is extracted, and a set of resolution difference images is output.
In the present embodiment, the resolution difference image is extracted as in the following (Expression 12) to (Expression 17).

Here, RS _I (i; c, s) is a luminance resolution difference image obtained from the luminance multi-resolution images of the c-th level and the s-th level, and is hereinafter referred to as a (c, s) -level luminance resolution difference image. To. Similarly, RS _RG (i; c, s) and RS _BY (i; c, s) are respectively a (c, s) level RG color resolution difference image, a (c, s) level BY color resolution difference image, and RS _O. (I; φ; c, s) is a (c, s) level direction resolution difference image of the rotation angle φ, RS _F (i; c, s) is a (c, s) level blinking resolution difference image,

(Hereinafter referred to as “RS _Mk (i; c, s)”) is a (c, s) level motion resolution difference image in the k direction. L _c and L _s are sets of resolution levels to be considered when extracting the luminance resolution difference image, and are referred to as a central resolution level set and a peripheral resolution level set, respectively.
At this time, | L _c · L _s | luminance resolution difference images are extracted, whereas 4 | L _c · L _s | color resolution difference images and n _φ | L _c · L _s |, blinking resolution difference images are extracted as | L _c · L _s |, and motion resolution difference images are extracted as 2 | L _c · L _s |

  As described above, the resolution difference image extraction unit 13 sets the luminance resolution difference image, the color resolution difference image, the direction resolution difference image, the blinking resolution difference image, and the motion resolution difference image as resolution difference images, respectively. A set of resolution difference images is output.

The saliency image extraction unit 14 is an image obtained by inputting the set of resolution difference images output from the resolution difference image extraction unit 13 and integrating the resolution difference images for each type of resolution difference image (luminance, color, etc.). A saliency image is extracted, and a set of these saliency images is output.
The extraction method of the saliency image is not particularly limited, but in this embodiment, the saliency image extraction unit 14 includes a resolution difference image normalization unit 141 and a normalized resolution difference image integration unit 142. A method for extracting the saliency image in this case will be described.

The resolution difference image normalization unit 141 receives the set of resolution difference images output from the resolution difference image extraction unit 13 and extracts a normalized resolution difference image that is an image obtained by performing normalization processing on each resolution difference image. Then, a set of normalized resolution difference images is output.
In the present embodiment, since the same processing is performed for any resolution difference image, (c, s) level luminance resolution difference image RS _I (i;) when a certain c∈L _c , s∈L _s is selected. The processing will be described using c, s) as an example. The normalization process for the resolution difference image is performed as follows (Equation 18) to (Equation 20) to obtain a normalized resolution difference image N (RS _I (i; c, s)).

Here, m ^* (RS _I (i; c, s)) is the maximum pixel value in the (c, s) level luminance resolution difference image.

（以下、￣ｍ（ＲＳ_Ｉ（ｉ；ｃ，ｓ））と記載）は、（ｃ，ｓ）レベル輝度解像度差分画像の局所領域における最大値の平均、Ａは、￣ｍ（ＲＳ_Ｉ（ｉ；ｃ，ｓ））を計算する際の局所領域、ｎ_Ａはそのような局所領域の総数である。各局所領域は、例えば、画像を格子状に分割したときの１つの格子（領域）とする。なお、以下では、文字の上に￣や〜が付いた文字は、￣や〜を文字の前に記載して表す（例えば、上述する「￣ｍ」）。

(Hereinafter referred to as ￣m (RS _I (i; c, s))) is the average of the maximum values in the local region of the (c, s) level luminance resolution difference image, and A is ￣ m (RS _I (i C, s)) the local region in calculating n, n _A is the total number of such local regions. Each local area is, for example, one grid (area) when an image is divided into a grid. In the following, a character with ￣ or ~ on the character indicates し て or ~ before the character (for example, “￣m” described above).

  Normalization processing can be performed in the same manner for other levels of luminance resolution difference images and other types of luminance resolution difference images, and the respective normalized resolution difference images, that is, normalized luminance resolution difference images, normal A normalized RG color resolution difference image, a normalized BY color resolution difference image, a normalized direction resolution difference image, a normalized direction resolution difference image, a normalized blinking resolution difference image, and a normalized motion resolution difference image are obtained.

The normalized resolution difference image integration unit 142 inputs the set of normalized resolution difference images obtained by the resolution difference image normalization unit 141, and integrates these normalized resolution difference images for each type (luminance, color, etc.). As a result, a saliency image is extracted, and a set of saliency images is output.
In the present embodiment, the normalized resolution difference image is integrated as in the following (Expression 21) to (Expression 25).

Here, CM _I (i), CM _C (i), CM _O (i), CM _F (i), and _CMM (i) are a luminance saliency image, a color saliency image, a direction saliency image, and blinking, respectively. A saliency image, a motion saliency image, a normalized resolution difference image N (RS _I (i; c, s)), a normalized RG color resolution difference image N (RS _RG (i; c, s)), and a normal BY color resolution difference image N (RS _BY (i; c, s)), normalized direction resolution difference image N (RS _O (i; φ; c, s)), normalized direction resolution difference image N (RS _F (I; c, s)), normalized blink resolution difference image N (RS _F (i; c, s)), normalized motion resolution difference image N (RS _Mk (i; c, s)) Is required.

  As described above, the saliency image extraction unit 14 sets the luminance saliency image, the color saliency image, the direction saliency image, the blinking saliency image, and the motion saliency image as saliency images, respectively. A set of these saliency images is output.

The saliency image integration unit 15 receives the set of saliency images output from the saliency image extraction unit 14, extracts a basic attention image that is an image obtained by integrating the saliency images, and the basic attention image. Is output.
The extraction method of the basic attention level image is not particularly limited, but in the present embodiment, the saliency image integration unit 15 includes a saliency image normalization unit 151 and a normalized saliency image integration unit 152. The basic attention level image extraction method when configured is described.

The saliency image normalization unit 151 receives the set of saliency images output by the saliency image extraction unit 14 and inputs each saliency image (luminance saliency image, color saliency image, direction saliency image, flashing saliency image). Normalized image and motion saliency image), a normalized saliency image that is an image subjected to normalization processing is extracted, and a set of normalized saliency images is output.
The normalization process for the saliency image is the same as the normalization process by the resolution difference image normalization unit 141.

The normalized saliency image accumulation unit 152 receives the set of normalized saliency images output by the saliency image normalization unit 151, and extracts the basic attention level image by integrating the normalized saliency images. The basic attention degree image is output.
In this embodiment, the normalized saliency image N (CM _j (i)) (j = I, C, O, F, M) is integrated as shown in the following (Equation 26). However, although ~ j is used in order to distinguish from j in a formula, it is ~ j = I, C, O, F, and M like j.

Here, S (i) is the basic attention degree image extracted from the i-th input image, and w _j (i) is the saliency extracted by the saliency image integration ratio calculation unit 2 of the third embodiment described later. Although the saliency image integration ratio corresponding to the image CM _j (i) is not used in the present embodiment and the second embodiment described later, the saliency image integration ratio calculation unit 2 is not used, and thus w _j (i) = Let 1/5 (i, j). (∀ indicates optional.)

  As described above, the basic attention level image extraction unit 1 extracts the basic attention level image and outputs it.

  The attention level instantaneous suppression unit 3 inputs the basic attention level image calculated from the current and previous input images by the basic attention level image extraction unit 1 and calculates the basic attention calculated from the previous input image. For the degree image, extract the attention degree image after instantaneous suppression by suppressing the maximum basic attention degree area, which is the area where the basic attention degree is the largest value of each pixel of the basic attention degree image, and after the instantaneous suppression Output attention level image.

  The method of extracting the attention level image after instantaneous suppression is not particularly limited, but in the present embodiment, the instantaneous level suppression unit 3 includes the maximum basic attention level area detection unit 31 and the maximum basic attention level area occlusion image. About the method of extracting the attention level image after instantaneous suppression in the case where the extraction unit 32, the attention degree progressive recovery image extraction unit 33, the attention level instantaneous suppression image generation unit 34, and the instantaneous suppression attention level image generation unit 35 are configured. State.

The maximum basic attention level area detection unit 31 inputs the basic attention level image calculated from the input image one time before, extracts the maximum basic attention level area in the input basic attention level image, and extracts the maximum basic attention level area. Output attention area.
Although the extraction method of the maximum basic attention level region is not particularly limited, in the present embodiment, the maximum basic attention level region MSR (i−1) is used as the pixel value of the basic attention level image S (i−1). Extraction is performed by a circle having a radius ε centered on a point ((￣x), (￣y)) having the largest (basic attention level). That is, it is expressed as (Equation 27) below. Here, argmax is an operator that returns a value that maximizes the immediately following term. For example, in (Expression 27), _{(x ′, y ′)} that maximizes S (i−1) (x ′, y ′) is returned.

The maximum basic attention level region occlusion image extraction unit 32 calculates the basic attention level image calculated from the current input image, the maximum basic attention level region calculated from the current and previous input images, and the attention at the previous time point. An instantaneous attention suppressed image that is an output of the instantaneous instantaneous suppression image generation unit 34 is input, and a maximum basic attention level region occlusion image that is an image that newly shields the maximum basic attention level region is extracted. Output occlusion image.
Although the extraction method of the maximum basic attention level region occlusion image is not particularly limited, in the present embodiment, extraction is performed by the following method.

It is assumed that the attention degree instantaneous suppression image ID (i-1) has already been obtained by the processing in the attention degree instantaneous suppression image generation unit 34 one point before.
Here, it is assumed that the current input image is the i-th input image. At this time, ID (i-1) is updated so as to newly shield the maximum basic attention level area MSR (i-1) one point before the current time point, so that the current maximum basic attention level area shielding image ID ₁ (i ) Is generated as in the following (Expression 28) to (Expression 29). However, mod indicates the remainder of division.

Here, μ (0 <μ ≦ 1) is a coefficient expressing the degree of shielding of the maximum basic attention level region. When μ = 1, the region is determined regardless of the size of the basic attention level in the region. Shield completely. Δt _I ≧ 1 is a constant that controls the interval of shielding by the maximum basic attention degree region. In particular, when Δt _I = 1, all frames are shielded by the maximum basic attention area.

As another embodiment, it is also possible to move the area that is shielded by the maximum basic attention area shadow image ID ₁ (i-1) one point before the time, following the movement of the area. The position MSR (i; k) at the time point i (i ≧ k) of the maximum basic attention level region MSR (k) calculated from the kth input image is expressed by the following (Expression 30) to (Expression 32). Is calculated as follows.

However, MSR (i; i) = MSR (i). At this time, the maximum basic attention level region occlusion image ID ₁ (i) is calculated as in the following (Expression 33) to (Expression 35).

  The first row of the above (Equation 33) is an operation for blocking the maximum basic attention level region one point before, and the second row is a maximum basic attention level region that has already been blocked using the pixel value of the motion feature image. The moving operation, the third line, is an operation for releasing the shielding of the area before moving in the second line.

The attention degree progressive recovery image extracting unit 33 is an attention degree progressive recovery image which is an image for reducing occlusion in an area shielded by the maximum basic attention degree area occlusion image calculated by the maximum basic attention area occlusion image extraction unit 32. , And this attention degree progressive recovery image is output.
In this embodiment, the attention degree progressive recovery image ID ₂ (i) is an image in which all pixel values are α (0 ≦ α ≦ 1).

The attention level instantaneous suppression image generation unit 34 inputs the maximum basic attention level region occlusion image calculated by the maximum basic attention level region occlusion image extraction unit 32 and the attention level progressive recovery image output by the attention level progressive recovery image extraction unit 33. Then, by integrating these images, an attention level instantaneous suppression image is generated, and this attention level instantaneous suppression image is output.
In the present embodiment, the attention degree instantaneous suppression image ID (i) is obtained as shown in (Expression 37) below. Here, min indicates the minimum value among the values in parentheses {}.

The attention-suppressed attention level image generation unit 35 inputs the attention-degree-of-interest instantaneous suppression image and the basic attention-level image, and generates an after-suppression attention level image by integrating these images. Output an image.
In the present embodiment, the after-suppression attention level image S _I (i) is obtained as shown in the following (formula 38).

Here, ω _I (i) ≧ 0 is a coefficient expressing the weight for the attention-degree instantaneous suppression image ID (i).

  As described above, the attention level instantaneous suppression unit 3 extracts the instantaneous suppression level attention level image and outputs it.

  The attention level video output unit 5 is a time series of the attention level images after instantaneous suppression extracted by repeatedly executing the basic attention level image extraction unit 1 to the attention level instantaneous suppression unit 3 for each input image in order. An attention level image after a certain instantaneous suppression is extracted, and this is output as the attention level image.

FIG. 2 shows an operation example of the present embodiment.
In the figure, the upper stage (a) is an input image, the middle stage (b) is an attention level instantaneous suppression image, and the lower stage (c) is an instantaneous suppression attention level image, which are arranged in chronological order from the left.

[Second Embodiment]
FIG. 3 is a functional block diagram of the attention area extracting apparatus according to the second embodiment of the present invention.
The attention area extracting apparatus shown in the present embodiment includes a basic attention degree image extraction unit 1, an attention degree instantaneous suppression unit 3, an attention degree gradual suppression unit 4, and an attention degree video output unit 5. Is input, and an attention level video which is a video displaying a region of high attention level in the input image is output. Note that the attention area extraction device can be configured by the basic attention level image extraction unit 1, the attention level progressive suppression unit 4, and the attention level video output unit 5 without using the attention level instantaneous suppression unit 3. is there. In the figure, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted. The basic attention level image extraction unit 1 and the attention level instantaneous suppression unit 3 are the same as those in the first embodiment.

  The attention degree gradual suppression unit 4 inputs the saliency image calculated from some of the current and previous input images and the basic attention degree image or the instantaneous saliency attention degree image. Extract a region with a remarkable value in the time axis direction, emphasize the region in the basic attention image or the instantaneous attention level image corresponding to that region, and suppress the regions that are not so that the attention after progressive suppression The degree image is extracted, and the attention degree image after the progressive suppression is output.

  The method of extracting the attention degree image after progressive suppression is not particularly limited. In the present embodiment, the attention degree progressive suppression unit 4 includes an attention degree progressive occlusion image generation unit 41 and an attention degree instantaneous recovery image generation unit. 42, the attention degree progressive suppression image generation unit 43 and the progressive suppression attention level image generation unit 44 will be described.

The attention degree progressive occlusion image generation unit 41 generates an attention degree progressive occlusion image that is an image that gradually obstructs the basic attention degree image, and outputs the attention degree progressive occlusion image.
The method of generating the attention degree progressive occlusion image is not particularly limited. In the present embodiment, all pixel values of the basic attention degree image S (i) are represented by β (0 <β ≦ 1) for each time point. ), The attention degree progressive occlusion image GD ₁ (i) is generated as in (Equation 39) below.

The attention level instantaneous recovery image generation unit 42 inputs the saliency images calculated from some input images of the current and previous input images, and sets the saliency images with a remarkable value in the time axis direction. A region of interest is extracted, and an attention level instantaneous recovery image, which is an image for releasing the suppression of the basic attention level of the region in the basic attention level image or the instantaneous suppression level attention level image corresponding to the region, is generated. Instant recovery image is output.
The method of generating the attention level instantaneous recovery image is not particularly limited. In this embodiment, the attention level instantaneous recovery image generation unit 42 includes the time saliency image generation unit 421 and the time saliency image binary. The attention level instantaneous recovery image generation method in the case of being configured by the conversion unit 422 will be described.

The temporal saliency image generation unit 421 inputs the saliency images calculated from some of the current and previous input images, and the saliency images have remarkable values in the time axis direction. A time saliency image that is an image showing a region is generated, and the time saliency image is output.
The method for generating the temporal saliency image is not particularly limited, but in the present embodiment, the following two methods will be described.

  In the first temporal saliency image generation method, L. Itti and P. Baldi, “A principled approach to detecting surprising events in video,” in Proc. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 631-637 , June 2005. (Non-Patent Document 5), a temporal saliency image is generated using the degree of coincidence with a predetermined probability model.

With respect to the probability density function P _γ (λ: ρ ₁ , ρ ₂ ) of the gamma distribution shown in the following (Equation 40), the output of this processing unit using the input image before one time point is used as a parameter of this probability density function. It is assumed that a certain gamma distribution coefficient (ρ ₁ , ρ ₂ ) = (ρ ₁ (i−1), ρ ₂ (i−1)) is determined.

  Here, Γ (·) is a gamma function.

The basic idea of the first temporal saliency image generation method is that the above gamma distribution coefficients ρ ₁ and ρ ₂ are held in a form corresponding to each pixel of each resolution difference image, and the resolution difference at each time point. This is a framework for generating a temporal saliency image while updating the gamma distribution coefficient according to the pixel value of the image. Hereinafter, for the sake of simplicity, the processing will be described by taking a certain (c, s) level luminance resolution difference image RS _I (i; c, s) as an example.

First, for each pixel (x, y) of the luminance resolution difference image RS _I (i; c, s), a gamma distribution estimated input coefficient λ _I (i) _{(x , Y)} is a pixel value around the target pixel (including both spatial and temporal meanings ₎ in the luminance resolution difference image and a gamma distribution coefficient ρ _{1, I} (i−1) _{(x, y)} , ρ _{2, I} (i-1) Derived from _{(x, y)} . Next, using the derived gamma distribution estimation input coefficient λ _I (i) _{(x, y)} , the gamma distribution coefficients ρ _{1, I} (i−1) _{(x, y)} , ρ _{2, I} (i−1) ) _{(X, y)} is updated as in the following (Expression 41) and (Expression 42).

Here, ζ (0 <ζ <1) is a coefficient expressing the reflection rate of the history in coefficient update. From the gamma distribution coefficient derived by the above processing, the (c, s) level luminance time saliency image SP _I (i; c, s), which is an image representing the saliency of the luminance component in the time direction, is as follows. To derive.

  Here, D (P‖P ′) is a Kullback-Liebler divergence between the probability density functions P and P ′. Ψ (x) is called a digamma function and is expressed by the following (formula 45).

  For simplification of description, I representing the luminance component and (x, y) representing the target pixel are omitted.

The luminance time saliency image SP _I (i) derived in this way, the color time saliency image SP _C (i), the direction time saliency image SP _O (i), and the blinking time derived by the same processing. The temporal saliency image SP (i) is generated by integrating the saliency image SP _F (i) and the exercise time saliency image SP _M (i). Specifically, it is generated as in the following (Expression 46) to (Expression 51).

On the other hand, in the second time saliency image generation method, a time saliency image is generated using weighted difference processing on the time axis based on a method similar to the resolution difference image extraction unit 13.
First, the time axis direction weighted by a Gaussian distribution from the saliency images CM _j (it) (0 ≦ t ≦ n _T ) of each type (brightness, color, etc.) up to the present time and before n _T time points. The time response image T _j (i) is calculated as shown in (Equation 52) by the difference processing.

Here, η _σ (t) is a probability density function of a Gaussian distribution having an average of 0 and a variance of σ ² . By integrating the time response images, the time saliency image SP (i) is calculated as shown in (Formula 53).

The time saliency image binarization unit 422 receives the time saliency image, binarizes the time saliency image, generates an attention level instantaneous recovery image, and outputs the attention level instantaneous recovery image.
In the present embodiment, the temporal saliency image SP (i) is binarized as follows, and the attention degree instantaneous recovery image GD ₂ (i) is generated as shown in (Formula 54).

  Here, θ is a threshold for binarization of the time saliency image, and is determined in advance.

As described above, the attention degree instantaneous recovery image generation unit 42 generates the attention degree instantaneous recovery image GD ₂ (i) and outputs it.

The attention degree gradual suppression image generation unit 43 inputs the attention degree gradual occlusion image and the attention degree instantaneous recovery image, and generates an attention degree gradual suppression image by integrating these images. Output.
In the present embodiment, the attention degree progressive suppression image GD (i) is obtained as shown in the following (Expression 55).

The attention level image generation unit 44 after progressive suppression inputs the attention level progressive suppression image and the basic attention level image or the instantaneous suppression degree of attention image, and integrates these images to obtain the attention level image after progressive suppression. And generating an attention degree image after the progressive suppression.
The method of generating the attention degree image after progressive suppression is not particularly limited, but in the present embodiment, the attention degree attention image S _I (i) after instantaneous suppression and the attention degree progressive suppression image GD (i) are integrated. Then, the attention degree image S _L (i) after progressive suppression is generated as in (Expression 56) to (Expression 57).

Here, ω _G (i) ≧ 0 is a coefficient expressing the weight for the attention degree progressive suppression image GD (i). As shown above, the basic attention level image S (i) output from the basic attention level image extraction unit 1 is expressed by an expression using the instantaneous attention level suppression image S _I (i) output from the attention level instantaneous suppression unit 3. It is possible to calculate by the formula using Here, when ω _I (i) = 0, this is equivalent to the case where the attention degree instantaneous suppression unit 3 is not used.

  As described above, the attention degree gradual suppression unit 4 extracts the gradual suppression attention degree image and outputs it.

  The attention level video output unit 5 is the above-described attention level image after progressive suppression extracted by repeatedly executing the basic attention level image extraction unit 1 to the attention level progressive suppression unit 4 sequentially for each input image. The attention degree image after progressive suppression that is a series is extracted and output as the attention degree image.

  FIG. 4 shows an operation example of this embodiment. In the figure, (a) is an input image, (b) is an attention degree progressive suppression image, and (c) is a progressive suppression attention degree image, which are arranged in chronological order from the left.

[Third Embodiment]
FIG. 5 is a functional block diagram of the attention area extracting apparatus according to the third embodiment of the present invention.
The attention area extraction apparatus shown in the present embodiment includes a basic attention level image extraction unit 1, a saliency image integration ratio calculation unit 2, an attention level instantaneous suppression unit 3, an attention level progressive suppression unit 4, and an attention level video output. An input video that is a target of interest level extraction is input, and an attention level video that is a video displaying a region of high attention level in the input image is output. In the figure, the same components as those in the first and second embodiments are denoted by the same reference numerals, and the description thereof is omitted. The basic attention level image extraction unit 1, the attention level instantaneous suppression unit 3, the attention level gradual suppression unit 4, and the attention level video output unit 5 are the same as those in the first or second embodiment.

  The saliency image integration ratio calculation unit 2 inputs the set of saliency images extracted from the current input image and the basic attention level image, and extracts the maximum basic attention level region from the basic attention level image. Then, the value in the region corresponding to the maximum basic attention level region extracted for each saliency image is calculated, and the saliency image integration ratio that is the weight of the corresponding saliency image is determined from the magnitude of the value. The set of the saliency image integration ratios is output.

  The calculation method of the saliency image integration ratio is not particularly limited. In this embodiment, V. Navalpakkam and L. Itti: “Optimal cue selection strategy,” in Advances in Neural Information Processing Systems (NIPS). , pp.987-994, December 2005. (Non-Patent Document 6), the saliency image integration ratio is sequentially updated using the pixel value of each saliency image in the maximum basic attention level region. .

The saliency image integration ratio w _j corresponding to each saliency image CM _j (i−1) (j = I, C, O, F, M) is obtained by the processing in this processing unit using the input image one point before. It is assumed that (i-1) has already been obtained. First, the maximum basic attention level region MSR (i) is extracted from the basic attention level image S (i). The extraction method is the same as the extraction method in the maximum basic attention level region detection unit 31 described above. Next, the saliency image integration ratio w _j (i) corresponding to each saliency image CM _j (i) is determined as follows.

  Here, δ is a constant that gives the reflection ratio of the history in the weight update.

Subsequently, experimental data using the embodiment of the present invention is shown below.
As input images, six types of images having a size of 640 × 480 pixels and a length of 8 to 15 seconds were prepared. Moreover, the following values were used as actual numerical values of the symbols shown in each embodiment.

n _φ = 4, n _F = 3, σ = 1.25, n _l = 8, L _c = {2, 3, 4}, L _s = {c + 3, c + 4} (c∈L _C ), n _A = 32 × 24 = 768, ε = 25, μ = 1.0, t _I = 10, α = 1 / t _I = 0.1, β = 0.0025, t _T = 8, θ = 0.25max _{(x , Y)} SP (i) _{(x, y)} , δ = 0.1

  In order to confirm the effect of the present invention, the degree of attention video obtained by the embodiment of the present invention and a known method was compared with how much human visual characteristics could be simulated. As the quantity that expresses human visual characteristics, the position of the line of sight when humans are actually watching the input video is adopted. The input video was presented to five subjects, and the gaze position in the input video of each subject was sequentially measured using an existing gaze measurement device. For each subject, one type of input video was presented twice. As a result, two time series of gaze positions were obtained for each subject and each input video. By associating the time series of the gaze position with each frame (that is, the input image) of the input video while maintaining time consistency, two gaze positions were obtained for each subject and each input image.

The degree of attention at the gaze position of the subject was adopted as an evaluation scale for whether or not human visual characteristics could be simulated. Attention level video corresponding to the input image IN (i; k) (k = 1, 2,..., 6: corresponding to the type of video) of the attention level video extracted from the input video by the method of the present invention or a known method. Consider the attention degree image S _F (i; k) which is a frame of In the first embodiment of the present invention, the attention-suppressed attention level image is equal to the attention level image, that is, S _F (i; k) = S _I (i; k), and the second embodiment of the present invention. , The attention degree image after progressive suppression is equal to the attention degree image, that is, S _F (i; k) = S _I (i; k). When the gaze position of the subject n (n = 1, 2,..., 5) in the input image IN (n; k) is expressed as (x (i; k, n), y (i; k, n)), The evaluation value V (i; k, n) of the attention degree image S _F (i; k) with the subject n as “teacher” was determined by the following (formula 60).

On the right side of (Equation 60), the denominator is given to normalize the attention level image. Attention level image {S _F (i; k)} The evaluation value V (k) of _i is the sum of the evaluation values for each attention level image, as shown in (Equation 61), and further takes the average for the subjects. Get in. n _E is the number of subjects (that is, “5” in this evaluation example).

Based on the above evaluation values, the methods according to the first to third embodiments of the present invention were compared with known methods. As a known method, the method described in Non-Patent Document 1 and the method described in Non-Patent Document 2 were used. FIG. 6 shows a method described in Non-Patent Document 1, a method described in Non-Patent Document 2, a method according to the first embodiment (Embodiment 1 of the present invention), and a method according to the second embodiment (Implementation of the present invention). Embodiment 2), a graph comparing evaluation values (NETR Value in the figure) for each input video (Video 1 to 6) when using the method according to the third embodiment (Embodiment 3 of the present invention), FIG. The graph which compared the evaluation value which took the average about all the input images | videos for each method is shown. In FIG. 7, from the left, the method described in Non-Patent Document 1 (Still image algorithm), the method described in Non-Patent Document 2 (Moving algorithm), the method according to the first embodiment (Case 1), and the second embodiment. The average evaluation value of the method (Case 2) according to the third embodiment and the method (Case 3) according to the third embodiment is shown. Here, in the methods according to the first to third embodiments of the present invention, three settings focused on the attention degree image generation unit 44 after progressive suppression are evaluated. In the first setting (in the drawing, the method according to the first embodiment of the present invention), the degree-of-advance attention degree image generation unit 44 (ω _I (i), ω _G (i)) = (1, 0) ∀i. In the second setting (in the figure, the method according to the second embodiment of the present invention), the degree-of-advance attention degree image generation unit 44 (ω _I (i), ω _G (i)) = (0, 1) ∀i. In the third setting (in the drawing, the method according to the third embodiment of the present invention), the degree-of-advance attention level image generation unit 44 (ω _I (i), ω _G (i)) = (1, 1) ∀i. As can be seen from FIG. 7, when the input video was averaged, the setting of the second embodiment of the present invention showed the best evaluation value. Further, as can be seen from FIG. 6, the setting of the second embodiment of the present invention showed an evaluation value that exceeded the other methods for any input video. Further, as can be seen from FIG. 6, the settings of the first and third embodiments of the present invention showed an evaluation value exceeding the known method for some input images.

  Note that the attention area extraction apparatus described above has a computer system therein. The basic attention level image extraction unit 1, the saliency image integration ratio calculation unit 2, the attention level instantaneous suppression unit 3, the attention level progressive suppression unit 4, and the attention level video output unit 5 of the attention area extraction device Is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing this program. The computer system here includes a CPU, various memories, an OS, and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device such as a hard disk built in the computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case is also used to hold a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

It is a block diagram which shows the structure of the attention area extraction apparatus by the 1st Embodiment of this invention. It is a figure which shows the operation example of the attention area extraction apparatus shown in FIG. It is a block diagram which shows the structure of the attention area extraction device by the 2nd Embodiment of this invention. It is a figure which shows the operation example of the attention area extraction apparatus shown in FIG. It is a block diagram which shows the structure of the attention area extraction device by the 3rd Embodiment of this invention. It is a graph which shows the evaluation value for every input image | video by the method using this invention and a prior art. It is a graph which shows the average of the evaluation value of each method shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Basic attention image extraction part 11 ... Basic feature image extraction part 111 ... Luminance feature image extraction part 112 ... Color feature image extraction part 113 ... Direction feature image extraction part 114 ... Flashing feature image extraction part 115 ... Motion feature image extraction part DESCRIPTION OF SYMBOLS 12 ... Multi-resolution image extraction part 13 ... Resolution difference image extraction part 14 ... Saliency image extraction part 141 ... Resolution difference image normalization part 142 ... Normalization resolution difference image integration part 15 ... Saliency degree image integration part 2 ... Saliency degree image Integrated ratio calculation unit 3 ... attention level instantaneous suppression unit 31 ... maximum basic attention level region detection unit 32 ... maximum basic attention level region occlusion image extraction unit 33 ... attention level progressive recovery image extraction unit 34 ... attention level instantaneous suppression image generation unit 35 ... attention degree image generation unit after instantaneous suppression 4 ... attention degree gradual suppression part 41 ... attention degree gradual occlusion image generation part 42 ... attention degree instantaneous recovery image generation part 43 ... attention degree gradual suppression image generation Part 44 ... gradual suppression after prominence image generating unit 5 ... attention video output unit

Claims (9)

A region-of-interest extraction method for extracting an attention level image, which is a video that displays a spatio-temporal region having remarkable characteristics in the input image, from a target input image,
A basic attention level image extraction process for extracting a basic attention level image, which is an image displaying a spatial region having a remarkable characteristic in the frame from a certain frame constituting the input video;
In the basic attention level image extracted from the previous frame of the input video by the basic attention level image extraction process, a maximum basic attention level area that is the area where the basic attention level that is the value of each pixel is the largest The attention degree instantaneous suppression process of extracting the attention degree image after instantaneous suppression by suppressing the basic attention degree image extracted from the current frame of the input video by the basic attention degree image extraction process;
Or
An area having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video of the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the attention degree gradual suppression processes for extracting the attention degree image after gradual suppression,
By repeatedly executing the basic attention level image extraction process and one or both of the attention level instantaneous suppression process and the attention level gradual suppression process sequentially for each frame of the input video, Attention area extraction characterized by comprising: an attention level image or a time series of attention level images after progressive suppression, which is a time series of the attention level image after progressive suppression, and an attention level video output process of outputting this as attention level video Method. The basic attention degree image extraction process includes:
A basic feature image extraction process for extracting a plurality of types of basic feature images from a frame of the input video;
For each type of basic feature image extracted by the basic feature image extraction process, a multi-resolution image extraction process for extracting a multi-resolution image that is a multi-resolution expression;
A resolution difference image extraction step of extracting a plurality of resolution difference images, which are differences between images of different resolutions, for each type of the multi-resolution image extracted by the multi-resolution image extraction step;
For each type of resolution difference image extracted by the resolution difference image extraction process, a saliency image extraction process for extracting a saliency image by integrating resolution difference images with different resolutions;
The saliency image extracted by the saliency image extraction process consists of a saliency image integration process for extracting a basic attention image by integrating a plurality of types of saliency images,
The attention area extraction method according to claim 1, wherein the attention degree gradual suppression process extracts a saliency image using a gradual suppression image instead of the basic attention degree image. further,
The maximum basic attention level region is extracted from the basic attention level image extracted by the basic attention level image extraction process, and a value in a region corresponding to the maximum basic attention level region is calculated for a plurality of types of saliency images. A saliency image integration ratio calculation process for determining a saliency image integration ratio that is a weight of the corresponding saliency image from the magnitude of the value, and
In the basic attention level image extraction process, the saliency image is weighted and integrated by the saliency image integration ratio calculated by the saliency image integration ratio calculation process for the previous frame of the input video, thereby integrating the basic attention level image. The attention area extraction method according to claim 1 or 2, wherein an attention degree image is extracted. The saliency image integration ratio calculation process includes:
The saliency image integration ratio calculated for one previous frame of the input video is set as an initial value, and the value in the maximum basic attention area calculated for each saliency image is calculated with respect to the initial value. The attention area extraction method according to claim 3, wherein a new saliency image integration ratio is updated as a difference value. The attention degree instantaneous suppression process is:
A maximum basic attention area detection process for extracting the maximum basic attention area from a basic attention image extracted from a previous frame of the input video by the basic attention image extraction process;
A maximum basic attention area occlusion image extraction process for extracting a maximum basic attention area occlusion image that is an image that occludes the maximum basic attention area extracted by the maximum basic attention area detection process;
Attention gradual extraction for extracting an attention degree progressive recovery image that is an image that reduces occluding in the maximum basic attention degree area occlusion image extracted by the maximum basic attention degree area occlusion image extraction process Recovery image extraction process,
Instantaneous attention level suppression by integrating the maximum basic attention level region occlusion image extracted by the maximum basic attention level region occlusion image extraction process and the attention level progressive recovery image extracted by the attention level progressive recovery image extraction process Attention level instantaneous suppression image generation process to generate an image,
After instantaneous suppression by integrating the instantaneous attention suppressed image generated by the attention instantaneous suppression image generation process and the basic attention image of the current frame of the input video extracted by the basic attention image extraction process The attention area extracting method according to any one of claims 1 to 4, further comprising: an attention degree image generation process after instantaneous suppression that generates an attention degree image. The attention degree gradual suppression process is:
Attention degree progressive occlusion image generation process for generating an attention degree progressive occlusion image that is an image that gradually occludes the basic attention degree image extracted by the basic attention degree image extraction process;
For a basic attention level image or saliency image calculated from several frames before the current frame of the input video, an area having a remarkable value in the time axis direction is extracted, and the basic attention level corresponding to the area is extracted. Attention degree instantaneous recovery image generation process for generating an attention degree instantaneous recovery image that is an image or an image for canceling the suppression of the basic attention degree of the area in the attention degree image after instantaneous suppression;
The attention degree progressive occlusion image generated by the attention degree progressive occlusion image generation process and the attention degree instantaneous recovery image generated by the attention degree instantaneous recovery image generation process are integrated to generate an attention degree progressive suppression image. Attention level progressive suppression image generation process,
The basic attention degree image extracted by the basic attention degree image extraction process, or the instantaneous attention degree image extracted by the attention degree instantaneous suppression process, and the attention degree generated in the attention degree gradual suppression image generation process The attention area extraction according to any one of claims 1 to 5, comprising a step of generating a post-progression attention level image by integrating a progressive suppression image and integrating the progressive suppression image. Method. An attention area extraction device that extracts an attention degree image, which is an image displaying a spatio-temporal region having a remarkable characteristic in the input image, from a target input image,
A basic attention level image extraction unit that extracts a basic attention level image, which is an image displaying a spatial region having remarkable characteristics in the frame, from a certain frame constituting the input video;
In the basic attention level image extracted from the previous frame of the input video by the basic attention level image extraction unit, the maximum basic attention level area that is the area where the basic attention level that is the value of each pixel is the largest An attention level instantaneous suppression unit that extracts an attention level image after instantaneous suppression by suppressing the basic attention level image extracted from the current frame of the input video by the basic attention level image extraction unit,
Or
A region having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the basic attention image extraction unit extracts the region of the input video. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression unit, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the degree-of-interest gradual suppression units for extracting the degree-of-interest attention level image,
For each frame of the input video, the attention level image after instantaneous suppression extracted by the attention level instantaneous suppression unit or the time series of the attention level image after progressive suppression extracted by the attention level progressive suppression unit An attention area extracting apparatus comprising: an attention degree video output unit that extracts an attention degree video after suppression and outputs the attention degree video as the attention degree video. From a target input video, to a computer used as a region of interest extraction device that extracts a video of attention level, which is a video displaying a spatio-temporal region with remarkable characteristics in the input video,
A basic attention level image extraction process for extracting a basic attention level image, which is an image displaying a spatial region having a remarkable characteristic in the frame from a certain frame constituting the input video;
In the basic attention level image extracted from the previous frame of the input video by the basic attention level image extraction process, a maximum basic attention level area that is the area where the basic attention level that is the value of each pixel is the largest The attention degree instantaneous suppression process of extracting the attention degree image after instantaneous suppression by suppressing the basic attention degree image extracted from the current frame of the input video by the basic attention degree image extraction process;
Or
An area having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video of the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the attention degree gradual suppression processes for extracting the attention degree image after gradual suppression,
By repeatedly executing the basic attention level image extraction process and one or both of the attention level instantaneous suppression process and the attention level gradual suppression process sequentially for each frame of the input video, A computer program that extracts an attention level image or a time series of attention level images after progressive suppression that is a time series of the attention level image after progressive suppression and outputs the attention level image as an attention level image. . From a target input video, to a computer used as a region of interest extraction device that extracts a video of attention level, which is a video displaying a spatio-temporal region with remarkable characteristics in the input video,
A basic attention level image extraction process for extracting a basic attention level image, which is an image displaying a spatial region having a remarkable characteristic in the frame from a certain frame constituting the input video;
In the basic attention level image extracted from the previous frame of the input video by the basic attention level image extraction process, a maximum basic attention level area that is the area where the basic attention level that is the value of each pixel is the largest The attention degree instantaneous suppression process of extracting the attention degree image after instantaneous suppression by suppressing the basic attention degree image extracted from the current frame of the input video by the basic attention degree image extraction process;
Or
An area having a remarkable value in the time axis direction is extracted from the basic attention image calculated from several frames before the current frame of the input video, and the input video of the input video is extracted by the basic attention image extraction process. In the basic attention level image extracted from the frame or the instantaneous transition attention level image extracted from the frame of the input video by the attention level instantaneous suppression process, the extracted region is emphasized and other regions are suppressed. Thus, one or both of the attention degree gradual suppression processes for extracting the attention degree image after gradual suppression,
By repeatedly executing the basic attention level image extraction process and one or both of the attention level instantaneous suppression process and the attention level gradual suppression process sequentially for each frame of the input video, A computer-readable recording of a computer program for extracting an attention level image or a time series of attention level images after progressive suppression, which is a time series of the attention level image after progressive suppression, and outputting the attention level image as an attention level image Possible recording media.

Images