JP6318451B2 - Saliency image generating apparatus, method, and program - Google Patents

Description

  The present invention relates to a saliency image generating apparatus, method, and program, and in particular, a saliency image generating apparatus, method, and program for generating a saliency image indicating saliency at each position of an input image of a frame at each time. About.

  Humans efficiently acquire information by instantly judging information that is considered important from the image captured in the retina by a mechanism called visual attention. By simulating these human perceptual characteristics on a computer, it is expected to construct an artificial visual system that actively selects information according to the degree of importance, similar to humans.

  As a method of simulating visual attention on a computer, a method based on visual saliency is common. In this method based on visual saliency, in each part of a given image signal, the visual saliency, which is the degree to which human attention is directed, is calculated, and a portion where the visual saliency is greater than a predetermined value is observed. It is a method to predict as.

  As gaze prediction methods based on visual saliency, methods described in Non-Patent Documents 1 and 2 have been proposed. Both of these methods employ a stochastic saliency model called Bayesian surprise. In this Bayesian surprise model, visual stimuli that are likely to occur in the future are predicted sequentially at each position in the image space with respect to the time series of input image signals, resulting from newly input image signals. This model assigns a high visual saliency to a location where the visual stimulus and the prediction are greatly deviated by a certain value or more.

L. Itti, P.F. Baldi “A principled approach to detecting surprising events in videos,” Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2005), pp.631-637, 2005. L. Itti, P.F. Baldi “Bayesian surprise attracts human attention,” Vision Research, Vol.49, No.10, pp.1295-1306, 2009.

  Many of the prior arts including Non-Patent Documents 1 and 2 have a problem that only a part of signals constituting an image, that is, image signals can be used. In particular, the acoustic signal, which is another main component that composes the video, is easy to direct the line of sight in the direction of the sound that draws attention, and it is easy to direct the line of sight to the object that moves in synchronization with the change in sound. As can be seen, since it has a great influence on human gaze behavior, it must be appropriately incorporated into the calculation of visual saliency. However, there has been little discussion on a visual saliency model that uses both image signals and sound signals and focuses on the interaction between the two.

  The present invention has been made in view of the above-described problems. The input image of each time frame constituting the input video and the acoustic signal constituting the input video are used at each position of the input image of the frame of each time. It is an object of the present invention to provide a saliency image generating apparatus, method, and program capable of generating a saliency image indicating saliency.

  In order to achieve the above object, the saliency image generating apparatus according to the present invention provides a remarkable characteristic in the input image for each of a plurality of feature types with respect to the input image of each time frame constituting the input video. A basic saliency image extraction unit that generates a basic saliency image that indicates the degree of possession and sets the basic saliency image as a set, and an acoustic signal that constitutes the input video indicates the degree of remarkable characteristics at each time An acoustic saliency signal calculation unit that generates an acoustic saliency signal, and for each of the plurality of feature types, for each time and each pixel, the features included in the set of basic saliency images for the frame of the time The correlation between the pixel of the basic saliency image for the type and the acoustic saliency signal at the time is calculated, and each time and each image for each of the plurality of feature types is calculated. An image basic saliency selection unit that generates a main image basic saliency component based on the correlation, a set of the basic saliency images for each time frame, and the main image basic saliency component An image saliency image calculating unit that generates a saliency image indicating the saliency at each position of the input image of the frame at each time.

  In the saliency image generation method according to the present invention, the image basic saliency image extraction unit has a remarkable characteristic in the input image for each of a plurality of feature types with respect to the input image of each time frame constituting the input video. Generating a basic saliency image indicating the degree of having a saliency, a set of basic saliency images, and the acoustic saliency signal calculation unit indicating the degree of remarkable characteristics at each time with respect to the acoustic signals constituting the input video An acoustic saliency signal is generated, and an image basic saliency selection unit is included in the set of basic saliency images for the frame of the time for each time and each pixel for each of the plurality of feature types. The correlation between the pixel of the basic saliency image for the feature type and the acoustic saliency signal at the time is calculated, and each time and each pixel for each of the plurality of feature types Based on the correlation, a main image basic saliency component is generated, and an image saliency image calculating unit converts the set of basic saliency images and the main image basic saliency component for each time frame. Based on this, a saliency image indicating the saliency at each position of the input image of the frame at each time is generated.

  According to the present invention, the image basic saliency image extraction unit indicates the degree of remarkable characteristics in the input image for each of a plurality of feature types with respect to the input image of each time frame constituting the input video. A basic saliency image is generated and set as a set of basic saliency images.

  The acoustic saliency signal calculation unit generates an acoustic saliency signal indicating the degree of remarkable characteristics at each time for the acoustic signals constituting the input video.

  An image basic saliency selection unit, for each of the plurality of feature types, for each time and each pixel, a basic saliency image for the feature type included in the set of basic saliency images for the frame of the time A correlation between the pixel and the acoustic saliency signal at the time is calculated, and a main image basic saliency component is generated based on the correlation at each time and each pixel for each of the plurality of feature types.

  The image saliency image calculation unit indicates the saliency at each position of the input image of the frame at each time based on the set of basic saliency images for the frame at each time and the main image basic saliency component. Generate a saliency image.

  As described above, the saliency image indicating the saliency at each position of the input image of the frame at each time is generated using the input image of the frame at each time constituting the input video and the acoustic signal constituting the input video. Can do.

  The program according to the present invention is a program for causing a computer to function as each unit of the saliency image generating apparatus.

  As described above, according to the saliency image generating apparatus, method, and program of the present invention, the input image of each time frame constituting the input video and the acoustic signal constituting the input video are used. The effect that the saliency image indicating the saliency at each position of the input image of the frame can be generated is obtained.

It is the schematic which shows the structure of the saliency image generation apparatus which concerns on the 1st Embodiment of this invention. FIG. 2 is a diagram mainly illustrating a configuration of an image basic saliency image calculating unit 1. FIG. 3 is a diagram mainly illustrating a configuration of an acoustic saliency signal calculation unit 2. It is a figure which shows the data flow from the image basic saliency image extraction part 15 and the acoustic saliency signal extraction part 22. FIG. It is a flowchart which shows the saliency image generation processing program which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the image basic saliency image calculation processing program of step 1S of FIG. It is a flowchart which shows the image basic feature-value image calculation processing program of step 11S of FIG. It is a flowchart which shows the acoustic saliency signal calculation processing program of step 2S of FIG. It is the schematic which shows the structure of the gaze position estimation apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the gaze position estimation processing program which concerns on the 2nd Embodiment of this invention. It is a figure which shows the outline | summary of the evaluation result with respect to the image | video 1E. It is a figure which shows the evaluation result for every flame | frame with respect to the image | video 1E. It is a figure which shows the outline | summary of the evaluation result with respect to the image | video 2E. It is a figure which shows the evaluation result for every flame | frame with respect to the image | video 2E. It is a figure which shows the outline | summary of the evaluation result with respect to the image | video 3E. It is a figure which shows the evaluation result for every flame | frame with respect to the image | video 3E.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Overview]
The present invention has been made in view of the above situation, and realizes gaze position estimation that solves the above problem by the following two points.
1. A video section having a high saliency calculated from the audio signal is detected, and main image signal components in the video section are selected. Thereby, it is possible to selectively extract an image signal component having a strong correlation with a remarkable acoustic signal.
2. When calculating the saliency from the image signal, the image signal component selected in 1. is emphasized. This makes it possible to calculate visual saliency due to the acoustic signal.

[First embodiment]

  Hereinafter, a saliency image generating apparatus according to a first embodiment of the present invention will be described with reference to the drawings. The saliency image generating apparatus is configured by a computer including a CPU, a RAM, and a ROM storing a program, and is functionally configured as follows. FIG. 1 shows an outline of the configuration of the saliency image generating apparatus according to the first embodiment. As shown in FIG. 1, the saliency image generating apparatus according to the first embodiment includes an input unit 10, an image basic saliency image calculating unit 1, an acoustic saliency signal calculating unit 2, and an image basic saliency selection. The unit 3, the image saliency image calculating unit 4, and the saliency video calculating unit 5 are configured. The saliency image generating apparatus inputs an input video and outputs a saliency video that is a video displaying the saliency at each position in a frame of the input video.

  FIG. 2 mainly shows the configuration of the image basic saliency image calculating unit 1. As shown in FIG. 2, the image basic saliency image calculation unit 1 is an image that displays the degree of remarkable characteristics for each pixel in the input image that is a frame of the input video and is input by the input unit 10. Some basic saliency images are calculated, and a set of these basic saliency images is output.

  The calculation method of the basic saliency image is not particularly limited, but in this embodiment, the methods shown in Non-Patent Documents 1 and 2 are adopted. As shown in FIG. 2, the image basic saliency image calculating unit 1 according to this method includes an image basic feature image extracting unit 11, an image multi-resolution image extracting unit 12, an image resolution difference image extracting unit 13, The image time difference image extraction unit 14 and the image basic saliency image extraction unit 15 are configured.

  The image basic feature image extraction unit 11 extracts an image basic feature image that represents a characteristic component of each pixel of the input image using a plurality of feature extraction methods from the input image, and the basic feature image for each feature extraction method. Is output to the image multi-resolution image extraction unit 12. The image basic feature image extraction method is not particularly limited, but in the present embodiment, as shown in FIG. 2, a luminance feature image extraction unit 111, a color feature image extraction unit 112, and a direction feature image extraction are performed. Unit 113, blinking feature image extraction unit 114, and motion feature image extraction unit 115. As will be described in detail later, the feature extraction method of the image basic feature quantity image extraction unit 11 is related to the method described in Patent Document 3 (Japanese Patent Laid-Open No. 2009-003615).

  FIG. 3 mainly shows the configuration of the acoustic saliency signal calculation unit 2. The acoustic saliency signal calculating unit 2 is a signal that displays a degree of remarkable characteristics at each time among input acoustic signals that are input from the input unit 10 and are acoustic signals constituting the input video. A signal is calculated, and this acoustic saliency signal is output to the image basic saliency selection unit 3 and the image saliency image calculation unit 4. The calculation method of the acoustic saliency signal is not particularly limited, but in the present embodiment, the method described in Non-Patent Document 5 in which the Bayesian surprise model is applied to the acoustic signal is adopted. As shown in FIG. 3, the acoustic saliency signal calculation unit 2 according to this method includes an acoustic basic feature amount extraction unit 21 and an acoustic saliency signal extraction unit 22.

(Non-Patent Document 5) Scheuerte and Stiefelhagen "Wow! Bayesian surprise for salient acoustic event detection," Proc. IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP2013), pp.6402-6406, 2013.

  FIG. 4 shows a data flow from the image basic saliency image extraction unit 15 and the acoustic saliency signal extraction unit 22. As shown in FIG. 4, data from the image basic saliency image extraction unit 15 and the acoustic saliency signal extraction unit 22 are input to the image basic saliency selection unit 3 and the image saliency image calculation unit 4, respectively.

  Next, the operation of the saliency image generating apparatus according to the first embodiment of the present invention will be described.

  FIG. 5 shows a flowchart showing the saliency image generation processing program according to the first embodiment of the present invention. When the saliency image generation processing program is started, the image basic saliency image calculation unit 1 executes image basic saliency image calculation processing in step 1S. FIG. 6 is a flowchart showing the image basic saliency image calculation processing program in step 1S of FIG. As shown in FIG. 6, in step 11 </ b> S, the image basic feature value image extraction unit 11 executes image basic feature value image extraction processing. FIG. 7 is a flowchart showing the image basic feature amount image calculation processing program in step 11S of FIG.

  As shown in FIG. 7, in step 111S, the luminance feature image extraction unit 111 executes a luminance feature image extraction process. Details are as follows. That is, the luminance feature image extraction unit 111 outputs a luminance feature image representing the luminance component of the t-th input image (= t-th frame of the input video) input from the input unit 10. The luminance feature image extraction unit 111 obtains the luminance feature image i (t) as an average of red (R), green (G), and blue (B) components of the input image as follows.

However, r (t), g (t), and b (t) are red (R), green (G), and blue (B) in the t-th input image (= t-th frame of the input video), respectively. It is a component image, and each pixel value is expressed by a real value of 0 or more and 1 or less. As another embodiment, each pixel value i (t) _x of the luminance feature image can be extracted by any one of the following expressions.

Here, r (t) _x is the pixel value of the image r (t) at the pixel position x.

  In step 112S of FIG. 7, the color feature image extraction unit 112 executes color feature image extraction processing. Details are as follows.

The color feature image extraction unit 112 outputs a color feature image representing the color component of each pixel of the t-th input image input from the input unit 10. That is, in the color feature image extraction unit 112, color feature images R (t), G (t), and B (t corresponding to red (R), green (G), blue (B), and yellow (Y), respectively. ), Y (t) are extracted from the following pixel values R (t) _x , G (t) _x, B (t) _x, Y (t) _x . For example, R (t) _x is a pixel value of the image R (t) at the position x.

  In step 113S of FIG. 7, the direction feature image extraction unit 113 executes the direction feature image extraction process. Details are as follows.

The direction feature image extraction unit 113 outputs a direction feature image representing the direction component of each pixel of the t-th input image input from the input unit 10. The direction feature image O _φ (t) is obtained as follows by applying a Gabor filter g _φ having a rotation angle _φ to the luminance feature image i (t) calculated from the current input image.

However, * is an operator expressing convolution. The direction feature image O _φ (t) is extracted for n _φ rotation angles. At this time, the rotation angle φ is selected so that, for example, π = 180 ° is divided into n _φ equally.

In step 114S of FIG. 7, the blinking feature image extraction unit 114 executes a blinking feature image extraction process. Details are as follows. The blinking feature image extraction unit 114 outputs a blinking feature image representing the blinking component of each pixel of the input image input from the input unit 10. The blinking feature image F (t) is obtained as follows from the luminance feature images i (t),..., I (t−n _F ) calculated from several current and previous input images. .

Here, n _F is the number of past luminance feature images to be referred to when extracting the blinking feature image. When n _F = 1, the method described in Non-Patent Document 4 is consistent.

(Non-Patent Document 4) Itti, Dhavale and Pighin "Realistic avatar eye and head animation using a neurobiological model of visual attention," Proc. SPIE International Symposium on Optical Science and Technology, pp.64-78, 2003.

  In step 115S of FIG. 7, the motion feature image extraction unit 115 executes motion feature image extraction processing. Details are as follows.

The motion feature image extraction unit 115 outputs a motion feature image representing a motion component of each pixel of the input image input from the input unit 10. The method for extracting the motion feature image is not particularly limited, but in the present embodiment, the luminance feature image i calculated from the input image at the present time and one time point before (the time corresponding to the frame one frame before). Extraction is performed by obtaining an optical flow at each point of (t) and i (t-1). The optical flow extraction method is not particularly limited. For example, a method based on an image gradient generally called the Lucas-Kanade method can be used, and the motion corresponding to the horizontal component and the vertical component of the motion can be used. feature image M _{x (t),} extracts a M _y (t) (see Patent Document 3 detailed extraction method).

Another example is the method described in Non-Patent Document 4. That is, when an image obtained by shifting the direction feature image O _φ (t) calculated from the current input image by one pixel in the direction perpendicular to the rotation angle _φ is S _φ (t), the motion feature image M _φ (t t) is calculated as follows using the direction feature images O _φ (t) and O _φ (t−1) calculated from the input images at the present time and one time before.

However, the operator x represents a product for each pixel. In this embodiment, a motion feature image M _φ (t) is extracted for each of n _φ rotation angles.

  As shown in FIG. 2, the image basic feature image extraction unit 11 sets a luminance feature image, a color feature image, a direction feature image, a blinking feature image, and a motion feature image as image basic feature images, and these image basic feature images. Are output to the image multi-resolution image extraction unit 12.

  When the set of image basic feature images is output to the image multi-resolution image extraction unit 12, the motion feature image extraction process in step 115S of FIG. 7 ends. When the motion feature image extraction process in step 115S ends, the process proceeds to step 12S in FIG.

  The direction feature image extraction unit 113, the blinking feature image extraction unit 114, and the motion feature image extraction unit 115 are configured to receive the luminance feature image from the luminance feature image extraction unit 111. Without the luminance feature image being input from 111, each of the direction feature image extraction unit 113, the blinking feature image extraction unit 114, and the motion feature image extraction unit 115 performs the same processing as the processing of the luminance feature image extraction unit 111. Thus, a luminance feature image may be obtained.

  In step 12S of FIG. 6, the image multi-resolution image extraction unit 12 executes image multi-resolution image extraction processing. Details are as follows.

  The image multi-resolution image extraction unit 12 extracts a multi-resolution image, which is a multi-resolution expression, for each image basic feature image of the set of image basic feature images input as described above. Output.

  In the present embodiment, since the same processing is performed for any basic feature image, the processing will be described below taking the luminance feature image as an example, and description of other feature images will be omitted.

  A luminance multi-resolution image, which is a multi-resolution representation of a luminance feature image, is extracted by repeatedly performing an operation for reducing the luminance feature image while applying a Gaussian filter to each luminance level.

Where G _σ is a Gaussian filter having variance σ, down () is a function for downsampling, i (t, l) is the l-th level luminance multi-resolution image extracted from the luminance feature image i (t), n _l Is the number of levels of the multi-resolution image. The brightness multi-resolution image at the 0th level is the brightness feature image itself, that is, i (t, 0) = i (t).

For other basic feature images, a multi-resolution image can be extracted by the same method. At this time, n _l luminance multi-resolution images are extracted, whereas color multi-resolution images R (t, l), G (t, l), B (t, l), Y (t, l) Is a total of 4n _l images, direction multi-resolution images O _φ (t, l) are total n _φ n _l images, blinking multi-resolution images F (t, l) are n _l images, motion multi-resolution images M _x (t, l _{), M y (t, l} ) is a total of 2n _l Like or n _phi n _l sheets are extracted, respectively.

  As described above, the image multi-resolution image extraction unit 12 sets the luminance multi-resolution image, the color multi-resolution image, the direction multi-resolution image, the blinking multi-resolution image, and the motion multi-resolution image as multi-resolution images, respectively. The set is output to the image resolution difference image extraction unit 13 (see FIG. 2).

  In step 13S of FIG. 6, the image resolution difference image extraction unit 13 executes image resolution difference image extraction processing. Details are as follows.

  The image resolution difference image extraction unit 13 extracts a resolution difference image that is a difference image between images having different resolution levels for each type (luminance, color, etc.) of the multi-resolution image input as described above. A set of resolution difference images is output.

  The method of extracting the resolution difference image is not particularly limited, but in the present embodiment, each type of resolution difference image is extracted as follows.

Here, up () is a function that performs upsampling, and L _c and L _s are sets of resolution levels to be considered when extracting a resolution difference image, and are called a central resolution level set and a peripheral resolution level set, respectively. Further, RS _I (t; l _c , l _s ) is a luminance resolution difference image obtained from the difference between the luminance multi-resolution images of the l _c level and the l _s level, and thereafter the (l _c , l _s ) level. This is called a luminance resolution difference image. _{Similarly, RS RG (t; l c} , l s) and _{RS BY (t; l c,} l s) a (l _c, l _s) level color resolution difference _{image, RS Oφ (t; l c} , l the _{s) (l c, l s} ) level direction resolution difference _{image, RS F (t; l c} , l s) a (l _c, l _s) level flashing resolution difference _{image, RS Mk (t; l c} , l _s) the (l _c, and l _s) level motion resolution difference image, termed respectively.

  As described above, the image resolution difference image extraction unit 13 sets the luminance resolution difference image, the color resolution difference image, the direction resolution difference image, the blinking resolution difference image, and the motion resolution difference image as the resolution difference images, respectively. The set is output to the image time difference image extraction unit 14 (see FIG. 2).

  In step 14S of FIG. 6, the image time difference image extraction unit 14 executes image time difference image extraction processing. Details are as follows.

  The image time difference image extraction unit 14 extracts, for each resolution difference image in the set of input resolution difference images, a time difference image that records a temporal transition of the resolution difference image, and outputs the set of time difference images. To do.

  The extraction method of the time difference image is not particularly limited, but in the present embodiment, the methods of Non-Patent Documents 1 and 2 assuming that each pixel value of the resolution difference image follows a Poisson distribution are used.

In the present embodiment, since the same processing is performed for any resolution difference image, the processing will be described below by taking the luminance resolution difference image as an example, and description of processing for other resolution difference images will be omitted. First, it is assumed that the pixel value λ _I (t, x) at the pixel position x of the luminance resolution difference image RS _I (t; l _c , l _s ) follows the following gamma distribution.

Where Γ () is a gamma function, and α and β are parameters of a gamma distribution. Also, the indexes l _c and l _s indicating the resolution level are omitted for simplicity. In the present embodiment, the parameters α and β of the gamma distribution are held at each pixel position x of the image, and are set as the pixels α _I (t, x) and β _I (t, x) of the luminance time difference image. At this time, the pixel values α _I (t, x) and β _I (t, x) at the pixel position x of the luminance time difference image are distributed one time before.

Is the prior distribution, and the posterior distribution when the pixel value λ _I (t, x) at the pixel position x of the current luminance resolution difference image is observed

And can be obtained as follows from the Bayes rule.

  In addition, another embodiment considering a time scale is possible. In this embodiment, the pixel value at the pixel position x of the luminance time difference image is obtained as follows.

Where ξ is a forgetting factor and n _d is the number of levels of the time difference image. Other (color / direction / flashing / motion) time difference images can be extracted in the same manner.

  As described above, the image time difference image extraction unit 14 uses the luminance time difference image, the color time difference image, the direction time difference image, the blinking time difference image, and the exercise time difference image as time difference images, respectively. The set is output to the image basic saliency image extraction unit 15 (see FIG. 2).

  In step 15S of FIG. 6, the image basic saliency image extraction unit 15 executes image basic saliency image extraction processing. Details are as follows.

  The image basic saliency image extraction unit 15 extracts a basic saliency image for each time difference image of the set of time difference images input as described above based on the temporal and spatial specificity of the time difference image. Then, a set of these basic saliency images is output.

  The method for extracting the basic saliency image is not particularly limited, but in the present embodiment, the Bayesian surprise model described in Non-Patent Documents 1 and 2 is used. In this Bayesian surprise model, the basic saliency is calculated by the prior distribution (post-distribution one point before) and the Kullback-Leibler divergence of the posterior distribution. Specifically, it is calculated as follows.

  In the present embodiment, since the same processing is performed for any time difference image, the luminance time difference image will be described as an example hereinafter. In the present embodiment, the luminance-basis saliency image in the time direction for calculating the divergence of the prior distribution and the posterior distribution by paying attention to the same pixel position, and the luminance in the spatial direction for calculating the divergence by paying attention also to surrounding pixel positions. The basic saliency images are calculated separately and integrated later. First, the pixel value at the pixel position x of the luminance basic saliency image in the time direction is calculated as follows.

  However, Ψ (·) is a digamma function. Next, the pixel value of the pixel position x of the luminance basic saliency image in the spatial direction is calculated as follows.

  However, DoG () is a function of Difference-of-Gaussian processing. Finally, the final luminance basic saliency image is configured by combining the luminance basic saliency image in the time direction and the luminance basic saliency image in the spatial direction as follows. The combination method is not particularly limited, but in the present embodiment, the combination described in Non-Patent Document 1 is adopted as it is, and calculation is performed using the following formula.

Other (color / direction / flashing / motion) basic saliency images can be extracted in the same manner.
As described above, the image basic saliency image extraction unit 15 converts the luminance basic saliency image, the color basic saliency image, the direction basic saliency image, the blinking basic saliency image, and the motion basic saliency image into the basic saliency image. Are output to the image basic saliency selection unit 3 and the image saliency image calculation unit 4 (see FIGS. 2 and 4). Thereby, the image basic saliency image extraction process in step 15S of FIG. 6 is completed.

  When the image basic saliency image extraction process in step 15S ends, the process proceeds to step 2S in FIG. In step 2S, the acoustic saliency signal calculation unit 2 executes an acoustic saliency signal calculation process. As described above, the acoustic saliency signal calculation unit 2 calculates the acoustic saliency signal, which is a signal indicating the degree to which the input acoustic signal, which is the acoustic signal constituting the input video, has remarkable characteristics at each time, This acoustic saliency signal is output. In the present embodiment, as described above, the method described in Non-Patent Document 5 in which the Bayesian surprise model is applied to the acoustic signal is adopted as the acoustic saliency signal calculation method.

  FIG. 8 is a flowchart showing the acoustic saliency signal calculation processing program in step 2S of FIG. 5 according to the method described in Non-Patent Document 5.

  In step 21 </ b> S of FIG. 8, the acoustic basic feature amount extraction unit 21 performs an acoustic basic feature amount extraction process. Details are as follows.

  The acoustic basic feature quantity extraction unit 21 extracts an acoustic basic feature quantity that is a characteristic quantity expressing the characteristics of the input acoustic signal, and outputs the acoustic basic feature quantity.

  The method of extracting the acoustic basic feature amount is not particularly limited, but in the present embodiment, a method of calculating the time frequency characteristic from the acoustic signal is adopted. That is, the spectrogram F (t, ω) is extracted for each frequency ω from the acoustic signal a (t) cut out with the width of the front and rear window width tw centered on the time t by using time-frequency conversion. At this time, short-time Fourier transform (STFT), discrete cosine transform (DCT), short-time cosine transform (STCT), or the like can be used as the time-frequency transform.

  In step 22S of FIG. 8, the acoustic saliency signal extraction unit 22 performs an acoustic saliency signal extraction process. Details are as follows.

  The acoustic saliency signal extraction unit 22 receives the acoustic basic feature amount, extracts an acoustic saliency signal indicating the degree of remarkable characteristics at each time in the acoustic signal, and outputs the acoustic saliency signal. .

  The method for extracting the acoustic saliency signal is not particularly limited, but in this embodiment, a Bayesian surprise model is used that assumes that the spectrogram at each time frequency is generated according to a Gaussian distribution or a gamma distribution.

  Assuming that the spectrogram is generated according to a Gaussian distribution, the prior distribution of the spectrogram F (t, ω) at time t and frequency ω is the spectrogram history F (t−1, ω),. It is expressed as follows using F (t−N, ω).

  Similarly, the posterior distribution of the spectrogram is expressed as follows.

At this time, the acoustic saliency signal S _A (t, ω) at time t and frequency ω is calculated as Kullback-Leibler divergence of the prior distribution and the posterior distribution as follows.

  On the other hand, assuming that the spectrogram is generated according to the gamma distribution, the prior distribution and posterior distribution of the spectrogram F (t, ω) at time t and frequency ω are calculated as follows.

At this time, the acoustic saliency signal S _A (t, ω) at time t and frequency ω is calculated as Kullback-Leibler divergence of the prior distribution and the posterior distribution as follows.

Finally, the acoustic saliency signal S _A (t) at time t is calculated as the average of the acoustic saliency signals S _A (t, ω) at all frequencies ω.

  As described above, the acoustic saliency signal extraction unit 22 calculates the acoustic saliency signal, and outputs the acoustic saliency signal to the image basic saliency selection unit 3 and the image saliency image calculation unit 4 (see FIG. 4). . Thereby, the acoustic saliency signal extraction process in step 22S of FIG. 5 ends.

  When the acoustic saliency signal extraction process in step 22S ends, step 2S in FIG. 5 ends. Note that the order of the image basic saliency image calculation process of step 1S and the acoustic saliency signal calculation process of step 2S is not limited to this, and the process of step 1S may be executed after the process of step 2S, or executed simultaneously. May be.

  In the above example (FIG. 5), when step 2S ends, the process proceeds to step 3S in FIG. In step 3S, the image basic saliency selection unit 3 executes an image basic saliency selection process. Details are as follows.

  The image basic saliency selection unit 3 selects a main image basic saliency component in a time interval with a large acoustic saliency based on the set of image basic saliency images and the acoustic saliency signal input as described above. Emphasize and output this as a main image basic saliency component.

  The method for selecting the image basic saliency component is not particularly limited, but in the present embodiment, a method based on a simple correlation between the acoustic saliency signal and the pixel value of the image basic saliency image is adopted.

  Hereinafter, in order to simplify the notation, an index is assigned to each basic saliency image at time t, and the index j is used.

Is written. That is, the index j is used to collectively represent differences in basic saliency image types (such as luminance and color) and time scales.

First, at each time t, for each pixel position x, the acoustic saliency signal S _A (t) and each image basic saliency image

Is calculated as follows.

Here, h (n, t) is a time window at time t having a width N _w (t). As the time window, any time window such as a rectangular window, a Hanning window, and a Hamming window can be used. As the width of the time window, a method of using a common value at all times t, a method of changing by the acoustic saliency signal, and the like can be considered. As a method for controlling the width of the time window by the acoustic saliency signal, the following method can be considered. If the continuous time interval in which the acoustic saliency signal exceeds the threshold θ _s is T _{S, i} (i = 1, 2,), the window width N _w (t) at time t is determined as follows.

However, w _a1 > 0 and w _b1 > 0 are predetermined integers, and w _b2 is set to 1 or 2 so that N _w (t) becomes an odd number. According to the above definition, the time window is set only at time t when the acoustic saliency signal S _A (t) exceeds the threshold θ _s , and its width is proportional to the length of the continuous time interval in which the acoustic saliency signal exceeds the threshold. Become longer. Correlation value at each time t

The average value of the correlation values is calculated for the pixel position x in which the value of the pixel takes a value between the upper p _u % and the upper p _l %, and the value is correlated with the correlation at time t

And

Subsequently, at each time T _S = {t _{s, 1} , t _{s, 2} ...} Where the acoustic saliency signal S _A (t) exceeds a predetermined threshold θ _S , the correlation

There removed index j of the image basis saliency image above a threshold theta _c predetermined, enumerate each index in all the time. The counting result can be expressed as a J-dimensional integer vector H = (h ₁ , h ₂ ,..., H _J ) ^T. Number i.e., element h _j of the vector, at the time when the acoustic saliency signal exceeds the threshold value theta _s, image basis saliency image with index j, which exceeds the threshold value theta _c correlation with the acoustic saliency signal Indicates.

Finally, an index in which the vector element h _j is larger than a predetermined threshold value θ _h is left, and this set of indices J _s = {j _{s, 1} , j _{s, 2} ,. The saliency component is output to the image saliency image calculation unit 4.

  In step 4S of FIG. 5, the image saliency image calculation unit 4 executes image saliency image calculation processing. Details are as follows.

  The image saliency image calculation unit 4 performs saliency at each position of the input image based on the set of image basic saliency images input as described above, the main image basic saliency component, and, if necessary, the acoustic saliency signal. A saliency image that is an image indicating the degree is output.

  The calculation method of the saliency image is not particularly limited, but in the present embodiment, a method of constructing the saliency image by selectively using the image basic saliency image selected as the main image basic saliency component Is adopted. That is, the saliency image S (t) at time t is calculated as follows.

Where θ _s2 is a predetermined threshold,

Is an indicator function, which returns 1 when the condition in parentheses is satisfied, and returns 0 otherwise. That is, the above equation constructs a saliency image using only the image basic saliency image selected as the main image basic saliency component at the time when the acoustic saliency signal S _A (t) exceeds the threshold θ _s2 , It is shown that a saliency image is constructed using all image basic saliency images at times other than. Setting the threshold θ _s2 to 0 is equivalent to constructing the saliency image using only the image basic saliency image selected as the main image basic saliency component at all times.

Moreover, you may perform the following methods as another embodiment.
First, as a preparation, the basic saliency image

Using an index f representing a feature type, an index σ representing a spatial scale, and an index d representing a time scale,

And rewrite. That is,

The basic saliency image

The index j is divided into three parts: an image basic feature type f, a spatial scale σ, and a time scale d. Also, the index set of the spatial scale σ is Σ, the index set D of the time scale d, the index set of the spatial scale σ included in the index set J _s of the main image basic saliency image is Σ _s , and the index set of the time scale d _{Let DS} be.
Using the above symbols, the saliency image S (t) at time t is calculated as follows.

  In step 5S of FIG. 5, the saliency video calculation unit 5 executes the saliency video calculation process. That is, the saliency video calculating unit 5 calculates a saliency video that is a time-series image obtained by connecting the saliency images calculated at each time, and outputs the saliency video.

  Note that the saliency video calculation unit 5 may include the input acoustic signal in the saliency video so as to correspond to the time corresponding to each time series.

  As described above, according to the saliency image generating device according to the first embodiment, the input image of each time frame constituting the input video and the acoustic signal constituting the input video are used to set the time of each time. A saliency image indicating the saliency at each position of the input image of the frame can be generated.

  Further, since the image basic saliency image extraction unit 15 and the acoustic saliency signal calculation unit 2 use the same Bayesian surprise model (probability model), the image basic saliency selection unit 3 has a correlation between different physical quantities. It can be physically meaningful.

In the above embodiment, the image basic saliency selection unit 3 may generate the main image basic feature amount component by using a method described in a third embodiment to be described later.
Further, the image saliency image calculation unit 4 may calculate the saliency image using a method described in a third embodiment to be described later.

[Second Embodiment]
Next, a gaze position estimation apparatus according to the second embodiment will be described. In addition, the same code | symbol is attached | subjected to the part which becomes the same structure as 1st Embodiment, and description is abbreviate | omitted.

  FIG. 9 shows an outline of the configuration of the gaze position estimation apparatus according to the second embodiment. As shown in FIG. 9, the gaze position estimation device of the present embodiment includes an input unit 10, an image basic saliency image calculation unit 1, an acoustic saliency signal calculation unit 2 in the saliency image generation device of the first embodiment, The image basic saliency selection unit 3, the image saliency image calculation unit 4, the saliency video calculation unit 5, and the gaze position estimation unit 6 are configured. The gaze position estimation apparatus according to the present embodiment outputs an estimated gaze position that is a result of estimating the gaze position of a human at each position in the frame of the input video that is the target of gaze position estimation, input by the input unit 10. .

  Next, the operation of the second embodiment will be described. Since the operation of the second embodiment has the same part as the operation of the first embodiment, only different parts will be described.

  FIG. 10 is a flowchart showing a gaze position estimation processing program according to the second embodiment.

  As shown in FIG. 10, when the saliency video calculation process in step 5S is executed, the gaze position estimation unit 6 executes the gaze position estimation process in step 6S. Details are as follows.

  The gaze position estimation unit 6 outputs an estimated gaze position that is a result of estimating a human gaze position at each position of the saliency image that is each frame of the saliency video input by the saliency video calculation unit 5.

  The method for estimating the gaze position is not particularly limited, but is described in a method in which the position where the pixel value of the saliency image is maximum is set as the estimated gaze position, Patent Document 6 (Japanese Patent Laid-Open No. 2009-259035), and the like. A method of estimating the gaze position based on the probabilistic model may be used.

  As described above, according to the gaze position estimation device according to the second embodiment, the saliency obtained by using the input image of the frame at each time constituting the input video and the acoustic signal constituting the input video. The gaze position can be estimated from the image.

 [Third embodiment]

  Next, a gaze position estimation apparatus according to the third embodiment will be described. In addition, since the structure of the gaze position estimation apparatus which concerns on 3rd Embodiment becomes the same as that of 1st Embodiment, it attaches | subjects the same code | symbol and abbreviate | omits description.

  In the gaze position estimation device according to the third embodiment, the image basic saliency selection unit 3 has the acoustic saliency based on the set of image basic saliency images and the acoustic saliency signal input as described above. A main image basic saliency component in a large time interval is selected (the image signal component is emphasized), and this is output as a main image basic saliency component.

  The selection method of the image basic saliency component is not particularly limited, but in the present embodiment, a correlation coefficient between the acoustic saliency signal based on the exponential smoothing method and the pixel value of the image basic saliency image is adopted. .

In this embodiment, a technique called exponential smoothing is used to predict a future time-series signal from time-series signals obtained up to the present time.
In exponential smoothing, a time series signal is predicted on the assumption that two time series signals are generated according to a simultaneous normal distribution. Two time series signals

Then each average value is

Similarly, the covariance is

Is calculated. Here, α is a predetermined constant or an output of a function that monotonously decreases with respect to time t. By using these statistics, the correlation coefficient of two time series signals

And mutual information

Is calculated as follows:

By using this exponential smoothing method, at each time t and pixel x, the acoustic saliency signal S _A (t) and each image basic saliency image

Correlation coefficient

And mutual information

Can be calculated. The square of this correlation coefficient or mutual information, or the binarized value is the importance of each feature type (index j), each time t, and each position x

Is output as a main image basic feature amount component.

  As another embodiment, a method of applying a spatial filtering process so that importance at adjacent pixel positions takes values close to each other can be considered.

  First, image basic saliency image

Is binarized. As a binarization method, for example, a method using an average pixel value as a threshold value can be considered. Next, binarized image basic saliency image

The importance image

To obtain an importance image in which the binarized image basic saliency image is non-zero only at non-zero pixel positions. This is subjected to a spatial smoothing filter such as a Gaussian filter.

Is used as the final importance and is output as the main image basic feature amount component.

  In yet another embodiment, the importance image

A method of manipulating the importance in accordance with the average and variance of the pixel values is also conceivable. Instead of this importance image, importance image with spatial filtering processing added

May be used. Importance image

The average value of pixels

The standard deviation

Then, each pixel value of the importance image after conversion

Is calculated as follows:

The image saliency image calculation unit 4 displays the saliency at each position of the input image based on the set of the input image basic saliency images, the main image basic saliency component, and if necessary, the acoustic saliency signal. A saliency image that is an image is output.

  The calculation method of the saliency image is not particularly limited, but in the present embodiment, a method of constructing the saliency image by selectively using the image basic saliency image selected as the main image basic saliency component Is adopted. In this embodiment, the main image basic feature amount component is given as the importance of each image basic saliency, each time, and each pixel position, and the saliency image S (t) at time t is calculated as follows. The

  Here, β is a predetermined constant. In other words, the first term is dominant when the acoustic saliency signal is close to 0, and the second term is dominant when the acoustic saliency signal is large. It means to control whether or not the result of the saliency selector 3 is reflected. When β = 0, all image basic feature components are used without using the selected main image basic feature component, and when β = 0, only the selected main image basic feature component is used. It becomes a special case.

  In addition, about the other structure and effect | action of the gaze position estimation apparatus which concern on 3rd Embodiment, since it is the same as that of 1st Embodiment, description is abbreviate | omitted.

 [Experimental result]

  Next, experimental results of the first embodiment of the present invention will be described. In this experiment, three types of images 1E to 3E within a length of 3.5 to 8.0 seconds were prepared as input images. The size of each video is 1024 × 576 pixels for video 1E and video 2E, and 1280 × 710 pixels for video 3E. In order to confirm the effect of the first embodiment, the degree of saliency video obtained by the method of the first embodiment and the known method is compared to how much human visual characteristics can be simulated. As a statistic that expresses human visual characteristics, we used the gaze position when humans were actually watching the input video. The input video was presented to 15 subjects, and the gaze position in the input video of each subject was sequentially measured using an existing gaze measurement device. Each subject was presented with each input video once in a random order. As a result, one time series of gaze positions was obtained for each subject and each input video. The gaze position was acquired for each subject and each input image by associating the time series of the gaze position with each frame (that is, the input image) of the input video while maintaining time consistency.

As an evaluation measure for the simulation of the visual characteristics between human employing a rating scale called normalized scan-path saliency (NSS) . This is the expected value obtained by normalizing the saliency value at the gaze position of the subject. From this definition, NSS is a scale that takes a larger value as the saliency value at the gaze position of the subject is larger. It is understood that there is. This NSS is calculated as follows. The saliency to be evaluated for the input image i _j (t) (t = 1, 2,..., T _j ) at the time t of the j-th input video I _j (j = 1, 2, 3). Let image S (t; I _j ) = {s (x, t; I _j )} _x . Further, the gaze position series of the subject n (n = 1, 2,..., N = 15) corresponding to the input video I _j is represented as V _n (I _j ) = {v _n (t; I _j )} _t . To do. At this time, the evaluation value NSS (t; I _j ) of the saliency image S (t; I _j ) at time t is calculated as follows.

However, s (t; I _j) and σ _S (t; I _j) is significantly level image S extracted from the input Film image I _j; pixel value s (x in _{(t I j), t;} I j ) Mean and variance.

The evaluation value NSS (I _j ) of the saliency video S (I _j ) = {S (t; I _j )} _t is obtained by averaging the evaluation values for the saliency images S (t; I _j ) at each time. It is obtained by.

  11, 13, and 15 schematically show the results of using NSS as an evaluation scale, and FIGS. 12, 14, and 16 show the evaluation results for each frame.

  11, FIG. 13, and FIG. 15 are diagrams each showing an overview of the evaluation results for the videos 1E to 3E. FIGS. 12, 14, and 16 show the evaluation results for each frame for the videos 1E to 3E. FIG.

As shown in FIG. 11, FIG. 13, and FIG. 15, in each of the case where only the time when the acoustic saliency is equal to or greater than the threshold θ _s is evaluated (upper column) and the case where all the times are evaluated (lower column). , NSS is larger in the proposed method (first embodiment) than in the conventional method according to Non-Patent Document 1. As described above, NSS is a scale that takes a larger value as the value of the saliency at the gaze position of the subject increases. Therefore, it can be seen from the results shown in FIG. 11, FIG. 13, and FIG. 15 that the method of the first embodiment obtains a better evaluation result than the conventional method.

Further, FIG. 11, as shown in FIG. 13, and FIG. 15, the threshold theta _s acoustic remarkable degree, in the case where the 0 optimal value (optimal), the value of NSS is not significantly different. Therefore, it can be seen that the method of the first embodiment for emphasizing main image feature amounts in frames with high acoustic saliency need not be applied only to frames with high acoustic saliency.

In FIG. 12, FIG. 14, and FIG. 16, the NSS value (left vertical axis) and acoustic saliency (surprise; right vertical axis) for each frame (Frame; horizontal axis) are shown. In FIG. 12, FIG. 14, and FIG. 16, a frame painted in gray indicates a frame corresponding to a time when the acoustic saliency is equal to or greater than a threshold value. The acoustic surprise corresponding to each frame is indicated by a solid line. The NSS value (Conventional) of the conventional method according to Non-Patent Document 1 is indicated by a dotted line. The NSS value (Surprise frame) of the proposed method (first embodiment) when the acoustic saliency threshold value θ _s is an optimum value (optimal) is indicated by a two-dot chain line. The NSS value (All frame) of the proposed method (first embodiment) when the acoustic saliency threshold θ _s is 0 is indicated by a one-dot chain line. As shown in FIGS. 12, 14, and 16, not only the NSS value in the frame corresponding to the time when the acoustic saliency is greater than or equal to the threshold value, but also the NSS value in the frame corresponding to the time when the acoustic saliency is less than the threshold value. Many of them are above a certain value. Therefore, from the results shown in FIGS. 12, 14, and 16, the method of the present invention for enhancing the selected image feature amount is not only for frames with high acoustic saliency but also for frames with low acoustic saliency. For many, it can be seen that the effect is high.

[Modification]
A program for executing each process of the saliency image generating device and the gaze position estimating device is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Accordingly, the above-described various processes relating to each of the saliency image generation apparatus and the gaze position estimation apparatus may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 Input part 1 Image basic saliency image calculation part 2 Acoustic saliency signal calculation part 3 Image basic saliency selection part 4 Image saliency image calculation part 5 Saliency video calculation part 11 Image basic feature image extraction part 12 Image multi-resolution Image extraction unit 13 Image resolution difference image extraction unit 14 Image time difference image extraction unit 15 Image basic saliency image extraction unit 21 Acoustic basic feature amount extraction unit 22 Acoustic saliency signal extraction unit 111 Luminance feature image extraction unit 112 Color feature image extraction Unit 113 direction feature image extraction unit 114 blinking feature image extraction unit 115 motion feature image extraction unit

Claims (8)

A basic saliency image indicating a degree of remarkable characteristics in the input image is generated for each of a plurality of feature types for an input image of each time frame constituting the input video, and a set of basic saliency images An image basic saliency image extracting unit,
An acoustic saliency signal calculation unit that generates an acoustic saliency signal indicating a degree of having a remarkable characteristic at each time for the acoustic signal constituting the input video;
For each of the plurality of feature types, for each time and each pixel, the pixel of the basic saliency image for the feature type included in the set of basic saliency images for the frame of the time, and the sound at the time An image basic saliency selection unit that calculates a correlation with a saliency signal and generates a main image basic saliency component based on the correlation for each time and each pixel for each of the plurality of feature types;
Image saliency that generates a saliency image indicating saliency at each position of the input image of each time frame based on the set of basic saliency images for each time frame and the main image basic saliency component. A degree image calculator,
A saliency image generating apparatus.   The image basic saliency selection unit, for each of the plurality of feature types, for each time and each pixel, a basic saliency image for the feature type included in the set of basic saliency images for the frame of the time A correlation value indicating a correlation between the pixel and the acoustic saliency signal at the time is calculated, and for each of the plurality of feature types, the number of times the correlation value exceeds a threshold is calculated. The saliency image generating apparatus according to claim 1, wherein a main image basic saliency component including the characteristic type that increases is also generated.   The image basic saliency selection unit, for each of the plurality of feature types, for each time and each pixel, a basic saliency image for the feature type included in the set of basic saliency images for the frame of the time A statistic indicating a correlation between the pixel and the acoustic saliency signal at the time, and for each of the plurality of feature types, based on the statistic for each time and each pixel, The saliency image generating apparatus according to claim 1, wherein a main image basic saliency component including importance for each time and each pixel is generated for each. The image basic saliency image extraction unit indicates a feature amount of the feature type of each pixel in the input image with respect to each of the plurality of feature types for the input image of each time frame constituting the input video. An image basic feature image is generated as a set of image basic feature images.
For each of the plurality of feature types, a basic saliency image in a spatial direction indicating a degree of having a spatially remarkable characteristic with respect to the image basic feature image for the feature type included in the set of image basic feature images, and A basic saliency image in a time direction indicating a degree having a remarkable characteristic in time is generated, and the basic saliency is generated based on the generated basic saliency image in the spatial direction and the basic saliency image in the time direction. Images are generated at predetermined time intervals, and set as the set of basic saliency images;
The acoustic saliency signal calculation unit extracts an acoustic basic feature amount at each time for an acoustic signal constituting the input video, and based on the extracted acoustic basic feature amount at each time, the image basic saliency level 3. The saliency image generating apparatus according to claim 1, wherein the acoustic saliency signal at each same time as the time when the basic saliency image is generated by the image extraction unit is generated at the predetermined time interval. The image basic saliency image extracting unit has a degree of spatial remarkable characteristics with respect to each of the plurality of feature types with respect to the image basic feature image corresponding to the feature type included in the set of image basic feature images. Generating a spatial saliency image and a temporal saliency image indicating the degree of temporal characteristics, and generating the generated spatial saliency image and temporal saliency image. Generating the basic saliency image based on a degree image and a predetermined probability model, and as a set of the basic saliency images,
The acoustic saliency signal calculation unit extracts an acoustic basic feature amount at each time from the acoustic signal constituting the input video, and the extracted acoustic basic feature amount at each time, and the predetermined probability model The saliency image generation device according to claim 4, wherein the saliency signal at each time is generated based on the sig- nal.   The saliency image generating apparatus according to claim 5, wherein the probability model is a gamma distribution. The image basic saliency image extraction unit generates a basic saliency image indicating the degree of remarkable characteristics in the input image for each of a plurality of feature types for the input image of each time frame constituting the input video. And a set of basic saliency images,
The acoustic saliency signal calculating unit generates an acoustic saliency signal indicating the degree of remarkable characteristics at each time for the acoustic signals constituting the input video,
An image basic saliency selection unit, for each of the plurality of feature types, for each time and each pixel, a basic saliency image for the feature type included in the set of basic saliency images for the frame of the time Calculating a correlation between the pixel and the acoustic saliency signal at the time, and generating a main image basic saliency component based on the correlation for each time and each pixel for each of the plurality of feature types;
The image saliency image calculation unit indicates the saliency at each position of the input image of the frame at each time based on the set of basic saliency images for the frame at each time and the main image basic saliency component. A method for generating a saliency image.   The program for functioning a computer as each part of the saliency image generation apparatus of any one of Claims 1-6.