JP5235770B2 - Striking area image generation method, saliency area image generation apparatus, program, and recording medium - Google Patents

Description

  The present invention relates to a saliency image generation method, a saliency image generation device, a program, and a recording medium. In particular, the present invention calculates prior information about a saliency area (object area) and a background area (non-salience area) by signal processing using human visual characteristics, and uses the calculated prior information to make a significant difference from the input video. The present invention relates to a saliency area image generation method, a saliency area image generation apparatus, a program, and a recording medium that divide an area and a non-salience area with high accuracy.

  Extracting regions of interest (hereinafter referred to as “object regions”) such as people, animals, and objects from images and video separately from regions other than object regions such as backgrounds (hereinafter referred to as “background regions”) Area segmentation technology can be applied to free image synthesis without using chroma keys, object recognition and image retrieval robust to changes in the background area, and image video coding that can adjust the bit rate according to the importance of the area. It is an important technology with a wide range. As a region segmentation technique, a method is known in which image region segmentation is formulated as a posterior probability maximization problem for a certain statistical model, and the posterior probability maximization problem is solved by finding the minimum cut of the graph equivalent to the statistical model. (For example, refer nonpatent literature 1). In addition, the method described in Non-Patent Document 1 is extended to a video signal, and a video segmentation is realized at high speed by efficiently obtaining a minimum cut of a graph using temporal continuity of the video signal. Is also known (see, for example, Non-Patent Document 2).

Y. Boykov and G.F.Lea, “Graph cuts and efficient N-D image segmentation,” International Journal of Computer Vision, Vol. 70, No. 2, pp. 109-131, 2006. P. Kohli and P. Torr, “Dynamic graph cuts for efficient inference in Markov random fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, No. 12, pp. 2079-2088, 2007.

  In the application of region segmentation technology, a method that can correctly segment the object region and the background region even in the situation where prior information (cue) about the object region and the background region is not given, in other words, including region segmentation without human intervention. Therefore, a method that can automatically execute all the procedures is desired. This is because the application range of the area division technique is expanded. However, since the methods described in Non-Patent Documents 1 and 2 assume that prior information on the object region / background region is partially manually given, no prior information on the object region / background region is given at all. If not, there is a problem that it cannot be used. That is, there is a problem that the application range of the region division technique is remarkably limited.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique for enabling region division even when prior information regarding an object region and a background region is not given at all. .

  In order to solve the above problem, a remarkable area image generation method according to an aspect of the present invention includes an attention level image extraction process for extracting an attention level image indicating a degree of attention that is a degree to which a person is likely to pay attention from an input image. A saliency area prior probability image extraction process for extracting a saliency area prior probability image indicating the probability that each position of the input image, which is each frame constituting the input video, is a saliency area, and the saliency area and saliency area of the input image From the feature amount likelihood calculation process for calculating the feature amount likelihood indicating the likelihood of the image feature amount included in each of the outer regions, the input image, the saliency region prior probability image, and the feature amount likelihood, the input A saliency area image extracting unit for extracting a saliency area image indicating a saliency area of the image; and for each of the input images, the attention degree video extraction process, the saliency area prior probability image extraction process, and the feature amount likelihood. And a saliency area image generation process for generating the saliency area image from the saliency area images obtained by executing the saliency area image extraction process and the saliency area prior probability image extraction process. A remarkable area prior probability image indicating the probability that each position of the input image is a remarkable area, an attention degree image that is an image corresponding to the input image in the attention degree video extracted by the attention degree video extraction process, and The feature amount likelihood calculation process extracts the feature amount likelihood from the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the previous time. It is calculated based on at least one of the calculated feature amount likelihoods.

  In the saliency area video generation method, the attention level image extraction step is a basis for calculating a basic attention level image that is an image displaying a spatial area having a remarkable characteristic in the input image based on the input image. Attention level image extraction process and a stochastic basic attention level image that is an image displaying prominence at each position of the current input image using a probabilistic expression was calculated by the basic attention level image extraction process The basic attention degree image, the probabilistic basic attention degree image calculated by the probabilistic basic attention degree image extraction process from the previous input image, and a first parameter that is sequentially updated and used for gaze position estimation. A stochastic basic attention degree image extraction process that is calculated based on a stochastic basic attention degree parameter, and a gaze position probability density image that is a frame of the gaze position probability density image in the current input image The probabilistic basic attention degree image calculated by the probabilistic basic attention level image extraction process, and the gaze position probability density image calculated by the gaze position probability density image extraction process from the previous input image, and sequentially updated A gaze position probability density image extraction process calculated based on a gaze position probability density parameter that is a second parameter used for gaze position estimation, the basic attention degree image extraction process, and the stochastic basic attention degree The time-series gaze position probability density image calculated by sequentially repeating the image extraction process and the gaze position probability density image extraction process for each of the input images is output as the gaze position probability density image. A line-of-sight position probability density image output process, and the line-of-sight position probability density image extraction process is a line-of-sight movement state that is a random variable that controls the magnitude of line-of-sight movement The line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction process from the previous input image, and the line-of-sight movement state variable calculated by the line-of-sight movement state variable update process from the previous input image And a line-of-sight movement state variable update process for outputting a line-of-sight movement state variable set that is a set of the line-of-sight movement state variables, and a representative line of sight in consideration of line-of-sight movement. A representative line-of-sight position set, which is a set of representative line-of-sight positions indicating a position, is obtained by a probabilistic basic attention level image calculated by the probabilistic basic attention level image extraction process and a representative gaze position update process from the previous input image. Representative visual line position update process for updating based on the updated representative visual line position set, the visual line movement state variable set, and the visual line position probability density parameter, respectively. A representative gaze position weight coefficient set, which is a set of representative gaze position weight coefficients composed of weights associated with the representative gaze position, and a probabilistic basic attention image calculated by the probabilistic basic attention image extraction process; The representative gaze position calculated based on the representative gaze position set updated by the representative gaze position update process, the gaze movement state variable set output from the gaze movement state variable update process, and the gaze position probability density parameter. Based on the weight coefficient calculation process, the representative eye position position set updated by the representative eye position update process, and the representative eye position weight coefficient set calculated by the representative eye position weight coefficient calculation process, the eye position probability density image A visual line position probability density image output process for calculating the representative visual line position set and the representative visual line position weighting coefficient set It may be calculated the linear position probability density image.

  In the saliency area image generation method, the saliency area prior probability image extraction process uses the saliency area prior probability image generation process for generating the saliency area prior probability image using only the attention level image, and the saliency area image. And a saliency area prior probability image update process for updating the saliency area prior probability image generated by the saliency area prior probability image generation process.

  In the saliency area video generation method, the feature amount likelihood calculation step includes the saliency area feature amount likelihood indicating the likelihood of the image feature amount included in the saliency area, the input image, the saliency area prior probability image, A saliency area feature amount likelihood calculation process based on at least one of the saliency area image and the saliency area feature amount likelihood calculated up to the previous time, and the likelihood of the image feature amount included in the area outside the saliency area Is calculated based on at least one of the input image, the saliency area prior probability image, the saliency area image, and the non-salience area feature amount likelihood calculated until the previous time. A non-significant area feature quantity likelihood calculation process and a feature quantity likelihood output process of adding the saliency area feature quantity likelihood and the non-salience area feature quantity likelihood to output as a feature quantity likelihood. even if There.

  In the saliency area video generation method, the saliency area feature amount likelihood calculating step includes generating a saliency area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image. A feature amount likelihood generation process and a saliency area feature amount likelihood update process for updating the saliency area feature amount likelihood generated by the saliency area feature amount likelihood generation process. The degree calculation process includes: a non-salience area feature amount likelihood generation process for generating the non-salience area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image; A non-salience area feature amount likelihood update process for updating the non-salience area feature amount likelihood generated by the feature amount likelihood generation process, and the saliency area feature amount likelihood update process includes the input image Updating the saliency area feature quantity likelihood based on at least one of the saliency area image and the updated saliency area feature quantity likelihood updated up to the previous time, and updating the non-salience area feature quantity likelihood May update the non-significant region feature value likelihood based on at least one of the input image, the non-significant region image, and the updated non-significant region feature value likelihood updated so far. Good.

  The saliency area image generation method includes a smoothed image group generation process for generating a smoothed image group including a plurality of smoothed images obtained by smoothing the input image at different resolutions, and the leveled image group, A saliency area prior probability image extraction process, the feature amount likelihood calculation process, and the saliency area image extraction process, and further including a saliency area image determination process for determining the saliency area image of the input image, The quantity likelihood calculation process and the saliency area image extraction process use the smoothed image instead of the input image, and the saliency area image generation process performs the attention level image extraction process and the saliency for each input image. The saliency obtained by executing an area prior probability image extraction process, the feature likelihood calculation process, the saliency area image extraction process, the smoothed image group generation process, and the saliency area image determination process. It may generate the salient region image from the area image.

  In order to solve the above-described problem, a remarkable area image generation device according to another aspect of the present invention extracts an attention level image that extracts an attention level image indicating a degree of attention that is a degree to which a person is likely to pay attention from an input image. A saliency area prior probability image extraction unit that extracts a saliency area prior probability image indicating a probability that each position of the input image that is each frame constituting the input video is a saliency area, and a saliency area of the input image and From the feature amount likelihood calculating unit that calculates the likelihood of the image feature amount included in each of the regions outside the saliency region, the input image, the saliency region prior probability image, and the feature amount likelihood, A saliency area image extraction unit that extracts a saliency area image indicating a saliency area of the input image; and, for each of the input images, the attention level video extraction unit, the saliency area prior probability image extraction unit, and the feature amount likelihood calculation. And a saliency area image generation unit that generates the saliency area image from each saliency area image obtained by executing the saliency area image extraction unit, and the saliency area prior probability image extraction unit includes the one input image A noticeable area prior probability image indicating a probability that each position of the noticeable area is a noticeable area, an attention degree image that is an image corresponding to the input image in the attention degree video extracted by the attention degree video extraction unit, and the salient area The feature amount likelihood calculating unit extracts the feature amount likelihood from the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the previous time. The calculation is based on at least one of the feature amount likelihoods.

  In the saliency area video generation device, the attention level video extraction unit calculates a basic attention level image that is an image displaying a spatial area having a remarkable characteristic in the input image based on the input image. The basic attention degree image extraction unit calculates a probabilistic basic attention degree image that is an image displaying the saliency at each position of the current input image using a probabilistic expression. The basic attention image, the probabilistic basic attention image calculated by the probabilistic basic attention image extraction unit from the previous input image, and a first parameter that is sequentially updated and used for gaze position estimation. A probabilistic basic attention degree image extraction unit that is calculated based on a probabilistic basic attention degree parameter, and a gaze position probability density image that is a frame of the gaze position probability density image in the current input image; The probabilistic basic attention level image calculated by the static basic attention level image extraction unit and the gaze position probability density image calculated by the gaze position probability density image extraction unit from the previous input image are sequentially updated, and the line of sight A gaze position probability density image extraction unit that is calculated based on a gaze position probability density parameter that is a second parameter used for position estimation, the basic attention level image extraction unit, and the probabilistic basic attention level image extraction unit And a line-of-sight position probability density image extracting unit that outputs the line-of-sight position probability density image in time series calculated as the line-of-sight position probability density image as the line-of-sight position probability density image. A probability density image output unit, and the line-of-sight position probability density image extraction unit converts a line-of-sight movement state variable, which is a random variable that controls the amount of line-of-sight movement, into the previous input image. The line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction unit, the line-of-sight movement state variable calculated by the line-of-sight movement state variable update unit from the previous input image, and the line-of-sight position probability density mother A line-of-sight movement state variable update unit that outputs a line-of-sight movement state variable set that is a set of the line-of-sight movement state variables, and a set of representative line-of-sight positions indicating representative line-of-sight positions that take line-of-sight movement into account A representative gaze position set calculated by the probabilistic basic attention level image extraction unit, and a representative gaze position set updated by the representative gaze position update unit from the previous input image, , A representative line-of-sight position update unit for updating based on the line-of-sight movement state variable set and the line-of-sight position probability density parameter, and weights associated with the representative line-of-sight positions The representative gaze position weighting coefficient set, which is a set of representative gaze position weighting coefficients, is updated by the probabilistic basic attention degree image calculated by the probabilistic basic attention degree image extracting unit and the representative gaze position updating unit. A representative gaze position weight coefficient calculating unit that calculates a representative gaze position set, a gaze movement state variable set output from the gaze movement state variable update unit, and the gaze position probability density parameter; and the representative gaze position A gaze position probability density image output unit that calculates the gaze position probability density image based on the representative gaze position weight set updated by the update unit and the representative gaze position weight coefficient set calculated by the representative gaze position weight coefficient calculation unit The visual line position probability density image including the representative visual line position set and the representative visual line position weighting coefficient set may be calculated.

  In the saliency area video generation device, the saliency area prior probability image extraction unit uses the saliency area prior probability image generation part that generates the saliency area prior probability image using only the attention level image, and the saliency area image. And a saliency area prior probability image update unit that updates the saliency area prior probability image generated by the saliency area prior probability image generation unit.

  In the saliency area video generation device, the feature amount likelihood calculating unit calculates the saliency area feature amount likelihood indicating the likelihood of the image feature amount included in the saliency area, the input image, the saliency area prior probability image, the The saliency area feature amount likelihood calculation unit that calculates based on at least one of the saliency area image and the saliency area feature amount likelihood calculated up to the previous time, and the likelihood of the image feature amount included in the area outside the saliency area Is calculated based on at least one of the input image, the saliency area prior probability image, the saliency area image, and the non-salience area feature amount likelihood calculated until the previous time. A non-significant region feature amount likelihood calculating unit and a feature amount likelihood output unit that adds the saliency region feature amount likelihood and the non-significant region feature amount likelihood and outputs the result as a feature amount likelihood. It may be.

  In the saliency area video generation device, the saliency area feature amount likelihood calculating unit generates the saliency area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image. The non-significant area feature quantity includes a feature quantity likelihood generation section and a saliency area feature quantity likelihood update section that updates the saliency area feature quantity likelihood generated by the saliency area feature quantity likelihood generation section. The likelihood calculation unit includes a non-salience region feature amount likelihood generation unit that generates the non-salience region feature amount likelihood based on the input image, the saliency region prior probability image, and the saliency region image; A non-significant region feature amount likelihood update unit that updates the non-salience region feature amount likelihood generated by the region feature amount likelihood generation unit, and the saliency region feature amount likelihood update unit includes the input image The prominent territory The saliency area feature quantity likelihood is updated based on at least one of the image and the updated saliency area feature quantity likelihood updated up to the previous time, and the non-salience area feature quantity likelihood update unit is configured to input the input The non-salience area feature amount likelihood may be updated based on at least one of the image, the non-salience area image, and the updated non-salience area feature amount likelihood updated so far.

  The saliency area video generation device includes a smoothed image group generation unit that generates a smoothed image group including a plurality of smoothed images obtained by smoothing the input image with different resolutions, and the leveled image group, A saliency area prior probability image extraction unit, the feature amount likelihood calculation unit, and a saliency area image extraction unit, and a saliency area image determination unit that determines the saliency area image of the input image. The feature amount likelihood calculation unit and the saliency area image extraction unit use the smoothed image instead of the input image, and the saliency area image generation unit performs the attention level image extraction unit, The saliency area image obtained by executing each process of the saliency area prior probability image extraction unit, the feature amount likelihood calculation unit, the saliency area image extraction unit, the smoothed image group generation unit, and the saliency area image determination unit It may generate an al the salient region image.

  In order to solve the above problem, a program according to another aspect of the present invention includes an attention level video extraction step of extracting an attention level video indicating a level of attention that is a degree to which a person is likely to pay attention from an input video, and an input A saliency area prior probability image extracting step for extracting a saliency area prior probability image indicating a probability that each position of the input image which is each frame constituting the video is a saliency area; The feature amount likelihood calculating step for calculating the feature amount likelihood indicating the likelihood of the image feature amount included in each region, and the input image, the saliency region prior probability image, and the feature amount likelihood A saliency area image extraction step for extracting a saliency area image indicating a saliency area; and, for each of the input images, the attention degree video extraction step and the saliency area prior probability image extraction stage. And a saliency area image generation step for generating the saliency area image from the saliency area images obtained by executing the feature amount likelihood calculation step and the saliency area image extraction step. The saliency area prior probability image extraction step includes a saliency area prior probability image indicating the probability that each position of the one input image is a saliency area in the attention level video extracted by the attention level video extraction step. Based on the attention level image that is an image corresponding to the input image and the saliency area image, and the feature amount likelihood calculating step includes calculating the feature amount likelihood as the input image, the attention level image, It is calculated based on at least one of the saliency area prior probability image, the saliency area image, and the feature amount likelihood calculated until the previous time. That.

  In order to solve the above problem, a recording medium according to another aspect of the present invention includes an attention level video extraction step for extracting an attention level video indicating a degree of attention that is a degree to which a human is easily directed from an input video; A saliency area prior probability image extracting step for extracting a saliency area prior probability image indicating a probability that each position of the input image which is each frame constituting the input video is a saliency area; From the feature amount likelihood calculating step for calculating the feature amount likelihood indicating the likelihood of the image feature amount included in each of the regions, the input image, the saliency region prior probability image, and the feature amount likelihood, the input image A saliency area image extracting step for extracting a saliency area image indicating a saliency area of the image, a focus level image extraction step, and a saliency area prior probability image extraction step for each input image. A program for causing a computer to execute a saliency area image generation step for generating the saliency area image from each saliency area image obtained by executing the feature amount likelihood calculation step and the saliency area image extraction step. In the recorded computer-readable recording medium, the remarkable area prior probability image extraction step includes: a remarkable area prior probability image indicating a probability that each position of the one input image is a remarkable area; Extracting based on the attention level image and the saliency area image, which are images corresponding to the input image in the attention level video extracted in the extraction step, and the feature amount likelihood calculating step includes calculating the feature amount likelihood. , The input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the feature amount calculated up to the previous time And calculating, based on at least one time.

  According to the present invention, it is possible to divide a region even when no prior information regarding the object region / background region is given. Therefore, even if there is no prior knowledge about the object region / background region, the object region and the background region can be divided with high accuracy and the region of interest (object region) can be extracted.

It is a schematic diagram of the calculation process of the saliency area prior probability image by the saliency area image generation device 1000 according to the first embodiment of the present invention. 3 is an example of a functional block diagram of a saliency area image generation device 1000. FIG. It is a functional block diagram of the attention level video extraction unit 1. 4 is a functional block diagram of a saliency area prior probability image extraction unit 2, a feature amount likelihood calculation unit 3, and a saliency area image extraction unit 4. FIG. It is a schematic diagram of the feature amount likelihood calculation process. It is an example of a remarkable area | region extraction graph. 3 is an example of a functional block diagram of a saliency area image generation device 1100. FIG. It is an example of remarkable area | region extraction. Comparison of salient region extraction

(First embodiment)
Hereinafter, a saliency image generating apparatus 1000 according to a first embodiment of the present invention will be described with reference to the drawings. In the present embodiment (the same applies to the second embodiment described later), in order to realize area division based on video saliency, hereinafter, “significant area extraction” and “area division” are used synonymously. Similarly, “saliency area” and “object area” are used synonymously, and “non-salience area” and “background area” are used synonymously. In the following description, a character with a ￣ in the upper part of the character in the formula is indicated with a ￣ in front of the character in the sentence. For example, a character in the formula (the following formula 1) is indicated as ￣x in the sentence.

  In addition, a character with “˜” attached to the upper part of the character in the formula is indicated by “˜” before the character in the sentence. For example, a character in the formula (Formula 2 below) is indicated by ˜η in the sentence, and a character in the formula (Formula 3 below) is indicated by ˜Σ in the sentence.

  In addition, the character in a formula (following formula 4) and g in a sentence are the same.

  As shown in FIG. 1, the saliency area video generation apparatus 1000 obtains an input video from the outside, and extracts each saliency area frame (extracting each saliency area from each input frame (each input image) constituting the input video ( A saliency area image composed of each saliency area image) is generated and output to the outside.

  As shown in FIG. 2, the saliency area image generation apparatus 1000 includes an attention degree image extraction unit 1, a saliency area prior probability image extraction unit 2, a feature amount likelihood calculation unit 3, a saliency area image extraction unit 4, and a saliency area image generation. Part 5 is provided. As shown in FIG. 3, the attention level video extraction unit 1 includes a basic attention level image extraction unit 11, a stochastic basic attention level image extraction unit 12, a stochastic basic attention level parameter sequential estimation unit 13, and a gaze position probability density image. An extraction unit 14 and a line-of-sight position probability density video output unit 15 are provided. The gaze position probability density image extraction unit 14 includes a gaze movement state variable update unit 141, a representative gaze position update unit 142, a representative gaze position weight coefficient calculation unit 143, a gaze position probability density image output unit 144, and a representative gaze position set reconstruction unit. 145. As shown in FIG. 4A, the saliency area prior probability image extraction unit 2 includes a saliency area prior probability image generation unit 21 and a saliency area prior probability image update unit 22. The feature amount likelihood calculation unit 3 includes a saliency region feature amount likelihood calculation unit 31, a non-salience region feature amount likelihood calculation unit 32, and a feature amount likelihood output unit 33, as shown in FIG. The saliency area feature quantity likelihood calculation unit 31 includes a saliency area feature quantity likelihood generation unit 311 and a saliency area feature quantity likelihood update unit 312. The non-significant region feature quantity likelihood calculating unit 32 includes a non-significant region feature amount likelihood generating unit 321 and a non-significant region feature amount likelihood updating unit 322. As shown in FIG. 4C, the saliency area image extraction unit 4 includes a saliency area extraction graph generation unit 41 and a saliency area extraction graph division unit 42.

  The attention level video extraction unit 1 acquires an input video. The attention level video extracting unit 1 extracts a attention level video that is a video showing a level of attention, which is the degree to which a person is likely to pay attention in each frame of the input video. The attention level video extraction unit 1 outputs (supplies) the extracted attention level video to the saliency prior probability image extraction unit 2.

Specifically, the attention level video extraction unit 1 includes an input video that is a target of eye gaze position estimation, a stochastic basic attention degree parameter Θ _s (t) that is a first parameter necessary for gaze position estimation, And a gaze position probability density parameter Θ _x (t), which is a second parameter necessary for gaze position estimation, is input at each position in each time-series input image (each frame) included in the input video. Then, a line-of-sight position probability density image X (t) showing the probability that the person turns the line of sight is calculated. Furthermore, the gaze position estimation apparatus 100 outputs a gaze position probability density image that is a time-series image of the calculated gaze position probability density image X (t).

  The basic attention level image extraction unit 11 extracts an input image (frame) for estimating the line-of-sight position from the input video input. Further, the basic attention level image extraction unit 11 extracts a basic attention level image that is an image showing a spatial region having a remarkable characteristic in the extracted input image. Then, the extracted basic attention level image is output to the probabilistic basic attention level image extraction unit 12 and the stochastic basic attention level parameter sequential estimation unit 13.

  Since the basic attention level image extraction processing in the basic attention level image extraction unit 11 is the same as the processing of the basic attention level image extraction unit 11 described in Patent Document 1, detailed description of the processing content is omitted. However, in the present embodiment, the basic attention degree image calculated from the input image at time i is represented by the following equation (5) (hereinafter referred to as “basic attention degree image ￣S (i)”).

The probabilistic basic attention level image extraction unit 12 extracts a probabilistic basic attention level image S (t) that is an image in which saliency at each position of the current input image is displayed using a probabilistic expression. Note that the stochastic basic attention level image S (t) is extracted by the probabilistic basic attention level image extraction unit 12 based on the basic attention level image ￣S (i) input from the basic attention level image extraction unit 11 and the probabilistic basic attention level image extraction unit 12. This is performed based on the previous stochastic basic attention level image S (t) calculated by the basic attention level image extraction unit 12 and the stochastic basic attention level parameter Θ _s (t).
The probabilistic basic attention image S (t) extracted by the probabilistic basic attention image extracting unit 12 is an expected value of the probabilistic basic attention s (t, y) at each position y (the following formula ( 6) An image holding a standard deviation σs (t, y | t) (hereinafter, expressed as “expected value ^ s (t, y | t)”).

The stochastic basic attention level image extraction unit 12 outputs the extracted probabilistic basic attention level image S (t) to the gaze position probability density image extraction unit 14 and the stochastic basic attention level parameter sequential estimation unit 13.
The stochastic basic attention degree image extraction unit 12 receives the stochastic basic attention degree parameter Θ _s (t + 1) updated by the stochastic basic attention degree parameter sequential estimation unit 13.

  The extraction of the stochastic basic attention level image S (t) in the stochastic basic attention level image extraction unit 12 can be calculated by the methods described in Non-Patent Documents 1 and 2. The method of extracting the stochastic basic attention level image S (t) in the stochastic basic attention level image extracting unit 12 is not particularly limited, but an estimation method using a Kalman filter will be described as an example.

  First, the pixel value s (t, y) (probability variable) at the position y of the current (time t) stochastic basic attention image S (t) (probability variable) is the current basic attention image. (7) (hereinafter referred to as “pixel value ￣ s (t, y)”) (7) (hereinafter referred to as “pixel value ￣ s (t, y)”) , And the pixel value s (t−1, y) at the position y of the stochastic basic attention level image S (t−1) before one time point (time t−1), the following formulas (9) and (10) It is assumed that the relational expression is satisfied.

Here, it is assumed that the stochastic basic attention degree parameter Θ _s (t) is given by the following equation (11) in a form depending on the time t and the position y.

  In the above formulas (9) and (10), p (a | b) indicates the probability density of a when b is given. Also, the following formula (12) shows the probability density of s according to the normal distribution whose expected value is the following formula (13) and the standard deviation is σ, and is expressed as the following formula (14).

  In the following description, the pixel value ￣s (t, y) is referred to as the basic attention level at the position y. Similarly, the stochastic basic attention level s (t, y) is referred to as the stochastic basic attention level at the position y. Further, the position y is omitted unless particularly necessary. For example, s (t, y) is represented as s (t).

  Subsequently, the stochastic basic attention level s (t−1) one point before is expressed by using the probability density as in the following formula (15) by the processing of the probabilistic basic attention level image extraction unit 12 so far. It is assumed that it has been extracted at

In the above formula (15), the following formula (16) is a series of basic attention degrees from time t ₁ to time t _2, and the following formula (17) is the following formula that is basic attention degrees from time 1 to time t _2. The expected value σ _s (t ₁ | t ₂ ) of the stochastic basic attention degree s (t ₁ ) at time t ₁ when (18) is given indicates the standard deviation at this time.

At this time, the stochastic basic attention level image extraction unit 12 uses the following formula (20) (hereinafter, “expectation”) which is an expected value in the probability density shown in the following formula (19) of the current stochastic basic attention level s (t). Value ^ s (t | t) ") and standard deviation σ _s (t | t) are updated as in the following equations (21) and (22).

Note that the update of the expected value ^ s (t | t) and the standard deviation σ _s (t | t) in the above-described stochastic basic attention level image extraction unit 12 may be executed independently at each position in the image. it can.

  Further, by using a relational expression such as the following expression (23) instead of the above expression (10), an embodiment in which a motion component at each position of the input image is taken into consideration can be obtained.

  In the above equation (23), Δy (t) is an optical flow at time t and position y. For example, the same method as the motion feature image extraction unit 115 described in Patent Document 1 is used.

  In the estimation method using the Kalman filter described above, the stochastic basic attention degree s (t, y) at each position is extracted independently in the spatial direction. It can also be introduced. Hereinafter, a description of a stochastic basic attention degree based on a statistical model called a dynamic Markov random field will be described, and a stochastic basic attention degree s (t, y) will be derived analytically by a statistical analysis technique called mean field approximation. Is described.

  First, the pixel value s (t, y) (probability variable) at the position y of the current (time t) probabilistic basic attention image S (t) (probability variable) is the current basic attention image ￣S (t ) At the position y of the probabilistic basic attention level image S (t−1) one time before (time t−1). ), And the following formula (25) that is the pixel value of the current probabilistic basic attention level image S (t) in the following formula (24) that is each position included in the vicinity D (y) of the position y: It is assumed that the relational expressions (26) to (30) are satisfied.

Here, it is assumed that the stochastic basic attention degree parameter Θ _s (t) is redefined as shown in the following formula (31) in a form depending on the time t and the position y.

  As a method for determining the neighborhood D (y), for example, four points on the top, bottom, left, and right of the position y, or eight points including four additional oblique positions may be considered.

  Subsequently, as in the estimation method using the Kalman filter described above, the stochastic basic attention level s (t−1, y) one point before is obtained by the processing of the probabilistic basic attention level image extraction unit 12 so far. It is assumed that it is extracted by an expression using a probability density such as the following formula (32).

In the above equation (32), the following equation (33) is a stochastic basic attention degree s at time t ₁ and position y when the following expression (34) which is a basic attention degree image up to time t ₂ is given. The expected value of (t ₁ , y), σ _s (t ₁ , y | t ₂ ) indicates the standard deviation at this time.

At this time, the stochastic basic attention level image extraction unit 12 expects the current value of the probabilistic basic attention level s (t, y) at the position y in the probability density represented by the following equation (35) ^ s (t, y | The objective is to update t) and the standard deviation σ _s (t, y | t).

  The update by the probabilistic basic attention level image extraction unit 12 is performed by the following formulas (36) to (41) using repetitive calculation.

  In the above formulas (36) to (41), | D (y) | indicates the number of elements of the set D (y). In addition, in the calculation using the above formulas (36) to (41), it is impossible to repeat the infinite steps as shown in the above formula (39). When the difference between (42) and the following equation (43) which is the output of the l-th step becomes sufficiently small, the calculation is repeatedly iterated.

Further, when the index 1 relating to the step is fixed, the update shown in the above equation (38) can be calculated independently at each position of the image. Further, other update formulas can be calculated independently at each position of the image by fixing the time t.
As a result, the update of the expected value ^ s (t, y | t) and the standard deviation σ _s (t, y | t) in the above-described stochastic basic attention level image extraction unit 12 is estimated using the above-described Kalman filter. Similar to the method, it can be executed independently at each position in the image, and these update processes can be easily parallelized.

The stochastic basic attention degree parameter sequential estimation unit 13 includes a basic attention degree image ￣S (i) input from the basic attention degree image extraction unit 11 and a stochastic basis input from the stochastic basic attention degree image extraction unit 12. Based on the attention degree image S (t) and the probabilistic basic attention degree parameter Θ _s (t) which is a parameter given in advance, the probabilistic basic attention degree parameter Θ _s (t) is sequentially obtained. Update.
The stochastic basic attention degree parameter sequential estimation unit 13 outputs the updated stochastic basic attention degree parameter Θ _s (t + 1) to the stochastic basic attention degree image extraction unit 12.

Note that the stochastic basic attention degree parameter sequential estimation unit 13 does not update the stochastic basic attention degree parameter Θ _s (t), and the probabilistic basic attention degree mother, which is a parameter given in advance. The number Θ _s (t) is output to the stochastic basic attention level image extraction unit 12 as a stochastic basic attention level parameter Θ _s (t + 1). That is, since the stochastic basic attention degree parameter Θ _s (t) cannot be updated in the initial stage where the stochastic basic attention degree image S (t) is not input from the stochastic basic attention degree image extraction unit 12, The probabilistic basic attention degree parameter Θ _s (t) is output to the probabilistic basic attention degree image extraction unit 12 as it is.

The estimation method of the stochastic basic attention degree parameter Θ _s (t + 1) in the stochastic basic attention degree parameter sequential estimation unit 13 is not particularly limited, but in this embodiment, the estimation method using an adaptive Kalman filter Is described.

In the stochastic basic attention degree parameter sequential estimation unit 13, the stochastic basic attention degree parameter Θ _s (t + 1) used at the next time t + 1 is represented by the following formula (44).

  The stochastic basic attention level parameter sequential estimation unit 13 is calculated by the basic attention level image ￣S (i) that has already been calculated by the basic attention level image extraction unit 11 and the probabilistic basic attention level image extraction unit 12. Using the expected value and standard deviation of the probabilistic basic attention degree that constitutes the probabilistic basic attention degree image S (t), calculation is performed as in the following formulas (45) to (52).

In the above formulas (45) to (52), the following formula (53) (hereinafter referred to as “￣σ _s1 ”) and the following formula (54) (hereinafter referred to as “￣σ _s2 ”) are basic stochastic basic attentions. It is a degree parameter and is determined in advance or calculated by learning in advance.

Also, λ _s1 and λ _s2 are predetermined mixing ratios of the parameters, and by appropriately determining these numerical values, the parameters obtained by successive updating are _expressed by the following equations (55) and (56). And the predetermined parameters ￣σ _s1 and ￣σ _s2 can be controlled.

Note that setting λ _s1 = λ _s2 = 0 is equivalent to not performing the process of estimating the stochastic basic attention degree parameter Θ _s (t + 1) by the stochastic basic attention degree parameter sequential estimation unit 13. N _s is the time length of a buffer that holds past information.

The gaze position probability density image extraction unit 14 includes a gaze movement state variable update unit 141, a representative gaze position update unit 142, a representative gaze position weight coefficient calculation unit 143, a gaze position probability density image output unit 144, and a representative gaze position set reconstruction unit. 145.
The line-of-sight position probability density image extraction unit 14 extracts a line-of-sight position probability density image X (t) that is a frame constituting the line-of-sight position probability density image. Note that the eye-gaze position probability density image X (t) is extracted by the eye-gaze position probability density image extraction unit 14 from the probabilistic basic attention level image S (t) input from the probabilistic basic attention level image extraction unit 12 and the line of sight. This is performed based on the line-of-sight position probability density image X (t) extracted by the position probability density image extraction unit 14 and the line-of-sight position probability density parameter Θ _x (t) given in advance.
Further, the line-of-sight position probability density image extraction unit 14 outputs the line-of-sight position probability density image X (t) to the line-of-sight position probability density video output unit 5.

The line-of-sight movement state variable updating unit 141 includes a line-of-sight position probability density image X (t) that has been output from the representative line-of-sight position set reconstruction unit 145 and a line-of-sight position probability that is a parameter given in advance. Based on the density parameter Θ _x (t), the gaze movement state variable u (t), which is a random variable for controlling the magnitude of the gaze movement included in the gaze position probability density image X (t) so far, is obtained. Update.
Also, the line-of-sight movement state variable update unit 141 converts the line-of-sight movement state variable set U (t), which is a set of the updated line-of-sight movement state variables u (t), to the representative line-of-sight position update unit 142 and the representative line-of-sight position set reconstruction unit 145. Output to.

  The method of updating the line-of-sight movement state random variable set U (t) in the line-of-sight movement state variable updating unit 141 is not particularly limited, but the method according to this embodiment will be described.

  First, as part of the output of the representative line-of-sight position set reconstruction unit 145, the line-of-sight movement state variable set U (t-1) one point before (time t-1) is given as in the following equation (57). It shall be.

In the above equation (57), _Nu represents the number of elements of the line-of-sight movement state variable set, that is, the number of samples of the line-of-sight movement state variable. Each eye movement state variable shall take one of the values as _{m u (1,2, ···, m} u).

At this time, a line-of-sight movement transition probability matrix Φ = {φ _{(i, j)} } _, which is one of the line-of-sight position probability density parameters Θ _x (t), from the samples u _n (t−1) of the line-of-sight movement state variables. _{Based on (i, j)} , a sample u _n (t) of the current line-of-sight movement state variable is randomly generated. The line-of-sight movement transition probability matrix is expressed by a matrix of m _u rows and m _u columns, and expresses a probability of transition from the state j to the state i by an element φ _{(i, j) of} i rows and j columns. Therefore, Φ satisfies the properties shown in the following formula (58).

In other words, the following equation (59), which is a set of samples u _n (t) of the line-of-sight movement state variables generated as described above, is the current line-of-sight movement state variable set U (t).
It should be noted that the empirical probability distribution of the line-of-sight movement state variable sample u _n (t) included in the current line-of-sight movement state variable set U (t) is an approximation of the occurrence probability of the line-of-sight movement state variable.

As another embodiment, the line-of-sight movement state variable updating unit 141 can perform no processing. However, this is equivalent to m _u = 1, that is, there is only one line-of-sight movement state in the update process of the line-of-sight movement state random variable set U (t) in the line-of-sight movement state variable update unit 141 described above. .

The representative line-of-sight position update unit 142 includes the line-of-sight position probability density image X (t) output from the representative line-of-sight position set reconstruction unit 145 and the line-of-sight movement state input from the line-of-sight movement state variable update unit 141. Based on the variable set U (t) and the gaze position probability density parameter Θ _x (t), which is a parameter given in advance, the gaze movement controlled by the gaze movement state variable u (t) is considered, A representative line-of-sight position set V (t), which is a set of representative line-of-sight positions representing representative line-of-sight positions included in the line-of-sight position probability density image X (t) so far, is updated.
Further, the representative visual line position update unit 142 outputs the updated representative visual line position set V (t) to the representative visual line position weight coefficient calculation unit 143, the visual line position probability density image output unit 144, and the representative visual line position set reconstruction unit 145. To do.

  The method of updating the representative line-of-sight position set V (t) in the representative line-of-sight position update unit 142 is not particularly limited, but the method according to the present embodiment will be described.

  First, the following two models will be described as models in which the line-of-sight position x (t) is controlled by the line-of-sight movement state variable u (t).

(Model 1): Occurrence probability of the line-of-sight position x (t) at the current time (time t) when the line-of-sight position before the time point (time t-1) is given as the line-of-sight position x (t-1) Is given by the following equation (60) in a form depending on the current line-of-sight movement state variable u (t).

In the above equation (60), γ _xi and σ _xi (i = 0, 1,..., M _u −1) are constants constituting the line-of-sight position probability density parameter Θ _x (t), respectively, ) (Hereinafter referred to as “probability density Q (x; ￣x, γ, σ)”), the center is the following formula (62), the most frequent distance is γ, and the parameter corresponding to the standard deviation from the most frequent distance is The probability density function shown in the following formula (63) as σ is represented.

In the above equation (63), ‖x‖ is the norm of the vector x, Z _L is the following equation (64) for setting the integral value in the entire domain of probability density Q (x; ￣x, γ, σ) to 1. The normalization constant represented by

(Model 2): Occurrence probability of the gaze position x (t) at the present time (time t) when the gaze position before the time point (time t-1) is given as the gaze position x (t-1) A beta distribution is used. The beta distribution for the one-dimensional variable x is defined by the following formula (66) in which the domain of definition is the following formula (65).

  In the above equation (66), a and b each indicate a parameter that characterizes the beta distribution. B (a, b) is called a beta function, and represents a normalization constant represented by the following formula (67) for setting the integral value in the entire domain of the beta distribution to 1.

  In the present embodiment, the following equation (69), which is a beta distribution using the distance between the position x and a predetermined origin x as a variable of the normalization constant, and having a defined region as the following equation (68), Use.

  That is, the following formula (70) which is the above-mentioned beta distribution is given by the following formula (71).

  By using the normalized beta distribution as in the above formula (71), the current position when the line-of-sight position one time before (time t−1) is given as the line-of-sight position x (t−1) ( The occurrence probability of the line-of-sight position x (t) at time t) is given by the following equation (72), depending on the current line-of-sight movement state variable u (t).

In the above formula (72), a _xi and b _xi (i = 0, 1, m _u −1) represent constants constituting the line-of-sight position probability density parameter Θ _x (t), respectively.

  The representative line-of-sight position update unit 142 updates the representative line-of-sight position set V (t) by a method using any of the above-described models as described below.

  First, as a part of the output of the representative line-of-sight position set reconstruction unit 145, a representative line-of-sight position set V (t−1) one point before (time t−1) is given as in the following equation (73). It shall be.

In the above equation (73), N _x represents the number of elements of the representative visual line position set V (t), that is, the number of samples of the representative visual line position. In a general embodiment, the number of elements N _x of the representative line-of-sight position set V (t) is made the same as the number of elements N _u of the line-of-sight movement state variable set U (t).

Also, the representative line-of-sight position update unit 142 calculates, from one sample x _n (t−1) of the representative line-of-sight position one time before (time t−1), as shown in the following formula (74) using one of the models described above. A sample x _n (t) of the representative line-of-sight position at the present time (time t) is randomly generated using the probability density function shown.

  Note that, in the method of randomly generating a sample using the probability density function represented by the above formula (74), the probability density function used for sample generation is complicated. For this reason, it is difficult to generate a random sample by a direct method. However, for the random sample generation as described above, for example, a sample generation method based on the Markov chain Monte Carlo method can be used.

  Next, a detailed method of sample generation based on the Markov chain Monte Carlo method generally called the Metropolis-Hastings algorithm will be described.

  First, the representative line-of-sight position update unit 142 sets the sample of the representative line-of-sight position one time before (time t−1) as the following expression (75) that is the initial value of the temporary sample of the representative line-of-sight position: Give like.

  Next, the following equation (77), which is a two-dimensional vector, is generated using the origin-symmetric probability density function, and the following equation (78), which is the two-dimensional vector, is expressed by By adding to the following formula (79) that is a temporary sample, the following formula (80) that is a temporary sample of the representative line-of-sight position of the k-th step is generated as the following formula (81).

This origin-symmetric probability density function only needs to satisfy the symmetry with respect to the origin, for example, a two-dimensional normal distribution centered on the origin, and uniform within the range of each element ± δ _x centered on the origin. Distribution, etc. can be considered.

  Then, the ratio of the occurrence probability of the above formula (80), which is a sample of the temporary representative line-of-sight position in the k-th step, to the occurrence probability of the above-described formula (79), which is a temporary sample of the representative line-of-sight position in the k-1 step. The following formula (82) is calculated based on the following formula (83).

  Finally, the following formula (85), which is a uniform random number of the following formula (84), is generated. Only in the case of the following formula (86), the above formula (80), which is a temporary sample of the representative line-of-sight position in the k-th step. ) Is replaced with the above-described mathematical expression (79), which is a temporary sample of the representative line-of-sight position in the (k−1) -th step.

Thereafter, the above-described tentative sample predetermined number (K _x times) the generation step of repeating, the K _x formula is a sample of tentative step (87) to the following equation (88) to show such time t The representative line-of-sight position is a sample.

  As described above, a sample is generated based on the Markov chain Monte Carlo method. The following expression (89), which is a set of the generated samples, is the current representative gaze position set V (t). Also, the experience probability distribution of representative visual line position samples included in the current representative visual line position set V (t) is an approximation of the occurrence probability of the visual line position.

The representative gaze position weighting coefficient calculation unit 143 includes the probabilistic basic attention level image S (t) input from the probabilistic basic attention level image extraction unit 12 and the representative gaze position set V ( Based on t) and a gaze position probability density parameter Θ _x (t) that is a parameter given in advance, a representative gaze position weight coefficient that is a weight associated with each representative gaze position is calculated.
In addition, the representative gaze position weight coefficient calculating unit 143 converts the following gaze position probability density image output unit 144 and the representative gaze position set as the representative gaze position weight coefficient set which is a set of the calculated representative gaze position weight coefficients. The data is output to the reconstruction unit 145.

  The method of extracting the representative line-of-sight position weighting coefficient set W (t) in the representative line-of-sight position weighting coefficient calculation unit 143 is not particularly limited, but in this embodiment, the representative line-of-sight position weighting coefficient set based on the signal detection theory is used. A method for extracting W (t) will be described.

The representative line-of-sight position weighting coefficient w _n (t) associated with the representative line-of-sight position sample x _n (t) (n = 1, 2,..., N _x ) is expressed by the following expressions (91) and (92). Is calculated by In addition, the following formula (91) and the following formula (92) indicate that the actual value of the probabilistic basic attention level s (t, y) at the position x _n (t) is a certain position set D _x (x _n (t)). The probability of being equal to or higher than the actual value of the probabilistic basic attention degree s (t, y) at the position y other than is calculated.

Note that the notation of s = s (t, x _n (t)) is used only in the above formula (91) and the above formula (92). In the above formula (91) and the above formula (92), the following formula (93) represents the probability distribution function of the current probabilistic basic attention level s (t, y) at the position y, and the current distribution at the position y. Corresponding to the probability density p (s (t, x)), it is defined as the following formula (94).

Various methods can be considered for giving a certain position set D _x (x). For example, a set of arbitrary positions other than the position x, and the following formula (95) which is a basic attention degree other than the position x is locally maximum. And a set of positions y where the following expression (96), which is an expected value of the probabilistic basic attention degree s (t, y), other than the position x is locally maximized.

  In the method of extracting the representative gaze position weight coefficient set W (t) based on the signal detection theory described above, the representative gaze position set V (t) and the representative gaze position weight coefficient set W (t) are extracted by sampling. However, the representative line-of-sight position weighting coefficient set W (t) can also be extracted without using sampling. Hereinafter, a method for extracting the representative gaze position weighting coefficient set W (t) without using sampling will be described.

  In the method of extracting the representative gaze position weight coefficient set W (t) without using sampling, the representative gaze position update unit 142 updates the representative gaze position set V (t) and the representative gaze position weight coefficient calculation unit 143 represents the representative. The line-of-sight position weighting coefficient set W (t) is extracted at the same time.

First, similarly to the method of extracting the representative gaze position weighting coefficient set W (t) based on the signal detection theory described above, the probabilistic basic attention s (t, y) at the position x (t) is obtained by the following equation (97). Is calculated at each position in the input image at a position in the input image that is equal to or greater than the actual value of the probabilistic basic attention level s (t, y) at a position y other than a position set D _x (x (t)). .

Subsequently, the probability distribution calculated by the above equation (97) is modeled by a mixed Gaussian distribution using an EM algorithm as shown in the following equations (98) to (102). That is, the mixing ratio π _n (t) (n = 1, 2,..., M _y ) of the Gaussian distribution, which is each parameter of the mixed Gaussian distribution, the following equation (103) that is an average vector of each Gaussian distribution, and A model of the mixed Gaussian distribution is obtained by repeating the modeling step according to the following equations (98) to (102) for the covariance matrix S _n (t) until each parameter converges for k = 1, 2,. To derive. When deriving a model of the mixed Gaussian distribution, a random variable z expressing which Gaussian distribution the position x belongs to is introduced.

In the above formulas (98) to (102), α _n (n = 1, 2,..., M _y ) represents a constant determined in advance so as to satisfy the following formula (104).

Note that the smaller Gaussian distribution than constant mixing ratio of the Gaussian distribution [pi _{n (t)} is predetermined, and removed as contribution to the mixing ratio of the Gaussian distribution [pi _{n (t)} is small, remaining finally A mixed Gaussian distribution is formed by N _x Gaussian distributions. Then, the above equation (103) (n = 1, 2,..., N _x ), which is each average position of the mixed Gaussian distribution, is converted into the representative line-of-sight position v _n (t) (n = 1, 2,..., N _x ).
From this, in the method of extracting the representative gaze position weight coefficient set W (t) without using sampling, the number of elements N _x of the representative gaze position set V (t) is not given in advance, and depends on the input image. I can see that they are different.

On the other hand, the representative line-of-sight position weighting coefficient w _n (t) (n = 1, 2,..., N _x ) is the representative line-of-sight position set one time before (time t−1) (105) Based on the following formula (106) which is a representative gaze position weighting coefficient set one time before (time t−1) and the mixture ratio π _n (t) of the above mixed Gaussian distribution, the following formula (107) is obtained. To calculate.

In other words, a mixed Gaussian distribution composed of the representative line-of-sight position set V (t) and the representative line-of-sight position weighting coefficient w _n (t) before one time point is modeled by the above equations (98) to (102). The transition is made to the mixed Gaussian distribution in consideration of the probability density p (s (t, x)) regarding the line-of-sight movement.

  As described above, in the method of extracting the representative gaze position weight coefficient set W (t) without using sampling, the representative gaze position update unit 142 and the representative gaze position weight coefficient calculation unit 143 are a representative gaze position set. The following formula (108) and the above formula (90) which is a representative gaze position weighting coefficient set are extracted and output to the gaze position probability density image output unit 144.

The line-of-sight position probability density image output unit 144 includes a representative line-of-sight position set V (t) input from the representative line-of-sight position update unit 142 and a representative line-of-sight position weight coefficient set W ( Based on t), a representative gaze position probability density image H (t) is extracted.
The line-of-sight position probability density image output unit 144 outputs the extracted representative line-of-sight position probability density image H (t) to the representative line-of-sight position set reconstruction unit 145.

  The method for calculating the representative eye-gaze position probability density image H (t) by the eye-gaze position probability density image output unit 144 is not particularly limited, but the method according to the present embodiment will be described.

  The gaze position probability density image output unit 144 uses the representative gaze position set V (t) and the representative gaze position weight as the pixel value at the position x (t) of the representative gaze position probability density image H (t) at the current time (time t). Based on the set W (t), calculation is performed as in the following formula (109).

  In the above equation (109), f (•) is a predetermined function, and for example, a delta function represented by the following equation (110), a two-dimensional normal distribution represented by the following equation (111), and the like are conceivable.

The representative line-of-sight position set reconstruction unit 145 includes a representative line-of-sight position set V (t) input from the representative line-of-sight position update unit 142, a line-of-sight movement state variable set U (t) input from the line-of-sight movement state variable update unit 141, And the representative gaze position weight coefficient set W (t) input from the representative gaze position weight coefficient calculation unit 143, the representative gaze position set V (t) and the gaze movement state variable set U (t) Reconfiguration is performed according to the weight distribution indicated by the weight coefficient set W (t).
Further, the representative gaze position set reconstruction unit 145 reconstructs the representative gaze position weight coefficient set W (t).
In addition, the representative gaze position set reconstruction unit 145 applies the reconstructed representative gaze position set V ^* (t), the gaze movement state variable set U ^* (t), and the representative gaze position weight coefficient set W ^* (t). The line-of-sight position probability density image X (t) is output to the line-of-sight position probability density image output unit 5.
In addition, the representative gaze position set reconstruction unit 145 outputs the gaze position probability density image X (t) to the gaze movement state variable update unit 141 and the representative gaze position update unit 142.

  The method for reconstructing the representative line-of-sight position set V (t) and the line-of-sight movement state variable set U (t) in the representative line-of-sight position set reconstruction unit 145 is not particularly limited, but the method according to this embodiment will be described.

First, a cumulative sum c _n (t) of representative line-of-sight position weighting factors w _n (t) (n = 1, 2,..., N _x ) is calculated by the following equation (112). If necessary in calculating the cumulative sum _c n (t), representative gaze position weight coefficient _w n descending order representative line-of-sight position of the _{(t) v n (t)} , eye movement state variable u (t) and The representative line-of-sight position weighting coefficient w _n (t) is rearranged.

For the subsequent processing, it is determined that c ₀ (t) = 0.

Then, randomly determined a certain number kappa ₁ in the range of the following formula (113), and later, n = 2,3, · · ·, for _{N x,} defining a kappa _n as the following equation (114).

Then, for each of n = 1, 2,..., N _x , an integer n ^* that satisfies the condition of the following formula (115) is obtained.

  Then, the following expression (116), which is a new representative line-of-sight position, is defined as the following expression (117).

  Further, the following formula (118), which is a new line-of-sight movement state variable, is defined as the following formula (119).

Incidentally, the new representative sight position weight coefficient formula (120) are all 1 / _{N x.}

Note that the above-described reconstruction of the representative line-of-sight position set V (t) and the line-of-sight movement state variable set U (t) does not necessarily have to be performed at all times, for example, at certain time intervals. Or you can do nothing at all.
Further, for example, it can be performed only when the condition regarding the bias of the representative gaze position weighting coefficient represented by the following formula (121) is not satisfied.

In the above formula (121), N _eff is a constant determined in advance so as to satisfy the following formula (122).

  Further, the representative gaze position set reconstruction unit 145 reconstructs the representative gaze position set V (t), the gaze movement state variable set U (t), and the representative gaze position weight coefficient set W (t) described above. Based on the following formula (123) that is a new representative gaze position set reconstructed by the following formula, the following formula (124) that is a new gaze movement state variable set, and the following formula (125) that is a new representative gaze position weight coefficient set, The line-of-sight position probability density image X (t) reconstructed from the line-of-sight position probability density image H (t) input from the line-of-sight position probability density image output unit 144 is used as the output of the line-of-sight position probability density image extraction unit 14 Output to the probability density video output unit 5.

  In the case where the representative visual line position set reconstruction unit 145 does not reconstruct the representative visual line position set V (t), the visual line movement state variable set U (t), and the representative visual line position weight coefficient set W (t) at all. The visual line position probability density image H (t) input from the visual line position probability density image output unit 144 is used as the visual line position probability density image X (t) that is the output of the visual line position probability density image extraction unit 14. Output to the video output unit 5.

  The line-of-sight position probability density video output unit 5 sequentially estimates a basic attention level image extraction unit 11, a probabilistic basic attention level image extraction unit 12, and a stochastic basic attention level parameter from each time-series input image included in the input video. The line-of-sight position probability density image X (t) extracted by the processing of the unit 13 and the line-of-sight position probability density image extracting unit 14 is extracted and output.

As described above, according to the first embodiment, the input image, the probabilistic basic attention degree parameter Θ _s (t), and the sight position probability density parameter Θ _x (t), which are targets for eye gaze position estimation. On the basis of the above, the stochastic basic attention degree parameter Θ _s (t) can be sequentially updated when the line-of-sight position probability density image is output.

  Further, according to the first embodiment, the expected value and the standard deviation in the probabilistic basic attention level image extraction unit 12 can be independently updated at each position in the input image. As a result, the expected value and standard deviation update processing by the probabilistic basic attention level image extraction unit 12 is easily parallelized on a computer capable of parallel processing, such as a computer having multiple cores or a graphic processor unit (GPU). And the processing speed can be increased.

  Note that the attention level video extraction unit 1 includes a probabilistic basic attention level image extraction unit 12, a stochastic basic attention level parameter sequential estimation unit 13, a gaze position probability density image extraction unit 14, and a gaze position probability density video output unit 15. Instead of the above-described method, the attention degree video can also be extracted by the methods described in Non-Patent Documents 3 and 4 below. However, the probabilistic basic attention level image extraction unit 12, the probabilistic basic attention level parameter sequential estimation unit 13, the gaze position probability density image extraction unit 14, and the gaze position probability density video output unit 15 employ parallel processing as described above. Therefore, compared to the methods described in Non-Patent Documents 3 and 4, it is possible to calculate at high speed (extract attention level video at high speed).

(Non-Patent Document 3) Derek Pang, Akisato Kimura, Tatsuto Takeuchi, Junji Yamato and Kunio Kashino, “A stochastic model of selective visual attention with a dynamic Bayesian network,” Proc. International Conference on Multimedia and Expo (ICME2008), pp. 1073-1076, Hannover, Germany, June 2008.
(Non-Patent Document 4) Akisato Kimura, Derek Pang, Tatsuto Takeuchi, Junji Yamato and Kunio Kashino, “Dynamic Markov random fields for stochastic modeling of visual attention,” Proc. International Conference on Pattern Recognition (ICPR2008), Mo.BT8.35 , Tampa, Florida, USA, December 2008.

  The saliency area prior probability image extraction unit 2 extracts a saliency area prior probability image indicating the probability that each position of the input image, which is each frame constituting the input video, is a saliency area. Specifically, the saliency area prior probability image extraction unit 2 corresponds to a corresponding frame in the input video from the attention level image which is one frame of the attention level video and the saliency area image extracted by the saliency area image extraction unit 4. A saliency area prior probability image that displays the probability that each position of the input image is a saliency area is extracted. In other words, the saliency area prior probability image extraction unit 2 extracts the saliency area prior probability image indicating the probability that each position of one input image is a saliency area in the attention level video extracted by the attention level video extraction unit 1. Are extracted based on the attention level image which is an image corresponding to the input image and the saliency area image corresponding to the input image extracted by the saliency area image extraction unit 4. The saliency area prior probability image extraction unit 2 outputs the extracted saliency area prior probability image to the feature amount likelihood calculation unit 3 and the saliency area image extraction unit 4. The method by which the saliency area prior probability image extraction unit 2 extracts the saliency area prior probability image is not particularly limited. In this embodiment, the saliency area prior probability image generation unit 21 and the saliency area prior probability image update unit 22 perform extraction. How to do will be described.

  The saliency area prior probability image generation unit 21 receives the attention level image and generates a saliency area prior probability image only from the attention level image. The method for generating the saliency area prior probability image from the attention level image by the saliency area prior probability image generation unit 21 is not particularly limited. In the present embodiment, a method using a Gaussian mixture distribution model will be described.

The saliency area prior probability image generation unit 21 first calculates the attention level image at time t (that is, the basic attention level image ￣S (t) or the gaze position probability density image X (t)) from the center position to x _j ( t) M _s Gaussian distributions with covariance matrix ~ Σ _{s, j} (t) (j = 1,2, ..., M _s ) and mixing ratio ~ η _{s, j} (t) Model parameters (ie, M _s center positions, covariance matrix, mixture ratio) are estimated from the attention degree image (b). Specific examples of the estimation method are the following two.

(Estimation method 1)
Derived using EM algorithm. At this time, it should be noted that each sample given to the EM algorithm corresponds to a specific position x of the attention degree image (b) and has a weight equal to the pixel value at the position x. The estimation of the Gaussian mixture distribution parameter by the EM algorithm is performed by repeating the following formula (126) to the following formula (129) with k = 1, 2,..., And the procedure is terminated when each parameter converges. Fix the parameters.

  Here, g (x; ˜x, Σ) is a multidimensional normal distribution, and is defined by the following formula (130) when the number of dimensions is D.

Further, since the pixel value at the position x of the line-of-sight position probability density image X (t) is used as a weight when the position x is regarded as a sample of the EM algorithm, it is expressed here as w _x (t). ing.

(Estimation method 2)
M _s maximum values of the pixel values of the attention level image are detected, and a position where the maximum value is obtained is determined as a center position to x _j (t) (j = 1, 2,..., M _s ). Let the pixel value of the attention level image be a mixture ratio ~ η _{s, j} (t). The covariance matrix ~ Σ _{s, j} (t) is obtained in the same manner as in the first estimation method or a predetermined value is used.

As described above, the saliency area prior probability image generation unit 21 estimates the model parameter of the Gaussian mixture distribution from the attention degree image, and generates the saliency area prior probability image. Specifically, the saliency area prior probability image generation unit 21 normalizes the mixture ratio, which is one of the model parameters, so that its maximum value is equal to 1, and then calculates the probability of the Gaussian mixture distribution at each position. The pixel value ξ ₁ (x, t) of the saliency area prior probability image _{１ 1} (t) at the position is calculated (the following formula (131)).

In the above-described embodiment, the saliency area prior probability image is generated by the method using the Gaussian mixture distribution at all positions. However, the second area described later in consideration of the fact that the saliency area tends to exist at the center position of the image. Embodiments are possible. That is, in this case, after generating the saliency area prior probability image in the same manner as in the first embodiment, the pixel values of the constant areas at the left and right ends or the upper and lower left and right ends of the image are forcibly set to zero. This method means eliminating the possibility of a significant area at the edge of the image. Alternatively, after the saliency area prior probability image is generated in the same manner as in the first embodiment, the weight proportional to the distance from the center position of the image is multiplied by the saliency area prior probability image, and the output is newly generated as the saliency area prior probability. An embodiment is also conceivable. As described above, the saliency area prior probability image generation unit 21 generates and outputs the saliency area prior probability image ￣Ξ ₁ (t).

  The saliency area prior probability image update unit 22 updates the saliency area prior probability image generated by the saliency area prior probability image generation unit 21 using the saliency area image. That is, the saliency area prior probability image update unit 22 inputs the saliency area prior probability image generated by the saliency area prior probability image generation unit 21 and the saliency area image extracted by the saliency area image extraction unit 4, and The saliency prior probability image is updated using the region image. The method by which the saliency area prior probability image update unit 22 updates the saliency area prior probability image is not particularly limited, but in the present embodiment, a method using the principle of the Kalman filter will be described.

The pixel value ξ ₁ (x, t) (probability variable) at the position x of the saliency area prior probability image Ξ ₁ (t) (probability variable) at the current time (time t) is the current saliency area prior probability image ￣Ξ1 before update. The pixel value ￣ξ1 (x, t) at the position x in (t) and the pixel value a (x, t− at the position x in the saliency area image A (t−1) one time before (time t−1). It is assumed that the following formulas (132) and (133) are satisfied for 1).

Here, θ = (σ ₁ , σ ₂ ) is a parameter given in advance. Further, f (•) is a function for converting the pixel value of the saliency area image into a real value, and is set as in the following formulas (134) and (135), for example.

At this time, the saliency area prior probability image update unit 22 uses the Kalman filter principle to calculate the pixel value ξ ₁ (x, t) at the position x of the current saliency area prior probability image _{１ 1} (t) as follows. It is updated by the equations (136) and (137).

  In the above embodiment, the saliency prior probability at each time is held, but this distribution may not be used when updating at the next time. That is, the following formula (138) may be added to the above formulas (136) and (137).

As described above, the saliency area prior probability image update unit 22 updates the saliency area prior probability image ￣Ξ ₁ (t) to Ξ ₁ (t), and outputs the updated saliency area prior probability image Ξ ₁ (t). .

  As described above, the saliency area prior probability image extraction unit 2 extracts (generates and updates) and outputs the saliency area prior probability image Ξ1 (t) as shown in FIG.

  The feature amount likelihood calculating unit 3 calculates a feature amount likelihood indicating the likelihood of the image feature amount included in each of the saliency area and the area outside the saliency area of the input image. Specifically, the feature amount likelihood calculating unit 3 determines the feature amount likelihood as the input image, the attention degree image, the saliency area prior probability image extracted by the saliency area prior probability image extracting unit 2, and the saliency area image extracting unit. 4 is calculated based on at least one of the salient region image extracted by 4 and the feature amount likelihood calculated until the previous time. For example, the feature amount likelihood calculation unit 3 calculates the feature amount likelihood from the input image, the saliency area prior probability image, the saliency area image, and the feature amount likelihood calculated until the previous time. The feature amount likelihood calculating unit 3 outputs the calculated feature amount likelihood to the saliency area image extracting unit 4. The method by which the feature amount likelihood calculation unit 3 calculates the collection likelihood is not particularly limited, but in the present embodiment, the saliency region feature amount likelihood calculation unit 31, the non-salience region feature amount likelihood calculation unit 32, and the feature A method of calculation by the quantity likelihood output unit 33 will be described.

  The saliency area feature amount likelihood calculation unit 31 calculates the saliency area feature amount likelihood indicating the likelihood of the image feature amount included in the saliency area by the input image, the saliency area prior probability image, the saliency area image, and the previous time. Calculation is made based on at least one of the saliency feature amount likelihoods. The method by which the saliency area feature amount likelihood calculating unit 31 calculates the saliency area feature amount likelihood is not particularly limited, but in the present embodiment, the saliency area feature amount likelihood generating unit 311 and the saliency area feature amount likelihood updating unit are included. 312 will be described.

  The saliency area feature amount likelihood generation unit 311 newly generates (calculates) and outputs a saliency area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image. The method for generating the saliency feature amount likelihood by the saliency feature amount likelihood generation unit 311 is not particularly limited, but in the present embodiment, a method using a Gaussian mixture distribution model will be described.

The saliency area feature amount likelihood generation unit 311 first calculates, at time t, the saliency area feature quantity probabilities, which are probability distributions of the characteristic quantities peculiar to the saliency area, from the mean to c _j (t) and the covariance matrix to Σ, respectively. _{f, j} (t) (j = 1, 2,..., M _f ), and a mixture ratio of ˜η _{f, j} (t) is composed of a mixture of M _f Gaussian distributions. Assuming these model parameters are estimated from the pixel values of the input image weighted with the pixel values of the saliency prior probability image. For example, an EM algorithm is used for estimating the model parameters. Specifically, the following formula (139) to the following formula (142) are repeated by k = 1, 2,..., And when each parameter converges, the procedure is terminated and the parameters are fixed.

  Here, the pixel value at the position x of the input image is expressed as c (x, t) as a three-dimensional RGB vector. As described above, the saliency area feature amount likelihood generation unit 311 calculates the saliency area feature amount likelihood from the model parameters of the estimated Gaussian mixture distribution. Specifically, it is calculated by the following equation (143) with a Gaussian mixture distribution characterized by the estimated model parameters as the likelihood.

As described above, the saliency area feature amount likelihood generation unit 311 generates (calculates) and outputs the saliency area feature amount likelihood ￣ψ ₁ (c, t).

  The saliency area feature amount likelihood update unit 312 updates the saliency area feature amount likelihood generated by the saliency area feature amount likelihood generation unit 311. Specifically, the saliency area feature amount likelihood updating unit 312 is based on at least one of the input image, the saliency area image, and the updated saliency area feature amount likelihood updated until the previous time, and the saliency area feature amount. The saliency area feature amount likelihood generated by the likelihood generation unit 311 is updated. The method by which the saliency feature amount likelihood update unit 312 updates the saliency feature amount likelihood (e1) is not particularly limited, but in the present embodiment, the following two methods will be described.

(Update method 1)
It is updated by mixing two types of saliency area feature likelihoods. Specifically, the saliency area feature amount likelihood ψ ₁ (c, t) at the current time (time t) to be obtained is determined as the saliency area feature amount likelihood before update output from the saliency area feature amount likelihood generation unit 311. ψ1 (c, t) and the saliency area feature likelihood ￣ψ ₁ (c, t-1) one point before (time t-1) are mixed at a predetermined mixing ratio λ _c Calculated according to equation (144).

(Update method 2)
Based on the saliency area image A (t−1) before one time point (time t−1), the saliency area feature amount likelihood ψ ₁ (c, t−1) before one time point is updated and the first A method similar to that of the embodiment is performed. Specifically, an area (a saliency area A _obj (t) indicated by the following formula (145)) that is regarded as a saliency area in the saliency area image A (t−1) before one point in time is extracted, and the saliency area A _The saliency feature amount likelihood ψ ₁ (c, t−1) is relearned from the pixel value in the input image at _obj (t) by the same method as the method shown in the saliency feature amount likelihood generation unit 311. To do. However, in the present embodiment, the saliency area image is used as the weight instead of the saliency area prior probability image. After re-learning the saliency feature amount likelihood ψ1 (c, t−1) one point before, the current saliency feature feature likelihood ψ ₁ (c, t) by the same method as in the first embodiment. Is generated.

As described above, the saliency area feature quantity likelihood update unit 312 updates the saliency area feature quantity likelihood ￣ψ ₁ (c, t) to ψ ₁ (c, t) and outputs it. As described above, the saliency area feature amount likelihood calculating unit 31 calculates and outputs the saliency area feature amount likelihood ψ ₁ (c, t).

  The non-significant region feature amount likelihood calculating unit 32 converts the non-significant region feature amount likelihood indicating the likelihood of the image feature amount included in the region outside the saliency region into an input image, a saliency region prior probability image, a saliency region image, and The calculation is performed based on at least one of the non-significant region feature amount likelihoods calculated so far. The method by which the non-significant region feature amount likelihood calculating unit 32 calculates the non-significant region feature amount likelihood is not particularly limited, but in the present embodiment, the non-significant region feature amount likelihood generating unit 321 and the non-significant region feature amount likelihood are calculated. A method of calculation by the likelihood update unit 322 will be described.

The non-salience area feature amount likelihood generation unit 321 newly generates (calculates) and outputs a non-salience area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image. The method by which the non-salience feature amount likelihood generation unit 321 generates the non-salience feature amount likelihood is not particularly limited, but in the present embodiment, a method using a Gaussian mixture distribution model will be described. The method is almost the same as the method of salient region feature amount likelihood generator 311 described above, instead of the salient region prior probability image, each pixel value of the notable region prior probability image Ξ _{1 (t)} ξ _{1 (} Assume that a non-salience area prior probability image Ξ ₂ (t), which is an image generated by converting x, t) according to a certain rule, is used. For example, the following two methods can be considered as the conversion rule.

(Method 1)
The pixel value ξ ₂ (x, t) at the position x of the non-salience area prior probability image (f) is converted into 1−ξ ₁ (x, t).
(Method 2)
Only in the position x where ξ ₁ (x, t) = 0, the pixel value of the non-salience area prior probability image (f) at that position is set to 1. For other positions, the pixel value is 0.

As described above, the non-salience area feature amount likelihood generation unit 321 generates (calculates) and outputs the non-salience area feature amount likelihood ￣ψ ₂ (c, t).

The non-significant region feature quantity likelihood update unit 322 updates the non-significant region feature amount likelihood generated by the non-significant region feature amount likelihood generation unit 321. Specifically, the non-salience area feature amount likelihood update unit 322 performs non-remarkable based on at least one of the input image, the non-salience area image, and the updated non-salience area feature amount likelihood updated up to the previous time. The non-salience area feature amount likelihood generated by the saliency area feature amount likelihood generation unit 321 is updated. The non-significant area image is an image relating to an area outside the saliency area extracted by the saliency area prior probability image extraction unit 2. The method by which the non-salience area feature amount likelihood update unit 322 updates the non-salience area feature amount likelihood is the same as the method of the saliency area feature amount likelihood update unit 312. However, instead of the saliency area prior probability image, the non-salience area prior probability image is replaced with the saliency area feature amount likelihood, and the non-salience area feature amount likelihood is replaced with the saliency area. The saliency area A _bkg (t)) indicated by is used.

As described above, the non-significant region feature value likelihood update unit 322 updates the non-significant region feature value likelihood ψ ₂ (c, t), and outputs the updated non-salience region feature value likelihood. As described above, the non-salience area feature amount likelihood calculating unit 32 extracts and outputs the non-salience area feature amount likelihood ψ ₂ (c, t).

  The feature quantity likelihood output unit 33 adds the saliency area feature quantity likelihood and the non-salience area feature quantity likelihood and outputs the result as a feature quantity likelihood.

  The saliency area image extraction unit 4 extracts a saliency area image indicating the saliency area of the input image from the input image, the saliency area prior probability image, and the feature amount likelihood. The saliency area image extraction unit 4 outputs the extracted saliency area image to the saliency area prior probability image extraction part 2, the feature amount likelihood calculation part 3, and the saliency area video generation part 5. The method of extracting the saliency area image by the saliency area image extracting unit 4 is not particularly limited, but in this embodiment, a method using a graph cut based on the method described in Non-Patent Document 1 will be described. In this method, the saliency area extraction graph generation unit 41 and the saliency area extraction graph division unit 42 extract the saliency area image.

  The saliency area extraction graph generation unit 41 inputs the input image, the saliency area prior probability image, and the feature amount likelihood, and generates and outputs a saliency area extraction graph that is a graph for extracting the saliency area image.

Specifically, the saliency area extraction graph generation unit 41 first sets vertices corresponding to the respective positions x∈I of the input image as vertices of the saliency area extraction graph G (t) at time t, and saliency areas / non-salience. Two types of vertices corresponding to the label of the area are prepared. That is, the total number of vertices is +2. Hereinafter, for simplicity, the vertex corresponding to each position x is represented as v _x , the vertex corresponding to the label of the saliency area is SOURCES, and the vertex corresponding to the label of the non-salience area is SINKT, respectively. Also, as the sides of the saliency area extraction graph, n-links, which are directed edges between vertices corresponding to neighboring positions, and directed edges, which are arranged from SOURCE to each vertex and from each vertex to SINK, respectively. Two types of t-link sides are prepared. As the neighborhood, for example, 4 neighborhoods in the upper, lower, left, and right directions, or 8 neighborhoods including an oblique direction are considered. In this way, the saliency area extraction graph is configured as a directed graph, for example, in the form shown in FIG.

Next, the saliency area extraction graph generation unit 41 gives a cost to each side of the saliency area extraction graph. The cost of t-link is calculated from the significant area prior probability image and the feature amount likelihood. Specifically, the cost C (S, v _x ; t) of t-link from SOURCES to vertex v _x is the sum of the corresponding non-salience area prior probability and non-salience area feature likelihood, and from vertex v _x The cost C (T, v _x ; t) of t-link to SINKT is given by the following equations (147) and (148) using the corresponding saliency area prior probability and saliency area feature likelihood.

On the other hand, the cost of n-link is calculated based on the similarity of luminance values between adjacent pixels. Specifically, the cost C (v _x , v _y ) of the n-link between the two points v _x and v _y is given by the following formula (149).

  The saliency area extraction graph dividing unit 42 receives the saliency area extraction graph, divides the saliency area extraction graph, and generates and outputs a saliency area image.

  Specifically, the saliency area extraction graph dividing unit 42 first considers dividing the vertices included in the saliency area extraction graph into a subset including SOURCE and a subset including SINK. At this time, the division is performed so that the cost of the side extending from the subset of the vertexes on the SOURCE side to the subset of the vertexes on the SINK side is minimized. Note that it does not take into account the cost in the opposite direction, ie, the edge spanning from a subset of vertices on the SINK side to a subset of vertices on the SOURCE side. Such a problem is called a graph minimum cut problem, and is known to be equivalent to a graph maximum flow problem. As a method for solving this maximum flow problem, in addition to Non-Patent Document 1, “Ford-Fulkerson algorithm” described in Non-Patent Document 5 below, “Goldberg-Tarjan algorithm” described in Non-Patent Document 6 below, etc. are generally widely known. It has been.

(Non-Patent Document 5) LRFord, DRFulkerson: “Maximal flow through a network,” Canadial Journal of Mathematics, Vol. 8, pp. 399-404, 1956.
(Non-Patent Document 6) AVGoldberg, REtarjan: “A new approach to the maximum-flow problem,” Journal of the ACM, Vol. 35, pp. 921-940, 1988.

As a result of dividing the salient region extraction graph by the above method, the pixel position corresponding to the vertex belonging to the subgraph including SOURCE is set to the salient region A _obj (t), and the pixel corresponding to the vertex belonging to the subgraph including SINK The position is made to belong to the non-salience area A _bkg (t). As shown in the following formula (150), the saliency area image is an image in which the pixel value of the position belonging to the saliency area is 1 and the pixel value of the position belonging to the non-salience area is 0.

  As described above, the saliency area extraction graph dividing unit 42 extracts and outputs the saliency area image A (x, t). That is, the saliency area image extraction unit 4 extracts the saliency area image A (x, t) and outputs the saliency area image.

  The saliency area video generation unit 5 is obtained by executing the attention level video extraction unit 1, the saliency area prior probability image extraction unit 2, the feature amount likelihood calculation unit 3, and the saliency area image extraction unit 4 for each input image. A saliency area image is generated from each saliency area image. In other words, the saliency area video generation unit 5 applies the attention level video extraction unit 1, the saliency area prior probability image extraction unit 2, the feature amount likelihood calculation unit 3, and the saliency area image extraction unit 4 in order to each input image. A set of salient area images extracted by repeated execution is input and a salient area video composed of salient area images in the set is generated (a salient area video is generated by arranging the salient area images in the set in time series) To do). The saliency area image generation unit 5 outputs the generated saliency area image to the outside.

(Second Embodiment)
Hereinafter, a saliency image generating apparatus 1100 according to a second embodiment of the present invention will be described with reference to the drawings. The saliency area image generation device 1100 is configured from each saliency area frame (each saliency area image) obtained by acquiring an input image from outside and extracting each saliency area from each input frame (each input image) constituting the input image. Generated saliency area video and output it to the outside.

  As shown in FIG. 7, the saliency area image generation device 1100 includes an attention degree image extraction unit 1, a saliency area prior probability image extraction unit 2, a feature amount likelihood calculation unit 3, a saliency area image extraction unit 4, and a saliency area image generation unit. Unit 5, smoothed image group generation unit 6, and saliency area image determination unit 7. The attention level video extraction unit 1, the saliency area image extraction unit 4 and the saliency area image generation unit 5 are the same as those in the first embodiment, and thus the description thereof is omitted.

  The smoothed image group generation unit 6 generates a smoothed image group composed of a plurality of smoothed images obtained by smoothing the input image with different resolutions. That is, the smoothed image group generation unit 6 inputs an input image and generates a smoothed image group obtained by smoothing the input image with different resolutions. The smoothed image group generation unit 6 outputs the generated smoothed image group to the feature amount likelihood calculation unit 3, the saliency area image extraction unit 4, and the saliency area image determination unit 7. A method for generating the smoothed image group by the smoothed image group generation unit 6 is not particularly limited. In the present embodiment, a method for repeating smoothing and reduction on the input image will be described.

The smoothed image group generation unit 6 uses the initial value H ₀ (t) of the smoothed image at time t, and when the smoothed image H _k−1 (t) is given for a certain integer k, the input image is predetermined. Is smoothed using a Gaussian smoothing filter having a standard deviation parameter σ _g of. Smoothed image group generating section 6, an image obtained by smoothing with a Gaussian smoothing filter, reduced with a predetermined magnification a _g which satisfies the following expression (151), generating a smoothed image H _{k (t)} To do.

Since the integer k corresponds to the degree of smoothing in the smoothed image, the integer k is hereinafter referred to as a smoothing coefficient. A smoothed image group is formed by repeating the above process at k = 1, 2,..., N _g −2 (the following formula (152)).

At this time, if σ _g = 0 and a _g = 1 in particular, all the smoothed images are the same as the input image. As described above, the smoothed image group generation unit 6 extracts and outputs a smoothed image group.

  The saliency area prior probability image extraction unit 2 displays the probability that each position of the input image that is a corresponding frame in the input video is a saliency area from the attention level image and the saliency area image that are one frame of the attention level video. The saliency prior probability image to be extracted is extracted. The method by which the saliency area prior probability image extraction unit 2 extracts the saliency area prior probability image is not particularly limited, but in this embodiment, the saliency area prior probability image generation unit 21 and the saliency area prior probability image update unit 22 A method of extracting will be described.

  Since the saliency area prior probability image generation unit 21 is the same as that of the first embodiment, the description thereof is omitted. The saliency area prior probability image update unit 22 is substantially the same as that in the first embodiment. However, the following points are different from the first embodiment.

1. When this process is executed for the first time at a certain time t, that is, in the subsequent feature amount likelihood calculation unit 3 and saliency area image extraction unit 4, a smoothed image with the maximum smoothing coefficient (the following equation (153)) is obtained. When used, the saliency prior probability image (d) is updated by the same method as in the first embodiment.

2. When this process is executed again at a certain time t, that is, in the subsequent feature amount likelihood calculation unit 3 and the saliency region image extraction unit 4, the smoothed image H _k (t) (k = _ng- 2, _ng- 3,..., 0), the same processing as in the first embodiment is performed after making the following changes.
(1) σ ₁ , which is one of the parameters of the update formula described in the first embodiment, and the variance of the saliency prior probability (timed t−1) before one time point (time (t−1)) (the following formula (154)) are forced Replace with 0.

(2) A saliency area image generated using a smoothed image H _{k + 1} (t) having one larger smoothing coefficient instead of the saliency area image A (t−1) one point before (time t−1). A (t; k + 1) is used.
(3) Only the average ξ ₁ (x, t) is updated by the same method as in the first embodiment without updating the variance of the saliency prior probability (the following formula (155)).

3. In order to clarify that the smoothed image H _k (t) having the smoothing coefficient k is used as an input, the saliency area prior probability image as an output is denoted as Ξ ₁ (t; k).

The feature amount likelihood calculating unit 3 is substantially the same as that in the first embodiment. However, the following points are different from the first embodiment.
1. Instead of the input image, one of the smoothed images H _k (t) (k = _ng− 1, _ng− 2,..., 0) may be used. At this time, when this process is executed j (j = 1, 2,..., N _g ) times at time t, a smoothed image (equation 156 below) with a smoothing coefficient k = _ng− j is used. It is done. This means that the smoothed image having the largest smoothing coefficient is used in order.

2. When first run the process at a certain time t, that is, when the smoothing factor n _g -1 smoothed image H _{ng-1 (t)} is used as an input, except the first item first embodiment It is the same as the form.
3. When this processing is executed again at a certain time t, that is, the smoothed image H _k (t) of the smoothing coefficient k (k = _ng- 2, _ng- 1, ..., 0) is used as an input. If so, use:
(1) Instead of the saliency area image A (t−1) one time before (time t−1), a smoothed image H _{k + 1} (t) having a larger smoothing coefficient at the current time (time t) is used. The generated saliency area image A (t; k + 1) is used.
(2) The saliency generated by using the smoothed image H _{k + 1} (t) having one larger smoothing coefficient at the current time instead of the saliency area feature likelihood ψ ₁ (c, t−1) one time before The region feature amount likelihood ψ ₁ (c, t; k + 1) is used.
(3) Instead of the non-significant region feature amount likelihood ψ ₂ (c, t−1) one point before, it is generated using a smoothed image H _{k + 1} (t) having one larger smoothing coefficient at the present time. The non-salience area feature likelihood ψ ₂ (c, t; k + 1) is used.
4). In order to clarify that the smoothed image H _k (t) having the smoothing coefficient k is used as an input, the saliency area feature quantity likelihood as an output is ψ ₁ (t; k), and the non-salience area feature quantity likelihood. Is denoted as ψ ₂ (t; k).

The saliency area image determination unit 7 executes the processes of the saliency area prior probability image extraction unit 2, the feature amount likelihood calculation unit 3, and the saliency area image extraction unit 4 for the leveled image group, and the saliency area image of the input image Confirm. That is, the saliency area image determination unit 7 applies the saliency area prior probability image extraction unit 2, the feature amount likelihood calculation unit 3, and the saliency area image extraction unit 4 to the smoothed image H _k (t) with the smoothing coefficient k. When the extracted saliency area image A (t; k) is input and there is no change from the saliency area image A (t; k + 1) extracted in the previous step, the current time (time The final saliency image for the input image of t) is determined, and this saliency image A (t) = A (t; k) is output. If there is a change, k is decreased by one. Then, the process returns to the saliency area prior probability image extraction unit 2 again.

FIG. 8 shows a saliency area image generation result (salience area extraction result) by the saliency area image generation apparatus 1000. The input video is 640 × 480 pixels, 30 to 90 seconds, and the numerical values of the parameters are σ ₁ = 0.0.31, σ ₂ = 0.037, M _f = 5, λ _c = 0.25, λ _i = 100, σ = 0, 1. In FIG. 8, the first row and the third row are the input video, the second row is the extraction result of the salient region corresponding to the first row, and the fourth row is the extraction result of the salient region corresponding to the third row. As shown in FIG. 8, according to the saliency area image generation device 1000, the saliency area is appropriately extracted, and the boundary is almost perfect.

  FIG. 9 shows a method using the saliency area video generation apparatus 1000 and a saliency area prior probability image update unit 22, a saliency area feature amount likelihood update part 312, and a non-salience area feature amount likelihood update part 322 from the saliency area image generation apparatus 1000. This is a comparison with a method in which the saliency area prior probability image and the feature amount likelihood are not sequentially updated. As shown in FIG. 9, in the method in which the saliency area prior probability image and the feature amount likelihood are not sequentially updated, the division result varies greatly depending on the frame, but the method using the saliency area image generation device 1000 can stably divide the image. .

  As described above, in the present invention, the above-described video salient region extraction method is realized mainly by the following two points.

(1) Image saliency calculation based on a model simulating a human visual mechanism by the attention level video extraction unit 1, and video by the saliency area prior probability image generation unit 21 and the saliency feature amount likelihood calculation unit 31 Generation of prior information on saliency areas and non-salience areas based on saliency (2) Sequential generation of prior information on saliency areas and non-salience areas by the saliency area prior probability image update unit 22 and non-salience area feature amount likelihood calculation unit 32 Even when no prior information regarding the updated object area / background area is given, the area can be divided. Therefore, even when there is no prior knowledge about the object region and the background region, the object region and the background region can be divided with high accuracy and the region of interest (object region) can be extracted.

  As a result, it is possible to divide the region even when no prior information regarding the object region / background region is given. Therefore, even if there is no prior knowledge about the object region / background region, the object region and the background region can be divided with high accuracy and the region of interest (object region) can be extracted.

  A program for executing each process of the saliency image generating apparatuses 1000 and 1100 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. The above-described various processes related to the saliency image generation apparatuses 1000 and 1100 may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 1 Attention degree video extraction part 2 Saliency area prior probability image extraction part 3 Feature quantity likelihood calculation part 4 Saliency area image extraction part 5 Saliency area image generation part 6 Smoothed image group generation part 7 Saliency area image determination part 11 Basic attention degree Image extraction unit 12 Probabilistic basic attention level image extraction unit 13 Probabilistic basic attention level parameter sequential estimation unit 14 Gaze position probability density image extraction unit 15 Gaze position probability density video output unit 21 Saliency area prior probability image generation unit 22 Saliency area Prior probability image update unit 31 Saliency region feature amount likelihood calculation unit 32 Non-salience region feature amount likelihood calculation unit 33 Feature amount likelihood output unit 41 Saliency region extraction graph generation unit 42 Salience region extraction graph division unit 141 Gaze movement state variable Update unit 142 Representative eye gaze position update unit 143 Representative eye gaze position weight coefficient calculation unit 144 Eye gaze position probability density image output unit 14 Representative line-of-sight position set reconstruction unit 311 Striking region feature amount likelihood generating unit 312 Striking region feature amount likelihood updating unit 321 Non-salience region feature amount likelihood generating unit 322 Non-salience region feature amount likelihood updating unit 1000, 1100 Striking region Video generation device

Claims (14)

Attention level video extraction process for extracting attention level video indicating the level of attention that is the degree to which human attention is likely to be drawn from the input video,
A saliency area prior probability image extraction process for extracting a saliency area prior probability image indicating the probability that each position of the input image that is each frame constituting the input video is a saliency area; and
A feature amount likelihood calculating process for calculating a feature amount likelihood indicating the likelihood of the image feature amount included in each of the saliency area and the non-salience area of the input image;
A saliency area image extraction process for extracting a saliency area image indicating a saliency area of the input image from the input image, the saliency area prior probability image, and the feature amount likelihood;
From each saliency area image obtained by executing the attention level video extraction process, the saliency area prior probability image extraction process, the feature amount likelihood calculation process, and the saliency area image extraction process for each input image. A saliency image generation process for generating a saliency image,
The saliency area prior probability image extraction process includes:
The degree of attention is an image corresponding to the input image in the attention level video extracted by the attention level video extraction process, wherein the remarkable area prior probability image indicating the probability that each position of the one input image is a remarkable area Extracting based on the image and the salient region image;
The feature amount likelihood calculation process includes:
The feature quantity likelihood is calculated based on at least one of the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the feature quantity likelihood calculated up to the previous time. A saliency area image generation method. The attention level video extraction process includes:
A basic attention degree image extraction process for calculating a basic attention degree image, which is an image displaying a spatial region having a remarkable characteristic in the input image based on the input image;
A stochastic basic attention image that is an image displaying prominence at each position of the current input image using a probabilistic expression, a basic attention image calculated by the basic attention image extraction process, and a previous The probabilistic basic attention degree image calculated by the probabilistic basic attention degree image extraction process from the input image and the first basic parameter that is sequentially updated and used for eye-gaze position estimation is the probabilistic basic attention degree parameter. A stochastic basic attention level image extraction process calculated based on
A gaze position probability density image that is a frame of the gaze position probability density image in the current input image is obtained from the probabilistic basic attention image calculated by the probabilistic basic attention image extraction process and the previous input image. A line-of-sight position calculated based on the line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction process and a line-of-sight position probability density parameter that is sequentially updated and used for line-of-sight position estimation Probability density image extraction process;
A time series calculated by sequentially repeating the basic attention level image extraction process, the probabilistic basic attention level image extraction process, and the gaze position probability density image extraction process for each of the input images. A line-of-sight position probability density image output process for outputting the line-of-sight position probability density image as the line-of-sight position probability density image;
The line-of-sight position probability density image extraction process includes:
A line-of-sight movement state variable, which is a random variable that controls the amount of line-of-sight movement, is calculated from the line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction process from the previous input image and the previous input image. Gaze movement that is updated based on the gaze movement state variable calculated by the gaze movement state variable update process and the gaze position probability density parameter and outputs a gaze movement state variable set that is a set of the gaze movement state variables State variable update process;
A representative gaze position set, which is a set of representative gaze positions indicating representative gaze positions considering gaze movement, a stochastic basic attention image calculated by the stochastic basic attention image extraction process, and the previous input A representative gaze position update process that is updated based on the representative gaze position set updated from the image by the representative gaze position update process, the gaze movement state variable set, and the gaze position probability density parameter;
A representative gaze position weight coefficient set, which is a set of representative gaze position weight coefficients composed of weights associated with each of the representative gaze positions, and a probabilistic basic attention image calculated by the probabilistic basic attention image extraction process, A representative line-of-sight calculated based on the representative line-of-sight position set updated by the representative line-of-sight position update process, the line-of-sight movement state variable set output from the line-of-sight movement state variable update process, and the line-of-sight position probability density parameter Position weighting factor calculation process,
Gaze position probability for calculating the gaze position probability density image based on the representative gaze position set updated by the representative gaze position update process and the representative gaze position weight coefficient set calculated by the representative gaze position weight coefficient calculation process The saliency area image generation according to claim 1, further comprising: calculating a gaze position probability density image including the representative gaze position set and the representative gaze position weighting coefficient set. Method. The saliency area prior probability image extraction process includes:
A saliency area prior probability image generation process for generating the saliency area prior probability image using only the attention level image; and
2. The saliency area prior probability image update process in which the saliency area prior probability image generated by the saliency area prior probability image generation process is updated using the saliency area image. Item 3. The saliency image generation method according to any one of Items 2 to 3. The feature amount likelihood calculation process includes:
The saliency area feature amount likelihood indicating the likelihood of the image feature amount included in the saliency area is calculated based on the input image, the saliency area prior probability image, the saliency area image, and the saliency area feature amount likelihood calculated up to the previous time. A saliency area feature likelihood calculation process to be calculated based on at least one of them,
The non-salience area feature amount likelihood indicating the likelihood of the image feature amount included in the area outside the saliency area is calculated as the input image, the saliency area prior probability image, the saliency area image, and the non-salience area calculated until the previous time. A non-significant region feature amount likelihood calculation process based on at least one of the feature amount likelihoods;
4. The feature amount likelihood output process of adding the saliency region feature amount likelihood and the non-salience region feature amount likelihood and outputting the feature amount likelihood as a feature amount likelihood. The saliency area | region video generation method of any one of these. The saliency area feature likelihood calculation process includes:
Based on the input image, the saliency area prior probability image, and the saliency area image, a saliency area feature amount likelihood generation process for generating the saliency area feature amount likelihood;
A saliency area feature amount likelihood update process for updating the saliency area feature amount likelihood generated by the saliency area feature amount likelihood generation process,
The non-significant region feature amount likelihood calculation process includes:
Based on the input image, the saliency area prior probability image, and the saliency area image, a non-salience area feature amount likelihood generation process for generating the non-salience area feature amount likelihood;
A non-salience area feature amount likelihood update process for updating the non-salience area feature amount likelihood generated by the non-salience area feature amount likelihood generation process,
The saliency area feature likelihood update process includes:
Updating the saliency area feature amount likelihood based on at least one of the input image, the saliency area image and the updated saliency area feature amount likelihood updated until the previous time,
The non-significant region feature amount likelihood update process includes:
The non-significant region feature value likelihood is updated based on at least one of the input image, the non-significant region image, and the updated non-significant region feature value likelihood updated up to the previous time. Item 5. The saliency image generation method according to Item 4. A smoothed image group generation process for generating a smoothed image group composed of a plurality of smoothed images obtained by smoothing the input image with different resolutions, respectively;
Perform the saliency area prior probability image extraction process, the feature amount likelihood calculation process, and the saliency area image extraction process on the leveled image group, and determine the saliency area image of the input image. And further comprising a process,
The feature amount likelihood calculation process and the saliency area image extraction process include:
Using the smoothed image instead of the input image,
The saliency image generation process includes:
For each input image, the attention level video extraction process, the saliency area prior probability image extraction process, the feature amount likelihood calculation process, the saliency area image extraction process, the smoothed image group generation process, and the saliency area image 6. The saliency area image generation method according to claim 1, wherein the saliency area image is generated from the saliency area image obtained by executing a determination process. An attention level video extraction unit that extracts a degree of attention video indicating a degree of attention that is a degree that a person is likely to pay attention from an input video;
A saliency area prior probability image extraction unit that extracts a saliency area prior probability image indicating the probability that each position of the input image that is each frame constituting the input video is a saliency area; and
A feature amount likelihood calculating unit that calculates a feature amount likelihood indicating the likelihood of the image feature amount included in each of the saliency area and the area outside the saliency area of the input image;
A saliency area image extracting unit that extracts a saliency area image indicating a saliency area of the input image from the input image, the saliency area prior probability image, and the feature amount likelihood;
From each saliency area image obtained by executing the attention level video extraction unit, the saliency area prior probability image extraction unit, the feature amount likelihood calculation unit, and the saliency area image extraction unit for each input image A saliency image generating unit for generating a saliency image,
The saliency area prior probability image extraction unit includes:
Attention degree which is an image corresponding to the input image in the attention degree video extracted by the attention degree video extracting unit, a saliency area prior probability image indicating the probability that each position of the one input image is a saliency area Extracting based on the image and the salient region image;
The feature amount likelihood calculating unit includes:
The feature quantity likelihood is calculated based on at least one of the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the feature quantity likelihood calculated up to the previous time. A saliency area image generating device. The attention level video extraction unit
A basic attention level image extraction unit that calculates a basic attention level image, which is an image displaying a spatial region having a remarkable characteristic in the input image, based on the input image;
A stochastic basic attention image, which is an image displaying prominence at each position of the current input image using a probabilistic expression, a basic attention image calculated by the basic attention image extraction unit, and a previous time The probabilistic basic attention level image calculated by the probabilistic basic attention level image extraction unit from the input image and a probabilistic basic attention level parameter that is sequentially updated and used as a first parameter for eye-gaze position estimation A stochastic basic attention level image extraction unit that is calculated based on
A gaze position probability density image, which is a frame of the gaze position probability density image in the current input image, is obtained from the probabilistic basic attention image calculated by the probabilistic basic attention image extraction unit and the previous input image. A line-of-sight position calculated based on the line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction unit and a line-of-sight position probability density parameter that is sequentially updated and used for line-of-sight position estimation A probability density image extraction unit;
A time series calculated by sequentially repeating the basic attention level image extraction unit, the probabilistic basic attention level image extraction unit, and the gaze position probability density image extraction unit for each of the input images. A line-of-sight position probability density image output unit that outputs the line-of-sight position probability density image as the line-of-sight position probability density image;
The line-of-sight position probability density image extraction unit,
A line-of-sight movement state variable, which is a random variable that controls the magnitude of line-of-sight movement, is calculated from the line-of-sight position probability density image calculated by the line-of-sight position probability density image extraction unit from the previous input image and the previous input image. A line-of-sight movement that is updated based on the line-of-sight movement state variable calculated by the line-of-sight movement state variable update unit and the line-of-sight position probability density parameter and outputs a line-of-sight movement state variable set that is a set of the line-of-sight movement state variables A state variable updater;
A representative gaze position set, which is a set of representative gaze positions indicating representative gaze positions considering gaze movement, a stochastic basic attention level image calculated by the probabilistic basic attention level image extraction unit, and the previous input A representative gaze position update unit that updates the representative gaze position set updated by the representative gaze position update unit based on the gaze movement state variable set and the gaze position probability density parameter;
A representative gaze position weighting coefficient set, which is a set of representative gaze position weighting coefficients composed of weights associated with each of the representative gaze positions, and a probabilistic basic attention degree image calculated by the probabilistic basic attention degree image extraction unit, The representative line-of-sight calculated based on the representative line-of-sight position set updated by the representative line-of-sight position update unit, the line-of-sight movement state variable set output from the line-of-sight movement state variable update unit, and the line-of-sight position probability density parameter A position weight coefficient calculation unit;
The line-of-sight position probability of calculating the line-of-sight position probability density image based on the representative line-of-sight position set updated by the representative line-of-sight position update unit and the representative line-of-sight position weight coefficient set calculated by the representative line-of-sight position weight coefficient calculating unit The saliency area image generation according to claim 7, further comprising: a density image output unit, wherein the sight line position probability density image including the representative sight line position set and the representative sight position weighting coefficient set is calculated. apparatus. The saliency area prior probability image extraction unit includes:
A saliency area prior probability image generation unit that generates the saliency area prior probability image using only the attention level image; and
8. The saliency area prior probability image update unit configured to update the saliency area prior probability image generated by the saliency area prior probability image generation unit using the saliency area image. Item 9. The saliency area image generation device according to any one of Items 8 to 9. The feature amount likelihood calculating unit includes:
The saliency area feature amount likelihood indicating the likelihood of the image feature amount included in the saliency area is calculated based on the input image, the saliency area prior probability image, the saliency area image, and the saliency area feature amount likelihood calculated up to the previous time. A saliency area feature amount likelihood calculating unit that calculates based on at least one of them,
The non-salience area feature amount likelihood indicating the likelihood of the image feature amount included in the area outside the saliency area is calculated as the input image, the saliency area prior probability image, the saliency area image, and the non-salience area calculated until the previous time. A non-significant region feature amount likelihood calculating unit that calculates based on at least one of the feature amount likelihoods;
10. The feature amount likelihood output unit configured to add the saliency region feature amount likelihood and the non-salience region feature amount likelihood and output as a feature amount likelihood. The saliency area | region video generation apparatus of any one of these. The saliency area feature likelihood calculating unit
A saliency area feature amount likelihood generating unit that generates the saliency area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image;
A saliency area feature amount likelihood update unit configured to update the saliency area feature amount likelihood generated by the saliency area feature amount likelihood generation unit;
The non-significant region feature amount likelihood calculating unit
A non-salience area feature amount likelihood generating unit that generates the non-salience area feature amount likelihood based on the input image, the saliency area prior probability image, and the saliency area image;
A non-salience area feature amount likelihood update unit configured to update the non-salience area feature amount likelihood generated by the non-salience area feature amount likelihood generation unit;
The saliency area feature likelihood update unit includes:
Updating the saliency area feature amount likelihood based on at least one of the input image, the saliency area image and the updated saliency area feature amount likelihood updated until the previous time,
The non-significant region feature amount likelihood update unit
The non-significant region feature value likelihood is updated based on at least one of the input image, the non-significant region image, and the updated non-significant region feature value likelihood updated up to the previous time. Item 13. The saliency image generating device according to Item 10. A smoothed image group generation unit that generates a smoothed image group composed of a plurality of smoothed images obtained by smoothing the input image at different resolutions;
A saliency area that executes processing of the saliency area prior probability image extraction unit, the feature amount likelihood calculation unit, and the saliency area image extraction unit for the leveled image group to determine the saliency area image of the input image An image determination unit;
The feature amount likelihood calculating unit and the salient region image extracting unit are:
Using the smoothed image instead of the input image,
The saliency area image generation unit includes:
For each input image, the attention level video extraction unit, the saliency area prior probability image extraction unit, the feature amount likelihood calculation unit, the saliency area image extraction unit, the smoothed image group generation unit, and the saliency area image 12. The saliency area image generation device according to claim 7, wherein the saliency area image is generated from the saliency area image obtained by executing each process of the determination unit. Attention level video extraction step for extracting an attention level video indicating a degree of attention that is the degree to which a human is likely to pay attention from an input video;
A saliency area prior probability image extraction step for extracting a saliency area prior probability image indicating a probability that each position of the input image that is each frame constituting the input video is a saliency area; and
A feature amount likelihood calculating step of calculating a feature amount likelihood indicating the likelihood of the image feature amount included in each of the saliency area and the area outside the saliency area of the input image;
A saliency area image extracting step of extracting a saliency area image indicating a saliency area of the input image from the input image, the saliency area prior probability image, and the feature amount likelihood; and
From each saliency area image obtained by executing the attention level video extraction step, the saliency area prior probability image extraction step, the feature amount likelihood calculation step and the saliency area image extraction step for each input image. A program for causing a computer to execute a saliency area image generation step for generating a saliency area image,
The saliency area prior probability image extraction step includes:
Attention degree which is an image corresponding to the input image in the attention degree video extracted by the attention degree video extraction step, a saliency area prior probability image indicating the probability that each position of the one input image is a saliency area Extracting based on the image and the salient region image;
The feature amount likelihood calculating step includes:
The feature quantity likelihood is calculated based on at least one of the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the feature quantity likelihood calculated up to the previous time. Program. Attention level video extraction step for extracting an attention level video indicating a degree of attention that is the degree to which a human is likely to pay attention from an input video;
A saliency area prior probability image extraction step for extracting a saliency area prior probability image indicating a probability that each position of the input image that is each frame constituting the input video is a saliency area; and
A feature amount likelihood calculating step of calculating a feature amount likelihood indicating the likelihood of the image feature amount included in each of the saliency area and the area outside the saliency area of the input image;
A saliency area image extracting step of extracting a saliency area image indicating a saliency area of the input image from the input image, the saliency area prior probability image, and the feature amount likelihood; and
From each saliency area image obtained by executing the attention level video extraction step, the saliency area prior probability image extraction step, the feature amount likelihood calculation step and the saliency area image extraction step for each input image. A computer-readable storage medium storing a program for causing a computer to execute a saliency area image generation step for generating a saliency area image,
The saliency area prior probability image extraction step includes:
Attention degree which is an image corresponding to the input image in the attention degree video extracted by the attention degree video extraction step, a saliency area prior probability image indicating the probability that each position of the one input image is a saliency area Extracting based on the image and the salient region image;
The feature amount likelihood calculating step includes:
The feature quantity likelihood is calculated based on at least one of the input image, the attention degree image, the saliency area prior probability image, the saliency area image, and the feature quantity likelihood calculated up to the previous time. A recording medium.