JP2015056041A - Image generation system, image generation method, image generation program, line of sight prediction system, line of sight prediction method, and line of sight prediction program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily obtain an image used for predicting an area having a tendency to attract visual attention in a scene.SOLUTION: An image generation system in one embodiment of the present invention is equipped with one or a plurality of processors. At least one of the processors (a) accepts an input image, (b) generates an image to be evaluated by referring to a storage unit for storing a pattern to be replaced and a replacement pattern in correlation with each other and replacing the pattern to be replaced that is specified within the input image with the replacement pattern, and (c) outputs the image to be evaluated in order to execute, in a line of sight prediction system, a process for predicting an area in the image to be evaluated that has a tendency to attract visual attention.

Description

  One aspect of the present invention relates to an image generation system, an image generation method, an image generation program, a gaze prediction system, a gaze prediction method, and a gaze prediction program.

  2. Description of the Related Art Conventionally, a technique for predicting a region that tends to attract visual attention in a scene is known. For example, Patent Document 1 below receives an input scene, applies a visual attention model to the input scene, and predicts a region in the input scene that tends to attract visual attention. A functional visual attention module and a robust functioning to interact with the visual attention module to determine the degree to which at least one of the identified areas is robust (prominent) or the scene is robust A computer system is described that includes a sexiness (saliency) assessment module. Patent Documents 2 and 3 below describe systems having functions similar to those of the computer system.

Special table 2012-504827 gazette Special table 2012-504828 gazette Special table 2012-504830 gazette

  When performing an evaluation relating to visual attention using the computer systems described in Patent Documents 1 to 3, that is, a gaze prediction, it is necessary to prepare an image of a scene to be evaluated. Even if one scene is evaluated, the line-of-sight prediction process is often performed while changing the pattern for a specific area in the scene. For example, even if the signboard that most effectively attracts the user's attention is predicted, many patterns are often evaluated while slightly changing the design (shape, pattern, color, etc.).

  Therefore, in general, when performing line-of-sight prediction, it is necessary to prepare a large number of images for a scene even when evaluating one scene. However, it is not easy for the user to prepare these images one by one. Therefore, a mechanism that can easily obtain an image used for predicting a region that tends to attract visual attention in a scene is desired.

  An image generation system according to an aspect of the present invention includes one or a plurality of processors, and at least one processor receives an input image and refers to a storage unit that stores a replacement target pattern and a replacement pattern in association with each other, By replacing the replacement target pattern specified in the input image with the replacement pattern, the evaluation target image is generated, and the process that predicts the region that tends to attract visual attention in the evaluation target image is executed in the gaze prediction system In order to do so, the evaluation target image is output.

  In such an aspect, a replacement pattern corresponding to the replacement target pattern is stored in advance, and the replacement target pattern included in the input image is replaced with the replacement pattern using the data. Then, the evaluation target image generated by this replacement is output for prediction of a region that tends to attract visual attention. Since a part of the input image is automatically converted by this series of processing, if the user prepares only one input image for a certain scene, an evaluation target image having a part of a different area is obtained from the input image. It becomes possible. That is, it is possible to easily obtain an image used for predicting a region that tends to attract visual attention in a scene.

  According to an aspect of the present invention, it is possible to easily obtain an image used for predicting a region that tends to attract visual attention in a scene.

It is a figure which shows the whole image of a gaze prediction system. It is a flowchart which shows operation | movement of an analysis system. It is a figure which shows the example of a display of the evaluation result of an analysis system. It is a figure which shows the example of pattern information. It is a figure which shows the hardware constitutions of the image generation system which concerns on embodiment. It is a block diagram which shows the function structure of the image generation system which concerns on embodiment. It is a figure which shows the example of an input image. It is a figure which shows the example of an evaluation object image. It is a flowchart which shows operation | movement of the image generation system which concerns on embodiment. It is a figure which shows the structure of the image generation program which concerns on embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant description is omitted.

  The function and configuration of the image generation system according to the embodiment will be described with reference to FIGS. In the present embodiment, the image generation system is a part of the line-of-sight prediction system 1. The line-of-sight prediction system 1 is a computer system that predicts a region that tends to attract visual attention in a scene. As shown in FIG. 1, the user can access the line-of-sight prediction system 1 from the user terminal T via the network N and obtain an evaluation result for a desired scene.

  The type of the user terminal T is not limited, and may be a stationary or portable personal computer, or a portable terminal such as a high-function mobile phone (smart phone), a mobile phone, or a personal digital assistant (PDA). The specific configuration of the network N is not limited, and may be configured from at least one of the Internet, a dedicated line, and a LAN (Local Area Network), for example. Although three user terminals T are shown in FIG. 1, the number of user terminals T in the line-of-sight prediction system 1 is not limited.

  As shown in FIG. 1, the line-of-sight prediction system 1 includes an image generation system 10 and an analysis system 20. The image generation system 10 efficiently generates an image obtained by photographing the scene to be evaluated (referred to as “evaluation target image” in this specification), and the analysis system 20 easily collects where the human eyes are gathered in the evaluation target image. Predict. The feature of this embodiment lies in the image generation system 10. First, in order to facilitate understanding of the image generation system 10, the operation of the analysis system 20 will be described.

  As shown in FIG. 2, the analysis system 20 receives the evaluation target image (step S11), executes line-of-sight prediction on the image (step S12), and finally outputs the evaluation result (step S13). A series of processes shown in steps S11 to S13 corresponds to an analysis step of predicting a region that tends to attract visual attention in the evaluation target image.

  The evaluation target image input to the analysis system 20 in step S11 may be a still image or a moving image. In this embodiment, in order to simplify the description, it is assumed that the evaluation target image is a still image. The subject of the evaluation target image, that is, the scene is not limited. For example, the evaluation target image may be an image showing an urban area or a store where an advertisement or a signboard exists, or an advertisement, a signboard, a product display shelf, an in-store POP, or a printed label or card itself. It may be an image, an image showing a product or a product package itself, or a captured image of a web page. These “images” may be photographs or computer graphics. In the present embodiment, description will be made using an example in which an image of an urban area where an advertisement or a signboard is present is used as an evaluation target image.

  In step S <b> 12, the analysis system 20 simulates the movement of a person's line of sight with respect to the evaluation target image. Since various conventional techniques can be used in this process, a description of a specific method is omitted in this specification. For example, eye gaze prediction is executed using the methods described in Patent Documents 1 to 3 above. Also good.

  In step S13, the analysis system 20 displays the evaluation result, but the expression method of the result is not limited. A display example of the evaluation result is shown in FIG. For example, the analysis system 20 may output a heat map 51 showing the distribution of the ease of gathering of the line of sight with color coding for the evaluation target image 50. Alternatively, the analysis system 20 may output a region map 52 showing the distribution divided into several stages (for example, three stages). The region map 52 is a technique for indicating the area where the line of sight gathers with a frame and indicating the concentration of the line of sight for each area with a numerical value, but the numerical value is omitted in FIG. Alternatively, the analysis system 20 may output a line-of-sight tracking chart 53 that indicates the movement of the line of sight as a line. The display of the evaluation result is also described in Patent Documents 1 to 3. As described above, there are various expression methods, but in any case, the user can know from the evaluation result what design is easy to collect lines of sight (highly noticeable).

  Also, the output destination of the evaluation result is not limited. For example, the analysis system 20 may transmit the evaluation result as shown in FIG. 3 to the user terminal T and display it on the terminal T, or instead of this transmission. Alternatively, the result may be stored in an arbitrary storage unit.

  The user can use this analysis system 20 to predict, for example, a signboard or an advertisement that attracts people's eyes. For this purpose, a plurality of types of design proposals are prepared in advance for one signboard or advertisement. The analysis system 20 needs to be predicted. For example, when it is desired to know what signboard stands out in a certain shopping street, the user needs to input images of a plurality of shopping streets that differ only in the signboard portion to the analysis system 20. Although it is not easy for the user himself to perform such work, the gaze prediction system 1 includes the image generation system 10, and thus the user can easily obtain a plurality of patterns of evaluation target images.

  The image generation system 10 refers to the pattern database 30 in order to generate an evaluation target image automatically or semi-automatically. In the line-of-sight prediction system 1, the image generation system 10 and the pattern database 30 may be separate. In this case, the image generation system 10 accesses the pattern database 30 via a network. Alternatively, one computer may implement both the image generation system 10 and the pattern database 30.

  The pattern database 30 is a device (storage unit) that stores a plurality of patterns prepared for each model. A “model” in this specification is an object to be replaced. Examples of the object include a signboard or advertisement installed in an urban area, a banner advertisement or a logo on a web page, but the object set as a model is not limited to these.

  An example of pattern information stored in the pattern database 30 is shown in FIG. A record of pattern information is prepared for each model, and a single marker as used in augmented reality (AR) and a plurality of patterns are associated with each model. A pattern is an image, and the creation method is not particularly limited. For example, the pattern may be created by computer graphics, may be created from an image of a real landscape, or may be created by scanning a hand-drawn illustration.

  In the example of FIG. 4, three patterns for model A and two patterns for model B are shown. For model A, the three patterns are common in that an asterisk is placed at the center, but the combination of foreground and background colors differs between the patterns. For model B, the arrangement of characters and the background design differ between patterns. The number of patterns prepared in each model is not limited, and the number of patterns does not need to be unified among the models.

  As shown in FIG. 4, each pattern is formed by a combination of an original pattern image (left side) and a mask image (right side) indicated by a black outer edge. The pattern image and the mask image have the same size. The mask image is used to make an extra portion of the pattern image transparent when the pattern image is superimposed on the input image. For example, the pattern of the model B is expressed as an octagonal mark instead of a quadrangle when superimposed on the input image. On the other hand, the marker is prepared as a single image with an outer frame, instead of being divided into two like a pattern.

  The specific configurations of the pattern database and the pattern information are not limited to those shown in FIG. 4, and any normalization or redundancy may be performed. For example, if the pattern is replaced without using a marker as will be described later, the marker can be omitted in the pattern information. In FIG. 4, the marker and the pattern are shown as a two-dimensional image, but the marker and the pattern may be a three-dimensional image.

  Next, the image generation system 10 will be described in detail. The hardware configuration of the image generation system 10 is as shown in FIG. The image generation system 10 includes a CPU 101 that executes an operating system, application programs, and the like, a main storage unit 102 that includes a ROM and a RAM, an auxiliary storage unit 103 that includes a hard disk, a flash memory, and the like, and a network card or The communication control unit 104 includes a wireless communication module, an input device 105 such as a keyboard and a mouse, and an output device 106 such as a display.

  Each functional component of the image generation system 10 to be described later reads predetermined software on the CPU 101 or the main storage unit 102, and controls the communication control unit 104, the input device 105, the output device 106, and the like under the control of the CPU 101. This is realized by operating and reading and writing data in the main storage unit 102 or the auxiliary storage unit 103. Data and a database necessary for processing are stored in the main storage unit 102 or the auxiliary storage unit 103.

  Note that the image generation system 10 may be configured by a single computer or a plurality of computers.

  As illustrated in FIG. 6, the image generation system 10 includes a reception unit 11, a generation unit 12, and an output unit 13 as functional components. These functions are realized mainly by the operation of at least one processor (CPU 101).

  The accepting unit 11 is a functional element that accepts an input image designated by the user. The reception unit 11 outputs the acquired input image to the generation unit 12. The receiving unit 11 may receive an input image sent from the user terminal T, or may read an image from a predetermined storage unit (for example, a memory or a database) based on an instruction from the user terminal T. Good. An example of the input image is shown in FIG. The input images 61 and 62 are both photographs of the urban area. In the input image 61, the actual designs are drawn on the signboards 61a and 61b. On the other hand, in the input image 62, markers are placed on the signboards 62a and 62b. It is listed. Since it is usually difficult to put a marker on a signboard part in the real world, it is more practical to use the input image 61, but the present invention does not exclude the input image 62 including the marker.

  The generation unit 12 is a functional element that generates an evaluation target image by replacing a part of the input image. Although replacement may occur in a plurality of regions in one input image, the replacement in one region will be described below for the sake of simplicity.

  First, the generation unit 12 specifies a replacement target pattern. When a markerless input image is input, the generation unit 12 may specify a replacement target pattern in accordance with a user operation. In this case, the user designates an area where the pattern is to be replaced by touch operation or mouse operation, and the user terminal T transmits data indicating the area to the image generation system 10, and the generation unit 12 is indicated by the data. Keep the area. Alternatively, the generation unit 12 may automatically specify the replacement target pattern without depending on such a user operation. In this case, the generation unit 12 specifies a replacement target pattern by comparing (pattern matching) an arbitrary region in the input image with each pattern in the pattern database 30. Note that pattern matching itself is well known. Specifically, there are methods such as feature amount matching (feature point base matching) and template matching (correlation base matching), but any method may be used.

  When an input image including a marker is input, the generation unit 12 specifies the marker in the input image as a replacement target pattern without depending on the user operation by pattern matching using the marker in the pattern database 30. Even in this case, a specific method of pattern matching may be arbitrarily determined.

  Subsequently, the generation unit 12 acquires a replacement pattern for replacement with the replacement target pattern. Here, the replacement pattern is a pattern having a model common to the replacement target pattern and different from the replacement target pattern. The generation unit 12 provides the user terminal T with a user interface for selecting one or more replacement patterns corresponding to the replacement target pattern from the pattern database 30, and obtains one or more replacement patterns selected via the interface. May be. Alternatively, the generation unit 12 may read the replacement pattern corresponding to the replacement target pattern without depending on the user operation with reference to the pattern database 30.

  Subsequently, the generation unit 12 deforms the replacement pattern by projective transformation in order to match the size and orientation of the acquired replacement pattern with the replacement target pattern, and superimposes the deformed replacement pattern on the input image.

If the coordinates before conversion are (x, y) and the coordinates after conversion are (x ′, y ′), the projective transformation is expressed by the following equation.
x ′ = (a ₁ x + b ₁ y + c ₁ ) / (a ₀ x + b ₀ y + c ₀ )
y ′ = (a ₂ x + b ₂ y + c ₂ ) / (a ₀ x + b ₀ y + c ₀ )

In these formulas, the number of independent unknowns is 8, so if the coordinates of four or more points are obtained, the unknowns can be obtained. Therefore, the generation unit 12 specifies the coordinates of the four vertices from each of the replacement target pattern and the replacement pattern. The generation unit 12 may acquire a point designated by the user as a vertex, or may acquire a vertex by analyzing each pattern. Alternatively, the coordinates of the four vertices of each pattern may be included in the pattern information in advance, and the generation unit 12 may read the vertex coordinates from the pattern database 30. Generator 12 vertices in pattern to be replaced _(x n, _{y n)} and vertex in substitution pattern _{(x'n, y'n) (both,} n = 1, 2, 3, 4) the expression of the projective transformation To obtain a projective transformation matrix P ₁ (second projective transformation matrix) represented by the following equation. Note that s is an arbitrary real number in the homogeneous coordinate expression (s ≠ 0).

  Subsequently, the generation unit 12 obtains the coordinates of a large number of feature points in each of the replacement target pattern and the replacement pattern, and calculates feature amounts for the individual feature points. The method of detecting feature points and determining the feature amount is arbitrary, and for example, a well-known algorithm such as ORB, SIFT, or SURF can be used. Since there is a possibility that the replacement target pattern in the input image is unclear, the generation unit 12 may perform sharpening processing and / or smoothing processing on the replacement target pattern before calculating the feature amount. The sharpening process is a filter process that enhances a blurred outline in an image, and is also referred to as unsharp masking. This sharpening process is a well-known technique. The smoothing process is a process for smoothing the luminance value in the image or a process for removing noise in the image, and this technique is also well known. This smoothing process is performed using a smoothing filter such as a Gaussian filter.

Subsequently, the generation unit 12 compares the feature amounts of both patterns, and uses, for example, RANSAC, which is one of robust estimation methods, to projective transformation matrix P ₂ (first projective transformation matrix) represented by the following formula: Is calculated. s is an arbitrary real number in the homogeneous coordinate expression (s ≠ 0).

Subsequently, the generation unit 12 converts the coordinates of the replacement pattern from (x, y) to (x ′, y ′) using the following expression including the matrices P ₁ and P ₂ indicating the distortion of the replacement target pattern. The coordinates (x, y) are the coordinates of the original replacement pattern (replacement pattern defined by the pattern information), and the coordinates (x ′, y ′) are the coordinates of the replacement pattern after deformation. The generation unit 12 superimposes the transformed replacement pattern on the input image.

As shown in this equation, generator 12 may not be used matrix P _1. Matrix P ₁ is used to further improve the accuracy of the deformation, for example, is remarkable its effect when the attitude of the pattern to be replaced is different than a predetermined and substitution pattern. Since the matrix P ₁ plays an auxiliary role, generating unit 12 may calculate and use only the matrix P ₂ in the projective transformation.

  The generation unit 12 matches the color and blur of the replacement pattern with the surrounding area in order to reduce the uncomfortable feeling of the superimposed replacement pattern. Well-known techniques can also be used for correction of color and blur. The generation unit 12 may superimpose the replacement pattern on the input image after correcting the color of the replacement pattern.

The color correction is expressed by the following equation using the conversion matrix M or M ′ corresponding to the color space. M is a conversion matrix when the RGB color system is used, and M ′ is a conversion matrix when the L * C * h color system is used. R, G, and B are the original three primary colors (red, green, and blue) of the replacement pattern, and R ′, G ′, and B ′ are the three primary colors after the replacement pattern is adjusted. L *, C *, and h are the original brightness, saturation, and hue of the replacement pattern, respectively. L * ′, C * ′, and h ′ are the brightness, saturation, and the adjusted pattern, respectively. Hue. Note that any color space may be used for color correction.

  The generation unit 12 may obtain a conversion matrix based on RGB values or L * C * h values input by a user via a user interface such as a slider. Alternatively, the generation unit 12 may obtain the conversion matrix using a well-known technique without depending on the user operation.

  The correction of the degree of blur can be realized by a smoothing process using a smoothing filter such as a Gaussian filter. The matrix H used for the smoothing process is determined according to the size of the smoothing filter (for example, 3 × 3 or 5 × 5). Regarding the filter size, the generation unit 12 may accept a value input by the user via a user interface such as a slider, or may determine using a known technique without depending on a user operation.

  The generation unit 12 corrects the color of the pattern by applying the conversion matrix M or M ′ to the replacement pattern superimposed on the input image. Further, the generation unit 12 applies the matrix H to the replacement pattern to blur the pattern.

With the above processing, the replacement target pattern is replaced with the replacement pattern in one area of the input image. If a plurality of replacement target patterns exist in one input image, or if a plurality of replacement target patterns are designated by the user, the generation unit 12 performs the above replacement for each replacement target pattern. At this time, various matrices P ₁ , P ₂ , M (or M ′), and H are generated for each replacement target pattern. The generation unit 12 generates a first evaluation target image by processing all the replacement target patterns.

The generation unit 12 may generate a plurality of evaluation target images from one input image. For example, if the model A shown in FIG. 4 is included in one region of the input image, the generation unit 12 can generate three evaluation target images by individually applying the patterns 1 to 3 to the model A. . In this case, the deformation of the replacement pattern, the color correction, and the blur correction in the region are common among the three patterns. Therefore, the generation unit 12 holds the various matrices P ₁ , P ₂ , M (or M ′), and H calculated when generating the _first evaluation target image, and generates the second and subsequent evaluation target images. These matrices are reused when doing so. If both models A and B shown in FIG. 4 are included in the input image, the generation unit 12 applies patterns 1 to 3 individually for model A, and applies patterns 1 and 2 individually to model B. Therefore, a total of 6 (= 3 × 2) evaluation target images can be generated. The generation unit 12 outputs the generated one or more evaluation target images to the output unit 13.

  The output unit 13 is a functional element that outputs an evaluation target image for processing in the analysis system 20 (that is, for processing that predicts a region that tends to attract visual attention in the evaluation target image). . FIG. 8 is an example of output, and shows three images to be evaluated to which each pattern of the model A shown in FIG. 4 is applied. If the input image 61 in which the pattern 1 is pasted on the signboards 61a and 61b is received, the evaluation target image 72 replaced with the pattern 2 and the evaluation target image 73 replaced with the pattern 3 are output. If the input image 62 in which the markers are pasted on the signboards 62a and 62b is received, the evaluation target image 71 replaced with the pattern 1 and the evaluation target images 72 and 73 are output. Although not represented in FIG. 8, the signboards 71a, 71b, 72a, 72b, 73a, and 73b are subjected to color correction and smoothing processing as necessary.

  The output unit 13 may transmit the image to the user terminal T in order to make the user confirm the evaluation target image. In this case, after confirming the sent evaluation target image, the user operates the user terminal T to instruct the analysis system 20 to perform line-of-sight prediction processing. Alternatively, the output unit 13 may output the evaluation target image to the analysis system 20, and in this case, the user confirms the evaluation result sent from the analysis system 20 alone or together with the evaluation target image.

  Next, the operation of the image generation system 10 will be described with reference to FIG. 9, and the image generation method according to the present embodiment will be described.

  When the reception unit 11 receives an input image (step S21, reception step), the generation unit 12 generates an evaluation target image. Specifically, the generation unit 12 specifies a replacement target pattern (step S22) and acquires a replacement pattern (step S23). These two types of patterns may be designated by the user, or may be automatically determined without depending on the user operation.

  Subsequently, the generation unit 12 deforms the replacement pattern by projective transformation based on the four vertices and projective transformation based on the feature points (step S24). In step S24, the generation unit 12 may perform only projective transformation based on feature points. Subsequently, the generation unit 12 superimposes the converted replacement pattern on the input image (step S25), and executes color correction and smoothing processing on the replacement pattern (step S26). Thereby, the insertion of one replacement pattern is completed.

  If there are other replacement target patterns to be processed, the generation unit 12 repeats the processing of steps S22 to S26 (see step S27), thereby completing one evaluation target image (step S28). If there are other evaluation target images to be generated, the generation unit 12 repeats the processes of steps S22 to S28 corresponding to the generation step (see step S29). Finally, the one or more evaluation target images generated by the output unit 13 are output (step S30, output step). The generated evaluation target image is input to the analysis system 20 by a user operation or automatically, and the analysis system 20 performs a line-of-sight prediction process on the evaluation target image. Therefore, the line-of-sight prediction method according to the present embodiment includes the processes shown in FIGS.

  Next, an image generation program P1 for realizing the image generation system 10 will be described with reference to FIG.

  The image generation program P1 includes a main module P10, a reception module P11, a generation module P12, and an output module P13.

  The main module P10 is a part that comprehensively controls the image generation function. The functions realized by executing the reception module P11, the generation module P12, and the output module P13 are the same as the functions of the reception unit 11, the generation unit 12, and the output unit 13, respectively.

  The image generation program P1 may be provided after being fixedly recorded on a tangible recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. The image generation program P1 may be provided as a data signal superimposed on a carrier wave via a communication network. The image generation program P1 may be provided as a line-of-sight prediction program together with an analysis program for realizing the analysis system 20.

  As described above, the image generation system according to one aspect of the present invention includes one or a plurality of processors, and at least one processor receives an input image and stores a replacement target pattern and a replacement pattern in association with each other. Processing for generating an evaluation target image by referring to the replacement part and replacing the replacement target pattern specified in the input image with a replacement pattern, and predicting a region that tends to attract visual attention in the evaluation target image Is output in the line-of-sight prediction system.

  The image generation method according to one aspect of the present invention includes an accepting step in which at least one processor receives an input image, and an input with reference to a storage unit that stores the replacement target pattern and the replacement pattern in association with each other. A generation step for generating an evaluation target image by replacing the replacement target pattern specified in the image with a replacement pattern, and a process for predicting an area in the evaluation target image that tends to attract visual attention. An output step of outputting the evaluation target image for execution in the line-of-sight prediction system.

  In addition, the image generation program according to one aspect of the present invention refers to a receiving step that receives an input image, and a storage unit that stores the replacement target pattern and the replacement pattern in association with each other. In order to execute a generation step of generating an evaluation target image by replacing a pattern with a replacement pattern and a process of predicting a region that tends to attract visual attention in the evaluation target image in the gaze prediction system. Causing the computer to execute an output step of outputting the target image.

  In such an aspect, a replacement pattern corresponding to the replacement target pattern is stored in advance, and the replacement target pattern included in the input image is replaced with the replacement pattern using the data. Then, the evaluation target image generated by this replacement is output for prediction of a region that tends to attract visual attention. Since a part of the input image is automatically converted by this series of processing, if the user prepares only one input image for a certain scene, an evaluation target image having a part of a different area is obtained from the input image. It becomes possible. That is, it is possible to easily obtain an image used for predicting a region that tends to attract visual attention in a scene.

  In another aspect of the present invention, at least one processor calculates a projective transformation matrix indicating distortion of a replacement target pattern in the input image, and transforms the replacement pattern by applying the projective transformation matrix to the replacement pattern. The identified replacement target pattern may be replaced with a modified replacement pattern. In this way, by changing the replacement pattern in accordance with the replacement target pattern in the input image and applying the replacement pattern to the input image, it is possible to generate an evaluation target image with less discomfort.

  In another aspect of the present invention, at least one processor calculates a first projective transformation matrix from the coordinates of the feature points of the replacement target pattern in the input image and the coordinates of the feature points of the replacement pattern. The replacement pattern may be transformed by applying a projective transformation matrix to the replacement pattern. By obtaining the projective transformation matrix from the coordinates of the feature points, the replacement pattern can be accurately deformed.

  In another aspect of the present invention, at least one processor generates a second projective transformation matrix from the coordinates of the four vertices specified in the replacement target pattern in the input image and the coordinates of the four vertices specified in the replacement pattern. The replacement pattern may be transformed by calculating, applying the second projective transformation matrix to the replacement pattern, and then applying the first projective transformation matrix. First, roughly replace the replacement pattern using the projection transformation matrix obtained from the vertex coordinates, and then further modify the replacement pattern using the projection transformation matrix obtained from the coordinates of the feature points. It can be deformed with high accuracy.

  In another aspect of the present invention, at least one processor identifies a replacement target pattern existing in the input image by comparing the replacement target pattern stored in the storage unit with an area in the input image. May be. In this case, the user can save time and labor for specifying the replacement target pattern, and the work load on the user can be further reduced.

  A line-of-sight prediction system according to one aspect of the present invention includes the above-described image generation system and an analysis system that predicts a region that tends to attract visual attention in an evaluation target image.

  A gaze prediction method according to an aspect of the present invention includes an analysis step of predicting a region that tends to attract visual attention in an evaluation target image output by the image generation method.

  A line-of-sight prediction program according to an aspect of the present invention includes the above-described image generation program and an analysis program for causing a computer to execute an analysis step of predicting a region that tends to attract visual attention in an evaluation target image. .

  The present invention has been described in detail based on the embodiments. However, the present invention is not limited to the above embodiment. The present invention can be variously modified without departing from the gist thereof.

  As described above, the present invention can also be applied to moving images. In this case, the image generation system 10 generates an evaluation object frame for the first frame in the same manner as in the above-described embodiment (processing for a still image), and for the subsequent frames, the optical flow (moving object tracking) is used for the previous frame. The evaluation target frame may be generated by moving the frame replacement pattern. Alternatively, the image generation system 10 may generate an evaluation target frame for each frame in the same manner as in the above embodiment (processing for a still image).

  In the above embodiment, the generation unit 12 performs the color correction and smoothing process, but one or both of these processes may be omitted. Further, the deformation of the replacement pattern by projective transformation can be omitted.

  The present invention can also be applied to a single computer. For example, you may implement the function of the whole gaze prediction system in a user terminal. Alternatively, the functions of the image generation system and the analysis system may be mounted on the user terminal, and the pattern database may be arranged on the network. In this case, the user terminal executes the creation of the evaluation target image and the gaze prediction while accessing the pattern database each time. Alternatively, the user terminal may also have the functions of the image generation system 10 and the pattern database 30, and the user terminal may access the analysis system 20 via the network N.

  DESCRIPTION OF SYMBOLS 1 ... Gaze prediction system, 10 ... Image generation system, 11 ... Reception part, 12 ... Generation part, 13 ... Output part, 20 ... Analysis system, 30 ... Pattern database (memory | storage part), 101 ... CPU (processor), P1 ... Image generation program, P10 ... main module, P11 ... reception module, P12 ... generation module, P13 ... output module, T ... user terminal.

Claims (10)

With one or more processors,
At least one of the processors is
Accept input images,
By referring to the storage unit that stores the replacement target pattern and the replacement pattern in association with each other, by replacing the replacement target pattern specified in the input image with the replacement pattern, an evaluation target image is generated,
Outputting the evaluation target image in order to execute a process of predicting a region that tends to attract visual attention in the evaluation target image in the gaze prediction system;
Image generation system. The at least one processor comprises:
Calculating a projective transformation matrix indicating distortion of the replacement target pattern in the input image;
Deforming the replacement pattern by applying the projective transformation matrix to the replacement pattern;
Replacing the specified replacement target pattern with the modified replacement pattern;
The image generation system according to claim 1. The at least one processor comprises:
Calculating a first projective transformation matrix from the coordinates of the feature points of the replacement target pattern in the input image and the coordinates of the feature points of the replacement pattern;
Deforming the replacement pattern by applying the first projective transformation matrix to the replacement pattern;
The image generation system according to claim 2. The at least one processor comprises:
Calculating a second projective transformation matrix from the coordinates of the four vertices specified in the replacement target pattern in the input image and the coordinates of the four vertices specified in the replacement pattern;
Applying the second projective transformation matrix to the replacement pattern, and then applying the first projective transformation matrix to deform the replacement pattern;
The image generation system according to claim 3. The at least one processor identifies the replacement target pattern existing in the input image by comparing the replacement target pattern stored in the storage unit with a region in the input image;
The image generation system as described in any one of Claims 1-4. An accepting step in which at least one processor accepts an input image;
The processor generates an evaluation target image by referring to a storage unit that stores the replacement target pattern and the replacement pattern in association with each other and replacing the replacement target pattern specified in the input image with the replacement pattern. Generating step to
An image generation method including: an output step of outputting the evaluation target image so that the processor predicts a region in the evaluation target image that tends to attract visual attention in the line-of-sight prediction system. A reception step for receiving an input image;
A generation step of generating an evaluation target image by replacing the replacement target pattern specified in the input image with the replacement pattern with reference to a storage unit that stores the replacement target pattern and the replacement pattern in association with each other. ,
An image generation program for causing a computer to execute an output step of outputting an evaluation target image in order to execute a process of predicting a region that tends to attract visual attention in the evaluation target image in the gaze prediction system. The image generation system according to any one of claims 1 to 5,
A line-of-sight prediction system comprising: an analysis system that predicts a region that tends to attract visual attention in the evaluation target image.   A line-of-sight prediction method including an analysis step of predicting a region that tends to attract visual attention in the evaluation target image output by the image generation method according to claim 6. An image generation program according to claim 7;
A line-of-sight prediction program including an analysis program for causing a computer to execute an analysis step of predicting an area that tends to attract visual attention in the evaluation target image.

Images