JP2009169644A - Image generating device, information terminal, image generating method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To estimate a presentation region optimal to a viewer by applying a dynamic outline model from a viewing history. <P>SOLUTION: This image generating device 1 generates one image from a plurality of input images. This image generating device includes a region estimating part 14 which determines an estimated region from the plurality of input images so that an energy function is minimized using the dynamic outline model by which the energy function containing narrowing energy to narrow down a region is defined to contain a most common image from the plurality of input images. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image generation device, an information terminal, an image generation method, and a program, and in particular, by applying a dynamic contour model from a viewing history that is a set of partial areas selected from an original image by a plurality of viewers, for example. The present invention relates to an image generation apparatus, an information terminal, an image generation method, and a program suitable for generating an image by estimating an optimum presentation area for a viewer.

  When viewing visual content such as moving image content and text, there is a spatial clipping range (optimum presentation region) that provides the highest satisfaction for the viewer. For example, in the case of moving image content obtained by shooting a soccer game, it refers to a case in which the vicinity of the ball is zoomed to follow the movement, not the entire field.

  In recent years, the viewing environment for videos has been diversified, such as the increase in the size of home TVs and the release of mobile phones with TVs. Therefore, a situation has arisen in which the viewer views high-resolution visual content on a low-resolution display. At that time, it is common to simply downsample the entire image of the high-resolution visual content.

  Therefore, Patent Document 1 proposes an interface that allows a viewer to freely perform spatial segmentation in an environment in which moving image content is viewed on a mobile terminal. Further, in Non-Patent Document 1, in order to efficiently collect operation histories of cut-out operation histories performed by a plurality of moving image content viewers, the viewers can make their own preferences while watching moving image content. In addition, an interface has been proposed that can be easily cut out.

  On the other hand, Non Patent Literature 2 proposes Snakes, which is an active contour model, as a model for extracting a contour from a still image. An active contour model is an algorithm used to extract an object region in an image, etc., defines an energy function determined by the contour, minimizes this energy, converges the contour, and achieves a predetermined purpose. Is.

Further, Non-Patent Document 3 proposes Active Tubes as a model for extracting a region corresponding to a moving object in a spatiotemporal image. Active Tubes can be regarded as having applied Snakes to a spatio-temporal image, and can extract an object in a spatio-temporal image using an energy minimization method as in Non-Patent Document 2.
JP 2005-341398 A Hidefumi Iwashita, Takamichi Miyata, Yoshinori Sakai, "Proposal of video content trimming based on viewer operation history," IMPS2006, pp. 55-56, (Nov. 2006). M. Kass, A. witkin, and D. Terzopoulos, "Snakes: active contour models," International Journal of Computer Vision, vol.1, no.4, pp. 321-331, (1988). Ryo Furukawa, Masakazu Imai, Takeshi Kanno, "Elastic contour model using spatio-temporal image and its convergence method," IEICE Transactions, D-II, Vol. J79-D-II, No.6, pp 1054-1063, (Jun. 1996).

  However, with the interface described in Patent Document 1, a clip region cut out by a viewer can be displayed on a screen, but an optimum presentation region cannot be estimated from different clip regions from a plurality of viewers. The interface described in Non-Patent Document 1 can collect operation histories from a plurality of viewers, but does not disclose a specific method for estimating the optimum presentation area from them.

  In addition, Snakes described in Non-Patent Document 2 extracts the outline of an object in a target image, but the extracted outline does not correspond to the format of the displayed screen. When displayed on the screen, it is not always the optimum presentation area. Furthermore, the Active Tubes described in Non-Patent Document 3 can be applied to spatio-temporal images as compared with Snakes. However, since the contour is extracted from the characteristics (luminance information) of the original image itself, it is optimally presented. The region is not necessarily extracted.

  The present invention has been made to solve such problems, and an image generation apparatus and an information terminal that can estimate an optimum presentation area for a viewer by applying a dynamic contour model from a viewing history An object of the present invention is to provide an image generation method and a program.

  The image generation apparatus according to the present invention generates one image from a plurality of input images. Estimating from the plurality of input images to minimize the energy function using an active contour model in which an energy function including a narrowing-down energy that narrows down a region so as to include the most common image from the plurality of input images is used. A region estimation unit for obtaining a region is provided.

  In the present invention, an area including the most common area is estimated from a plurality of selected areas for one original image. Thereby, the presentation area including many common areas can be estimated, and an average area can be estimated in a plurality of selection areas.

  Further, the narrowing energy includes internal energy relating to the inside of the estimation region and external energy relating to the outside of the estimation region, and the internal energy becomes smaller as the image of the estimation region is shared, The external energy may be such that the smaller the image common outside the estimation area, the smaller the energy. As a result, a more accurate region can be estimated.

  The energy function may further include size adjustment energy that brings the size of the estimated region close to a target value. Thereby, the presentation area close to the target value of size can be estimated.

  Further, the input image is obtained by cutting out a partial region of the original image, and the energy function further includes inter-image adjustment energy that becomes smaller as the positions cut out between the original images match. It is good to do so. Thereby, the movement amount of the position between frames is adjusted, and the quality of the estimated region can be improved.

  Further, the plurality of original images is time-series data composed of a plurality of frames, and the inter-image adjustment energy connects the cut-out positions of the previous and subsequent frames with straight lines, and the cut-out positions of the frames between the front and rear It is preferable that the energy decreases as the value approaches the straight line. Thereby, the movement amount of the position between frames is adjusted, and the quality of the estimated region can be further improved.

  The estimation area may be a rectangle, and the energy function may further include an aspect ratio adjustment energy that maintains a constant aspect ratio of the estimation area. Thereby, the quality in the aspect ratio of an area | region can be improved.

  The plurality of input images may be images obtained by cutting out some or all of one or a plurality of images. Thereby, in a moving image, the common area | region in a part in image can be estimated.

  The plurality of input images may be viewing histories that are collections of regions selected by a plurality of viewers viewing the original image. Thereby, the presentation area which many viewers desire can be estimated.

  Further, it is preferable to further include a viewing history storage unit that stores the viewing history, and a totaling processing unit that generates a histogram by counting the number of viewers for each region with reference to the viewing history storage unit. Thereby, the number of viewers can be defined as energy as density.

  The region estimation unit minimizes the energy function by a greedy method, and when the number of repetitions of the greedy method is a predetermined number or less, a region that minimizes the energy function other than the external energy and the inter-image adjustment energy. It is good to ask for. Thereby, convergence efficiency can be improved.

  Further, it is preferable to further include an input receiving unit that receives an area selected by the viewer as an input image and stores it in the viewing history storage unit. Thereby, the viewer can freely select a range.

  In the active contour model, an energy function further including energy related to the layout information of the original image is defined, and the region estimation unit may obtain an estimated region using the dynamic contour model. Thereby, the presentation area | region using the layout information of a WEB content can be estimated.

  The information terminal according to the present invention displays the original image, determines an area selected by the viewer, and transmits information on the area to the image generation apparatus according to the present invention. Thereby, automatic collection of the selection area | region is attained.

  The image generation method according to the present invention generates one image from a plurality of input images. Estimating from the plurality of input images to minimize the energy function using an active contour model in which an energy function including a narrowing-down energy that narrows down a region so as to include the most common image from the plurality of input images is used. A region estimation step for obtaining a region is provided. Thereby, the presentation area including many common areas can be estimated, and an average area can be estimated in a plurality of selection areas.

  Further, the narrowing energy includes internal energy relating to the inside of the estimation region and external energy relating to the outside of the estimation region, and the internal energy becomes smaller as the image of the estimation region is shared, The external energy may be such that the smaller the image common outside the estimation area, the smaller the energy. As a result, a more accurate region can be estimated.

  The energy function may further include size adjustment energy that brings the size of the estimation region close to a target value. Thereby, the presentation area close to the target value of size can be estimated.

  In addition, the input image is obtained by cutting out a part of the image area of the original image, and the energy function further reduces inter-image adjustment energy that becomes smaller as the positions cut out between the original images match. It is good to include. Thereby, the movement amount of the position between frames is adjusted, and the quality of the estimated region can be improved.

  Further, the plurality of original images is time-series data composed of a plurality of frames, and the inter-image adjustment energy is obtained by connecting the cut-out positions of the preceding and following frames with straight lines, and cutting out the frames between the previous and next frames. The energy may be reduced as the position approaches the straight line. Thereby, the movement amount of the position between frames is adjusted, and the quality of the estimated region can be further improved.

  The estimation area may be a rectangle, and the energy function may further include aspect ratio adjustment energy that maintains a constant aspect ratio of the estimation area. Thereby, the quality in the aspect ratio of an area | region can be improved.

  The plurality of input images may be images obtained by cutting out some or all of one or a plurality of images. Thereby, in a moving image, the common area | region in a part in image can be estimated.

  The plurality of input images may be viewing histories that are collections of regions selected by a plurality of viewers viewing the original image. Thereby, the presentation area which many viewers desire can be estimated.

  Further, it is preferable to further include an aggregation processing step of generating a histogram in which the number of viewers is aggregated for each area from the viewing history. Thereby, the number of viewers can be defined as energy as density.

  The region estimation step minimizes the energy function by a greedy method, and minimizes the energy function other than the external energy and the inter-image adjustment energy when the number of repetitions of the greedy method is a predetermined number or less. It is good to ask for. Thereby, convergence efficiency can be improved.

  Further, it is preferable to further include an input receiving step of receiving an area selected by the viewer as an input image and storing it in the viewing history storage unit. Thereby, the viewer can freely select a range.

  In the active contour model, an energy function further including energy related to the layout information of the original image is defined, and the region estimation step may obtain an estimated region using the dynamic contour model. Thereby, the presentation area | region using the layout information of a WEB content can be estimated.

  A program according to the present invention causes a computer to execute the processing of the above-described image generation method.

  According to the present invention, it is possible to provide an image generation device, an information terminal, an image generation method, and a program that can estimate an optimum presentation area for a viewer by applying a dynamic contour model from a viewing history.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary for the sake of clarity.

Embodiment 1 of the Invention
The present invention relates to an image generation apparatus that generates one image from a plurality of input images. The image generation apparatus includes an area estimation unit that obtains an estimation area that includes the most common image from the plurality of input images using an active contour model. At this time, as the active contour model, an energy function including a narrowing energy for narrowing down a region including the most common image from the plurality of input images is defined. Then, the region estimation unit generates an image based on the estimated region by obtaining an estimated region from the plurality of input images so as to minimize the energy function.

  In Embodiment 1 of the present invention, as a plurality of input images, a selection area in which a plurality of viewers cut out a desired portion from moving image content in advance is used, and there are many common areas among the selection areas. Obtained as an estimated area. And in order to obtain | require an estimation area | region, the energy function based on an active contour model is defined. The energy function includes narrowing-down energy that considers the number of viewers in the selected region as the viewer's attention and narrows the region with a large number of viewers as the estimated region. Further, in addition to the above-described narrowing energy, the energy function includes energy that brings the size of the estimated area closer to the target value, energy that adjusts the size of the selected area, the aspect ratio, and between the frames of the moving image content. The energy that smoothes the movement is defined. Next, the image generation apparatus according to Embodiment 1 of the present invention will be described in more detail.

  1 and 2 are diagrams illustrating a configuration example of the image generation apparatus 1 according to the first embodiment of the present invention. FIG. 1 illustrates a flow in which the image generation apparatus 1 distributes visual content to the information terminals 3a, 3b, and 3c, and acquires a viewing history of visual content for each viewer who uses the information terminals 3a, 3b, and 3c. It is a figure explaining. In FIG. 2, the image generation device 1 performs region estimation processing based on the viewing history to generate estimated region content, and the estimated region content is transmitted to the information terminals 3d, 3e, and 3f and other information terminals. It is a figure explaining the flow to deliver. Thus, the viewer can view the optimum presentation area in the visual content.

  The image generation apparatus 1 includes a content distribution unit 11, an input reception unit 12, a totalization processing unit 13, a region estimation unit 14, a cutout processing unit 15, a content storage unit 21, a viewing history storage unit 22, a histogram storage unit 23, and an estimated region content. A storage unit 24 is provided. The image generation apparatus 1 only needs to be configured by a general-purpose computer system, and may be, for example, a WEB server that distributes moving image content. The image generation apparatus 1 is connected to the information terminals 3a, 3b, and 3c via a network (not shown), and can transmit and receive visual content and other information.

  The content storage unit 21 stores visual content distributed to the information terminals 3a, 3b, and 3c. Here, the visual content is, for example, moving image content composed of time-series data composed of a plurality of frames, text, or the like. Each frame has a rectangular shape and includes a plurality of pixels, and the position of each pixel can be specified by coordinates within the frame.

  The content distribution unit 11 refers to the content storage unit 21 in response to a content distribution request from the information terminal 3a, 3b, or 3c, and transmits visual content to the information terminal that is the request source. Note that the content distribution unit 11 may distribute the visual content stored in the content storage unit 21 to the information terminals 3a, 3b, and 3c all at once. In addition, the content distribution unit 11 transmits the cut-out visual content sent from the cut-out processing unit 15 to the information terminal that is the request source. Furthermore, the content distribution unit 11 refers to the estimated area content storage unit 24 in response to the estimated area content distribution request from the information terminal 3d, 3e, or 3f, and transmits the estimated area content to the requesting information terminal.

  The input receiving unit 12 receives viewing history data from the information terminals 3 a, 3 b, or 3 c and stores it in the viewing history storage unit 22. At that time, the input receiving unit 12 associates each viewer who operates the information terminal 3a, 3b, or 3c with the selected area. Further, the input receiving unit 12 sends the selected area to the cutout processing unit 15.

  The cutout processing unit 15 acquires a selection area from the input receiving unit 12, refers to the content storage unit 21, cuts out visual content based on the selection area, and sends the cutout visual content to the content distribution unit 11.

  The viewing history storage unit 22 stores viewing history data of the visual content by each viewer who operates the information terminals 3a, 3b, and 3c. Here, the viewing history data is information representing a selection region that is a partial region in which each viewer selects the visual content displayed on the screens of the information terminals 3a, 3b, and 3c for each frame. The selection area is a rectangle. For example, the coordinates of two points (pixels) at the upper left and lower right of the rectangle, the coordinates of the four corners of the rectangle, the coordinates of one corner of the rectangle, the height of the rectangle, and the width Or the coordinates of all the pixels in the rectangle. That is, the viewing history data is information indicating which pixel set of each frame is selected per viewer. The selection area is rectangular in this embodiment, but is not limited to this.

  The totalization processing unit 13 refers to the viewing history storage unit 22, totals the number of viewers per pixel from the viewing history data, generates total histogram information, and stores the total histogram information in the histogram storage unit 23. Here, the total histogram information is information whose value is the number of viewers in each frame and each pixel.

  The histogram storage unit 23 stores total histogram information generated by the total processing unit 13. FIG. 7 is a diagram showing an example in which the aggregate histogram information is displayed in a graph. The graph shown in FIG. 7 is a graph of a three-dimensional space expressed by the frame height, the frame width, and the number of viewers in a certain frame. Here, the frame height and the frame width correspond to x and y coordinates per pixel in a certain frame. Therefore, an area having a high number of viewers on the graph indicates an area selected by more viewers, in other words, an area common to many selected areas.

  The region estimation unit 14 refers to the histogram storage unit 23, obtains a region for minimizing energy from an energy function described later, and estimates it as an estimation region. Thereafter, the region estimation unit 14 refers to the content storage unit 21 based on the estimation region, generates image data from the visual content, and stores the image data as the estimation region content in the estimation region content storage unit 24.

  The estimated area content storage unit 24 stores estimated area content generated from the visual content based on the area estimated by the area estimation unit 14 described later.

  The content storage unit 21, the viewing history storage unit 22, the histogram storage unit 23, and the estimated area content storage unit 24 may be a non-volatile storage device such as a hard disk drive or a flash memory, or a dynamic random access memory (DRAM). ) Or other volatile storage devices.

  Note that the configuration of the image generation apparatus 1 is not limited to FIGS. 1 and 2. For example, the content distribution unit 11, the input reception unit 12, the totalization processing unit 13, and the cutout processing unit 15 are realized by different computer systems and can be connected to the image generation apparatus 1 or communicated via a network. Good. Further, the content storage unit 21, the viewing history storage unit 22, the histogram storage unit 23, and the estimation area content storage unit 24 may be realized by different storage devices and connected to the image generation device 1.

  The information terminals 3a, 3b, and 3c shown in FIG. 1 are connected to the image generation apparatus 1 via a network, transmit a content distribution request to the image generation apparatus 1, and are distributed from the image generation apparatus 1. Is received and displayed on the screen. In addition, the information terminals 3a, 3b, and 3c are provided with an interface that can cut out an area viewed by the viewer with respect to the visual content displayed on the screen. For example, the information terminals 3a, 3b, and 3c only need to have an interface that can cut out (trim) an image while viewing the moving image content disclosed in Non-Patent Document 1. In addition, the information terminals 3a, 3b, and 3c are provided with an input device such as a mouse in order to instruct to cut out an image. The number of information terminals 3a, 3b, and 3c is not limited to this. The functions of the information terminals 3a, 3b, and 3c may be realized in the image generation device 1.

  The information terminals 3d, 3e, and 3f illustrated in FIG. 2 are connected to the image generation apparatus 1 via a network, transmit an estimated area content distribution request to the image generation apparatus 1, and are distributed from the image generation apparatus 1. The estimated area content is received and displayed on the screen. The information terminals 3d, 3e, and 3f may be general-purpose computer systems. For example, it may be a computer provided with a WEB browser. The number of information terminals 3d, 3e, and 3f is not limited to this. In addition, the information terminals that receive and display the estimated area content may be the information terminals 3a, 3b, and 3c.

  Note that the cutout processing unit 15 may be in the information terminals 3a, 3b, and 3c. For example, the information terminal 3a acquires a selection area cut out by a viewer who uses the information terminal 3a, cuts out the received visual content based on the selection area, and displays the cut-out visual content on the screen. do it.

  Hereinafter, schematic processing of an application example of the first embodiment of the present invention will be described with reference to a flowchart of FIG. FIG. 1 is a diagram showing the flow of processing in step S11 of FIG. 3, and is a diagram for explaining the operation of collecting viewing histories from the information terminals 3a, 3b, and 3c. FIG. 2 is a diagram showing the flow of processing in steps S12 to S14, and is a diagram for explaining the operation of generating estimated area content from the collected viewing history.

  As shown in FIG. 1, first, the image generation apparatus 1 collects information on a selected area in each viewer from the information terminals 3a, 3b, and 3c (S11). Specifically, first, the image generation device 1 receives a content transmission request from the information terminal 3a, and the content distribution unit 11 transmits the visual content stored in the content storage unit 21 to the requesting information terminal 3a. To do. Next, the information terminal 3a receives the visual content and displays it on the screen. Then, the viewer of the information terminal 3a selects a viewing area from the displayed visual content through the area selection screen 50 shown in FIG. Thereafter, the information terminal 3a transmits the selected area to the image generation device 1. Then, the input receiving unit 12 of the image generating apparatus 1 receives the viewing history data from the information terminal 3 a and stores it in the viewing history storage unit 22. At this time, the cutout processing unit 15 generates visual content after cutout from the visual content stored in the content storage unit 21 based on the selected area acquired from the input receiving unit 12, and the content distribution unit 11 The cut-out visual content is transmitted to the information terminal 3a. Step S11 is performed for all of the information terminals 3a, 3b, and 3c.

  Note that the selection area collection processing in step S11 is not limited to the above-described processing. For example, information representing a plurality of selection areas for the visual content may be stored in the viewing history storage unit 22 directly from the outside of the image generation apparatus 1.

  Next, as illustrated in FIG. 2, the image generation device 1 generates aggregate histogram information (S12). Specifically, the totalization processing unit 13 of the image generation apparatus 1 refers to the viewing history storage unit 22, divides each viewer's selection area into pixels, totals the number of viewers per pixel, and totals histogram Information is generated, and the total histogram information is stored in the histogram storage unit 23. That is, if the same pixel exists in the viewing history data of different viewers, it is considered that the same pixel has been selected. That is, the total histogram information has a larger value (number of viewers) as the number of common pixels (images) increases in a plurality of selected areas.

  Subsequently, as illustrated in FIG. 2, the image generation apparatus 1 estimates a region (S13). Specifically, first, the region estimation unit 14 of the image generation device 1 refers to the histogram storage unit 23 and acquires aggregate histogram information. Next, the region estimation unit 14 inputs the total histogram information into an energy function described later, and obtains an optimal solution by the energy minimization process shown in FIG. That is, the region estimation unit 14 calculates a set of pixels indicating a region that minimizes the energy function. Subsequently, the region estimation unit 14 refers to the content storage unit 21 based on the estimated region that is the optimal solution, and generates image data from the visual content. Then, the region estimation unit 14 stores the generated image data in the estimation region content storage unit 24 as the estimation region content.

  Thereafter, as shown in FIG. 2, the image generation device 1 distributes the estimated area content to the information terminals 3a, 3b, and 3c (S14). Specifically, first, the image generation apparatus 1 receives the estimated area content transmission request from the information terminal 3a, and the content distribution unit 11 uses the estimated area content stored in the estimated area content storage unit 24 as a request source. It transmits to the information terminal 3a. Next, the information terminal 3a receives the estimated area content and displays it on the screen. Thereby, the viewer of the information terminal 3a can view the visual content in which the optimum presentation area is estimated, not the original visual content. Similarly, the image generation device 1 can distribute the estimated area content to the information terminals 3b and 3c and other information terminals.

  Here, the active contour model employed in the region estimation process according to the first embodiment of the present invention will be described. FIG. 4 is a diagram illustrating an application example of the active contour model. 4A shows an initial state when the active contour model is applied to the target object 4 located in the center, and FIG. 4B shows the contour of the target object 4 by applying the active contour model. Represents the state after convergence. The target object 4 has a target area surrounded by edges. The dynamic contour model extracts the contour of the target object 4 by extracting the edge as a closed curve.

  In FIG. 4A, a region 40 is a region surrounded by representative points 41 to 48, and represents the contour of the target object 4 in the initial state. Here, the number and position of the representative points are arbitrarily given in the initial state, and are not limited thereto. The representative points 41 to 48 are moved in the directions of arrows 401 to 408, respectively, by obtaining points that minimize the energy function based on the energy function defined so as to become smaller as the target object 4 is approached. To do. Thereafter, the representative points 41 to 48 are stopped at the positions shown in FIG. 4B, and the region 40 is converged as the contour of the target object 4.

  Note that, in a general active contour model, the edge is determined based on a difference in pixel values at the boundary between the target object 4 and the background image. However, in the active contour model according to the first embodiment of the present invention, the edge is determined. The edge is determined based on the difference in the number of viewers.

Next, the energy function used in Embodiment 1 of the present invention will be described. Here, the contour in the active contour model is represented as a finite number of representative points (nodes) v _{i, j} connected by a curve. i represents a node number in the same frame, and j represents a frame number.

In the following, the energy function E _tube used in Embodiment 1 of the present invention is defined in Equation (1).

Here, α, β, γ, δ, and ε are weighting coefficients for each term. E _hin, and _{E hout} is, places that many of the viewers are paying attention, that is, the energy close to the popular position. That is, the energy converges to an area where there are many common parts in the selected area of the viewing history data or an area including the most common image. E _frame is energy for smoothing the camera work. E _dis and E _asp are energy that keeps the estimated area in an appropriate size and energy that approaches the aspect ratio (aspect ratio) of the screen.

Each term in the formula (1) is expressed as the following formula (2), formula (3), formula (4), formula (5), and formula (6). However, v _{i, j} = (x _{i, j} , y _{i, j} ) indicates the coordinates of the node i in the frame j. At this time, N is the total number of viewers of the content, P _IN and P _OUT are the numbers of pixels inside and outside the estimated area, respectively, Σ _IN and Σ _OUT are respectively inside the estimated area, and This is the total number of viewers at each outer pixel.

As shown in Expression (2), E _hin is energy relating to the inside of the estimation area, and becomes smaller as the number of viewers inside the estimation area increases or as the number of images common to the estimation area increases. Energy.

As shown in Expression (3), E _hout is energy related to the outside of the estimation area, and becomes smaller as the number of viewers outside the estimation area decreases or as the number of images common outside the estimation area decreases. Energy.

As shown in Expression (4), E _frame is energy that becomes smaller as the coordinate positions at the same node i in the preceding and _succeeding frames coincide. That is, E _frame is energy for smooth movement between frames. _{Alternatively} , E _frame is energy that becomes smaller as the cut-out positions of the target frames before and after the target frame are connected by a straight line, and the cut-out positions of the target frame between the front and rear approaches the straight line.

As shown in Expression (5), E _dis is energy that becomes smaller as the size approaches _xidea and _yidea of the target estimation region in the frame j. That is, E _dis is energy for unifying the size of the estimated region. Here, _xidea and _yidea represent the vertical and horizontal widths that the estimation area should approach, respectively.

As shown in Equation (6), E _asp is energy that decreases as the aspect ratio of the estimated region in frame j approaches the target aspect ratio h / w. That is, E _asp is energy for keeping the aspect ratio constant. Here, w and h represent the width and height of the displayed screen, respectively.

Further, x _size (j) and y _size (j) are represented by the following formulas (7) and (8).

In order to realize the region estimation processing according to the first embodiment of the present invention, E _hin needs to be defined at a minimum in the energy function E _tube . This is because the viewing history data is used. However, the content of Formula (2) is arbitrary.

Needless to say, the contents of Expression (3), Expression (4), Expression (5), and Expression (6) are not limited thereto. Further, in the energy function E _tube , other energy may be additionally defined in addition to the equation (1).

  FIG. 5 is a flowchart showing energy minimization processing according to the first exemplary embodiment of the present invention. Greedy algorithm is used for energy minimization. The Greedy Algorithm is well known as a publicly known technique, and therefore detailed description thereof is omitted here.

Here, it is assumed that the parameters in the equation (1) are N = 32, w = 180, h = 120, _xidea = 360, and _yidea = 240. Also, the search neighborhood for the Greedy Algorithm is 7 × 7 pixels. That is, in the energy minimization process according to the first exemplary embodiment of the present invention, the energy function value is calculated for each representative point at a point of 7 × 7 pixels around the representative point. However, these parameters and the number of pixels in the vicinity of the search are not limited to this.

  The region estimation unit 14 of the image generation device 1 performs the energy minimization process using the viewing history data as an input based on the energy function defined in Expression (1). First, the region estimation unit 14 inputs initial representative points (S21). Here, the initial representative points are two arbitrary points for each frame. For example, the upper left and lower right points of each frame may be used.

Next, the region estimation unit 14 calculates energy function values of neighboring pixels of each representative point (S22). Specifically, the region estimation unit 14, the representative point v _{i, j,} and calculates the energy function values at representative points v _i, near the _j pixels. This is performed for all frames and nodes.

Then, the region estimation unit 14 determines whether there is a neighboring pixel having an energy function value smaller than that of the representative point for each representative point v _{i, j} (S23). If it is determined that there is a neighboring pixel having an energy function value smaller than that of the representative point at any of the representative points v _{i, j} , the process proceeds to step S24. If it is determined that no neighboring pixel exists, the process is terminated. An area represented by the representative points is assumed as an estimated area.

If it is determined that there is a neighboring pixel having an energy function value smaller than that of the representative point, the region estimation unit 14 moves the corresponding representative point v _{i, j} to the neighboring pixel having the smallest energy function value. The representative point is v _{i, j} (S24). This is repeated until there is no corresponding representative point in step S23.

  In step S23, the process is repeated until there is no corresponding representative point. However, an upper limit value of the corresponding representative point may be set. Thereby, the convergence time is shortened.

Furthermore, in order to help convergence of the equation (1), it is desirable to change the parameters α, β, γ, δ, and ε according to the number of iterations r as shown in the equation (9). Thereby, convergence efficiency can be raised more. That is, while the number of iterations in the energy minimization process is less than 150, the parameters β and γ are set to 0, and the equation (1) is defined only by the equations (2), (5), and (6). Is an energy function. After the number of iterations reaches 150 or more, the parameters β and γ are set to other than 0, the definitions of the equations (3) and (4) are further added to the equation (1), and the parameters δ and ε Is changed to adjust the weight of the energy value defined in Equation (5) and Equation (6). Accordingly, although the value of the parameter α itself is not changed, as a result, the ratio in the expression (1) of the expression (2) is adjusted as compared with the case where the number of iterations is less than 150.

  The energy minimization process described above is not limited to the Greedy Algorithm. For example, a variational method or a dynamic programming method can be applied. However, the energy minimization processing in Embodiment 1 of the present invention is preferably the Greedy Algorithm as compared with the variational method and the dynamic programming method. For example, in the case of the variational method, calculation of an inverse matrix occurs for each iterative process in energy minimization, which is not optimal in terms of calculation amount and accuracy. Note that the variational method and the dynamic programming are well known as publicly known techniques, and thus detailed description thereof is omitted here.

  FIG. 6 is a diagram showing an example of the area selection screen 50 according to the first embodiment of the present invention. The area selection screen 50 is an interface provided by an application for cutting out a selection area that operates in the information terminals 3a, 3b, and 3c.

  The area selection screen 50 is displayed on the screens of the information terminals 3a, 3b, and 3c. The area selection screen 50 includes a downsampled original image 51 and a cutout image 53. The downsampled original image 51 is an area for displaying visual content distributed from the image generating device 1 and is displayed at a low resolution with respect to the original visual content. The downsampled original image 51 includes a cutout region 52. The position and size of the cutout area 52 can be changed by the operation of the viewer's mouse or the like. Therefore, the viewer can change the position and size of the cutout area 52 in the downsampled original image 51.

  The cutout image 53 is an area where the image surrounded by the cutout area 52 is enlarged and displayed to the same size as the downsampled original image 51. Therefore, the viewer can cut out the image after confirming the cutout region 52 selected by the viewer using the cutout image 53.

  FIG. 7 is a diagram showing a comparison result of application of the related art and the first embodiment of the present invention to moving image content. In this example, the moving image content shows a state in which one object (lion) exists in a frame and the object moves between frames. The comparison targets are frames 0, 30, 60, 90, and 120 in the moving image content. Each viewer operates the area selection screen 50 shown in FIG. 6 and arbitrarily selects the vicinity of the object as a selection area.

  FIG. 8 shows a group of images generated as a result of simply down-sampling the high-resolution moving image content that is the original image. FIG. 9 shows a group of images generated as estimated region contents as a result of applying the original image to the image generating apparatus 1 according to the first embodiment of the present invention. From this, it can be seen that FIG. 9 displays the object more clearly than FIG. In addition, this result can be similarly obtained for a plurality of objects in the target moving image content.

  As described above, by using the image generation apparatus 1 according to the first embodiment of the present invention, it is possible to estimate the optimum presentation area for the whole from a lot of history information collected from a large number of viewers. That is, by using collective intelligence, it is possible to estimate a region selected by a larger number of viewers among partial regions included in the original image.

  In addition, by generating aggregate histogram information of the number of viewers in the selected area from the viewer's operation history, it is possible to more accurately represent the viewer's attention location, and the optimal presentation area, that is, the satisfaction of the entire viewer It is possible to estimate a region where the average degree is the highest.

  Furthermore, by using the aggregate histogram information of the number of viewers in the selected region, the subjective quality of the entire viewer is increased by applying a dynamic contour model in which the density of the number of viewers in the selected region is defined as energy. An estimation area can be obtained.

  In particular, in the active contour model in the image generation method according to the first embodiment of the present invention, various constraint conditions are expressed by one-dimensional energy in order to estimate the optimum presentation area from the operation history of a plurality of viewers. The energy function was defined. In the energy function, by introducing the cumulative number of viewers in the selected region and the consistency of camera work, it is possible to generate a moving image with higher subjective quality than when the entire moving image is simply reduced. it can.

  In the above-described region estimation unit 14, the estimated region content is generated from the estimated region and stored in the estimated region content storage unit 24 in advance, but the present invention is not limited to this. For example, the region estimation unit 14 stores the estimation region itself separately, and when the estimated region content is distributed, the content distribution unit 11 generates the estimated region content from the estimated region and distributes it each time. May be.

Embodiment 2 of the Invention
In recent years, so-called “full browser” has been installed in mobile phones, so that WEB content originally created on the premise of browsing on a PC (personal computer) can be viewed on a mobile phone without destroying its original layout. became. However, the display of the mobile phone still has a low resolution, and the range that can be viewed at one time must be a part of the original WEB content cut out (trimmed). Thus, a great effort is required before a user (viewer) who uses a mobile phone can browse a portion desired by the user in the WEB content. In order to help this, a full browser of a mobile phone is often equipped with a display function of a reduced image that presents which part of the entire WEB content the part currently being viewed is. However, since it is difficult to grasp the details of each part of the WEB content from the reduced image, the problem is only partially solved.

  Therefore, Embodiment 2 of the present invention provides a target for WEB content for mobile phones as a modification of the area estimation processing in Embodiment 1 of the present invention. In other words, by using the viewing history of other users who have browsed the WEB content using a similar mobile phone, a particularly popular location can be quickly displayed to display the desired location on the mobile phone display. The labor can be reduced dramatically.

  FIG. 10 is a diagram showing an outline of an application example in the second embodiment of the present invention. In FIG. 10, the image generation apparatus 1 of FIGS. 1 and 2 is replaced with a carrier server 1a and a WEB server 1b, and the information terminals 3a, 3b, and 3c of FIG. 1 are replaced with a mobile phone terminal 31. The information terminals 3d, 3e, and 3f in FIG. 2 are replaced with the mobile phone terminal 33. Also, the content distribution unit 11 according to the first embodiment of the present invention is replaced with the content distribution unit 11a, and the estimated region content storage unit 24 according to the first embodiment of the present invention is replaced with the estimated region information storage unit 25. Is. In addition, in FIG. 10, the same code | symbol is attached | subjected to what has a function equivalent to FIG. 1 and FIG. 2, and description is abbreviate | omitted.

  The mobile phone terminal 31 is a mobile phone terminal that can communicate with the carrier server 1a via a network and includes a full browser 32. The full browser 32 can display the WEB content received from the carrier server 1a, and can enlarge and display a part of the WEB content. Further, the full browser 32 can transmit the position displayed in an enlarged manner to the carrier server 1a. As a result, it is possible to treat an enlarged area of the WEB content displayed on the full browser 32 as viewing history data for the viewer.

  Note that the mobile phone terminal 33 and the full browser 34 included in the mobile phone terminal 33 may have the same functions as the mobile phone terminal 31 and the full browser 32, and thus description thereof is omitted.

  The carrier server 1a can communicate with the mobile phone terminal 31, the mobile phone terminal 33, and the WEB server 1b via a network (not shown), and relays between the mobile phone terminal 31, the mobile phone terminal 33, and the WEB server 1b. It is a server to do. This is because communication by a mobile phone terminal including WEB browsing needs to go through a server of a mobile phone company (carrier) for convenience of billing and the like.

  The estimated area information storage unit 25 of the carrier server 1a stores estimated area information obtained by the area estimating unit 14. Here, the estimated area information is position information in the WEB content. For example, it may be layout information included in the WEB content.

  The content distribution unit 11a of the carrier server 1a relays a web content request from the mobile phone terminal 31 or the mobile phone terminal 33 to the WEB server 1b, and sends the WEB content from the WEB server 1b to the mobile phone terminal 31 or Transmit to the mobile phone terminal 33.

  The carrier server 1 a receives the viewing history data in the WEB content from the mobile phone terminal 31 or the mobile phone terminal 33 and stores it in the viewing history storage unit 22. In addition, the carrier server 1 a generates region estimation information by the aggregation processing unit 13 and the region estimation unit 14 in the same manner as the image generation device 1. Then, the region estimation unit 14 stores the estimated region information in the estimated region information storage unit 25. Further, when the estimated area content is requested from the mobile phone terminal 31 or the mobile phone terminal 33, the content distribution unit 11a adds the estimated area information to the WEB content and transmits it.

  The WEB server 1 b is a general WEB server that can communicate with the carrier server 1 a and includes the content storage unit 21. The content storage unit 21 stores WEB content for PC.

  Next, the flow of processing in Embodiment 2 of the present invention will be described. The schematic process of the application example of the second embodiment of the present invention is the same as that in FIG.

  First, the carrier server 1a collects information on the selected area of the viewer from the mobile phone terminal 31 (S11). Specifically, first, the mobile phone terminal 31 transmits a web content request to the carrier server 1a. Next, the carrier server 1a receives the web content request from the mobile phone terminal 31, and transmits the request to the web server 1b that holds the web content. Then, the WEB server 1b refers to the content storage unit 21, and transmits the WEB content to the carrier server 1a. Thereafter, the carrier server 1a receives the WEB content from the WEB server 1b and transmits it to the mobile phone terminal 31 that is the request source. Here, the mobile phone terminal 31 receives the WEB content from the carrier server 1 a and displays it on the full browser 32. The viewer uses the full browser 32 to enlarge and display a portion desired by the viewer on the WEB content. At this time, the full browser 32 transmits the enlarged display position in the WEB content as viewing history data to the carrier server 1a. Then, the carrier server 1 a receives the viewing history data in the WEB content from the mobile phone terminal 31 and stores it in the viewing history storage unit 22.

  Next, the carrier server 1 a performs generation of aggregate histogram information by the aggregation processing unit 13 (S 12), region estimation processing by the region estimation unit 14 (S 13), and stores the estimated region information in the estimation region information storage unit 25. . At this time, the energy function used in the region estimation process can be defined by, for example, Expression (2), Expression (3), Expression (5), and Expression (6). Furthermore, the accuracy of area estimation can be further improved by using the layout information included in the WEB content. Here, the layout information may be anything described in an HTML (HyperText Markup Language) tag or CSS (Cascading Style Sheets).

  Thereafter, the carrier server 1a distributes the estimated area content to the mobile phone terminal 33 (S14). Specifically, first, the carrier server 1a receives an estimated area content transmission request from the mobile phone terminal 33, and transmits a request for WEB content to the WEB server 1b. Thereafter, the carrier server 1a receives the WEB content from the WEB server 1b, acquires the estimated region information stored in the estimated region information storage unit 25 by the content distribution unit 11a, and stores the estimated region information in the WEB content. To the mobile phone terminal 33 that is the request source. Next, the mobile phone terminal 33 receives the WEB content and the estimated area information. Then, the full browser 34 interprets the position information of the estimated area information and enlarges and displays the corresponding position of the WEB content.

  Thereby, the viewer of the mobile phone terminal 33 can browse the position enlarged by the viewer of the mobile phone terminal 31 in advance without moving from the top of the WEB content to a desired location. For this reason, after the estimated area information is obtained by the carrier server 1a, it is possible to recommend the optimum presentation area to the viewer who browses the WEB content, so that the user can appropriately browse inside the WEB content. The effort to find a place can be greatly reduced.

  The full browser 32 described above may weight the viewing history data according to the display time of the enlarged display area. For example, a portion of the WEB content where the display time is short is likely to be a portion that has been passed when moving to a desired portion of the viewer. Therefore, it is assumed that the viewer is less interested and the weight is reduced and displayed. Since it is highly possible that a portion with a long time is a portion desired by the viewer, it is preferable that the weight is increased by regarding the viewer as having great interest. Thereby, estimated area information with higher accuracy can be obtained.

  As described above, in the second embodiment of the present invention, viewing history data of each WEB content is centrally managed on the carrier server 1a with respect to the WEB content distributed in a plurality of WEB servers all over the world. By doing so, it is possible to recommend the optimum presentation area in each WEB content without modifying each WEB server distributed in the world.

  The second embodiment of the present invention is not limited to a mobile phone terminal. For example, even an arbitrary information terminal such as a PC can be realized by storing viewing history data in a nearest server such as a communication provider and performing region estimation processing.

Other Embodiments of the Invention
In the first embodiment of the present invention, the visual content is not limited to moving image content. For example, even single image data can be applied.

  In the first embodiment of the present invention, the frame and the selection area are not limited to a rectangle. For example, in the case of a selection area having a shape other than a rectangle, this can be realized by modifying the definitions of Expressions (5) and (6). Further, in the case of an estimated area having a shape other than a rectangle, display on the screen may be performed while maintaining the shape of the estimated area.

  In the first and second embodiments of the present invention, viewing history data is input to the active contour model, but the present invention is not limited to viewing history. For example, a plurality of image data may be input.

  Note that each function of the image generation apparatus 1 according to the first embodiment of the present invention may be realized inside the information terminals 3a, 3b, and 3c. That is, the information terminals 3a, 3b, and 3c are the content distribution unit 11, the input reception unit 12, the totalization processing unit 13, the region estimation unit 14, the content storage unit 21, and the viewing history storage unit according to the first embodiment of the present invention. 22, any one or all of the histogram storage unit 23 and the estimated region content storage unit 24 may be provided.

  For example, a viewer who uses the information terminal 3a selects an area that he / she likes for visual content stored inside the information terminal 3a, and the information terminal 3a performs area estimation based on the selected area. It can be carried out. Thereafter, the viewer can view the estimated area content stored in the information terminal 3a with respect to the visual content. Thereby, the viewer who uses the information terminal 3a can view the estimated area content that better matches his / her preference for the visual content.

  Furthermore, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention described above.

It is a figure explaining the operation | movement which collects viewing history in the image generation apparatus 1 concerning Embodiment 1 of this invention. It is a figure explaining the operation | movement which produces | generates an estimation area | region content from viewing history in the image generation apparatus 1 concerning Embodiment 1 of this invention. It is a flowchart figure which shows the schematic process of the example of application of Embodiment 1 of this invention. It is a figure which shows the example of application of an active contour model. It is a flowchart figure which shows the energy minimization process concerning Embodiment 1 of this invention. It is a figure which shows the example of the area | region selection screen concerning Embodiment 1 of this invention. It is a figure which shows the example of the histogram of viewing history. It is a figure which shows the image group produced | generated as a result of applying a prior art (downsampling) to moving image content. It is a figure which shows the image group produced | generated as an estimation area content as a result of applying the image generation apparatus 1 concerning Embodiment 1 of this invention to moving image content. It is a figure which shows the outline of the example of application in Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image production | generation apparatus 1a Carrier server 1b WEB server 11 Content delivery part 11a Content delivery part 12 Input reception part 13 Total processing part 14 Area estimation part 15 Cutout process part 21 Content storage part 22 Viewing history storage part 23 Histogram storage part 24 Estimation area | region Content storage unit 25 Estimated area information storage unit 3a, 3b, 3c, 3d, 3e, 3f Information terminal 31 Mobile phone terminal 32 Full browser 33 Mobile phone terminal 34 Full browser 4 Target object 40 Area
41 to 48 representative points 401 to 408 arrow 40 initial region 41 representative point 42 convergence energy 43 representative point 44 convergence energy 45 convergence region 46 representative point 47 representative point 50 region selection screen 51 downsampled original image 52 cutout region 53 cutout image v _{i, j} representative point (node) i node number j frame number E _tube energy function α, β, γ, δ, ε weighting factors E _hin , E _hout , E _frame , E _dis , E _asp energy N Viewer of the content the number of pixels inside the whole number P _iN estimation region of P _OUT viewership on the inside of each pixel outside the number of pixels sigma _iN estimation region of the estimated area total sigma _OUT outside the estimated region of the audience at each pixel Total _xidea , _yidea target estimation area Size w Displayed screen width h Displayed screen height

Claims (26)

An image generation device that generates one image from a plurality of input images,
Estimating from the plurality of input images to minimize the energy function using an active contour model in which an energy function including a narrowing-down energy that narrows down a region so as to include the most common image from the plurality of input images is used. An image generation apparatus including an area estimation unit for obtaining an area. The focused energy is
Internal energy with respect to the interior of the estimated region;
External energy relating to the outside of the estimation region,
The internal energy is
The more the images of the estimated area are in common, the smaller the energy,
The external energy is
The image generating apparatus according to claim 1, wherein the smaller the image common to the outside of the estimation area, the smaller the energy. The energy function is
The image generation apparatus according to claim 2, further comprising size adjustment energy for bringing the size of the estimated area close to a target value. The input image is a part of the original image cut out,
The energy function is
The image generation apparatus according to claim 3, further comprising inter-image adjustment energy that decreases as the positions cut out between the original images match. The plurality of original images is time-series data composed of a plurality of frames,
The inter-image adjustment energy is
The image generation apparatus according to claim 4, wherein the cutting positions of the front and rear frames are connected by a straight line, and the energy decreases as the cutting position of the frame between the front and rear approaches the straight line. The estimated area is rectangular;
The energy function is
The image generation apparatus according to claim 4, further comprising an aspect ratio adjustment energy that maintains a constant aspect ratio of the estimation area. The plurality of input images are:
The image generation apparatus according to claim 1, wherein the image generation apparatus is an image obtained by cutting out a part or all of one or a plurality of images.   The image generation apparatus according to claim 1, wherein the plurality of input images are viewing histories that are a set of regions selected by a plurality of viewers viewing the original image. A viewing history storage unit for storing the viewing history;
The image generation apparatus according to claim 8, further comprising: a totalization processing unit that generates a histogram by counting the number of viewers for each region with reference to the viewing history storage unit. The region estimation unit
Minimizing the energy function by a greedy method;
The image generating apparatus according to claim 6, wherein a region that minimizes the energy function other than the external energy and the inter-image adjustment energy is obtained when the number of repetitions of the greedy method is a predetermined number or less.   The image generation apparatus according to claim 9, further comprising an input receiving unit that receives an area selected by the viewer as an input image and stores the input image in the viewing history storage unit. The active contour model is
An energy function further including energy related to layout information of the original image is defined,
The region estimation unit
The image generation apparatus according to claim 1, wherein an estimated area is obtained using the dynamic contour model.   The information terminal which displays the said original image, discriminate | determines the area | region selected by the said viewer, and transmits the information of the said area | region to the image generation apparatus of any one of Claim 1 thru | or 12. An image generation method for generating one image from a plurality of input images,
Estimating from the plurality of input images to minimize the energy function using an active contour model in which an energy function including a narrowing-down energy that narrows down a region so as to include the most common image from the plurality of input images is used. An image generation method including an area estimation step for obtaining an area. The focused energy is
Internal energy with respect to the interior of the estimated region;
External energy relating to the outside of the estimation region,
The internal energy is
The more the images of the estimated area are in common, the smaller the energy,
The external energy is
The image generation method according to claim 14, wherein the smaller the image common to the outside of the estimation area, the smaller the energy. The energy function is
The image generation method according to claim 15, further comprising size adjustment energy for bringing the size of the estimated region closer to a target value. The input image is obtained by cutting out a part of the image area of the original image,
The energy function is
The image generation method according to claim 16, further comprising inter-image adjustment energy that decreases as the positions cut out between the original images match. The plurality of original images is time-series data composed of a plurality of frames,
The inter-image adjustment energy is
The image generation method according to claim 17, wherein the cutout positions of the preceding and following frames are connected by a straight line, and the energy decreases as the cutout position of the frame between the front and back approaches the straight line. The estimated area is rectangular;
The energy function is
The image generation method according to claim 17 or 18, further comprising aspect ratio adjustment energy for maintaining a constant aspect ratio of the estimation area. The plurality of input images are:
The image generation method according to any one of claims 14 to 19, which is an image obtained by cutting out part or all of one or a plurality of images.   21. The image generation method according to claim 14, wherein the plurality of input images are viewing histories that are collections of regions selected by a plurality of viewers viewing the original image.   The image generation method according to claim 21, further comprising: an aggregation processing step of generating a histogram in which the number of viewers is aggregated for each area from the viewing history. The region estimation step includes:
Minimizing the energy function by a greedy method;
The image generation method according to claim 19, wherein a region that minimizes the energy function other than the external energy and the inter-image adjustment energy is obtained when the number of repetitions of the greedy method is a predetermined number or less.   24. The image generation method according to claim 22 or 23, further comprising an input receiving step of receiving an area selected by the viewer as an input image and storing it in the viewing history storage unit. The active contour model is
An energy function further including energy related to layout information of the original image is defined,
The region estimation step includes:
The image generation method according to claim 14, wherein an estimated area is obtained using the dynamic contour model.   A program for causing a computer to execute the processing according to any one of claims 14 to 25.