JP4746568B2 - Information providing apparatus, information providing method, and program - Google Patents

Description

  The present invention relates to an information providing apparatus, an information providing method, and a program for providing information in response to a request from a user.

Sales promotion activities using, for example, banner advertisements are widely performed, which are services that display information related to images displayed on the screens of terminal devices connected via a communication network such as the Internet. For example, the banner advertisement is provided in a part of a web page, and when clicked by the user, another website linked to the banner advertisement is displayed. And the technique which enables it to order goods directly via such a banner advertisement is proposed (for example, patent document 1).
JP 2003-331189 A

  However, in the technique of Patent Document 1, if a banner for ordering a product that the user desires to purchase is not displayed on the display screen of the user's terminal device, the user eventually reaches the sales page desired to purchase. There is a problem that it takes a long time. For the user, for example, it is not information related to a target such as a leading actor in a moving image displayed on a display screen, but is related to a background object such as a specific mountain or a building. Information may be rather useful.

  SUMMARY An advantage of some aspects of the invention is that it provides an information providing apparatus, an information providing method, and a program that allow a user to display information about a background object in a moving image.

  (1) An information providing device that provides information in response to a request from a user, and acquires moving image information indicating a moving image composed of a plurality of whole images (image frames) from inside or outside the information providing device. An image information acquisition unit; an optical flow analysis unit that analyzes an optical flow of each pixel of the entire image; and a motion that indicates the movement of each pixel with respect to each pixel based on an analysis result of the optical flow analysis unit A labeling processing unit that associates a vector with each of the pixels, a background image extraction unit that extracts a background image that is a partial image constituting a part of the entire image based on the motion vector, and at least one background image. Based on a background object specifying unit for specifying a background object and an association for associating the background object with the video information And related information associated with the specific background object while the specific background object in the moving image transmitted in response to a request from the user terminal is displayed on the display screen of the user terminal. And a matching unit to be displayed on the user terminal.

  According to the invention of (1), the information providing apparatus can analyze the optical flow of each pixel, generate a motion vector indicating the motion of each pixel, and associate the motion vector with each pixel. Then, the information providing apparatus can extract a background image based on the motion vector associated with each pixel, and further specify a background object. For example, when only a part of a background object that is a large mountain is displayed in each entire image, the background object may not be specified by only one background image. In this regard, since the information providing apparatus has the background object specifying means, even if it is not possible to specify the background object with one background image, the information providing apparatus includes a background image included in two or more entire images (image frames). Based on this, the background object can be identified. Then, the information providing device displays the related information associated with the specific background object while the specific background object is displayed on the display screen of the user terminal in the video transmitted in response to the request from the user terminal. It can be transmitted to the user terminal and displayed. As described above, the information providing apparatus can cause the user to display information on the background object in the entire image (image frame) to be transmitted.

  (2) The background image extraction unit is configured to determine that an image formed by the pixels having a relatively large motion vector among the plurality of partial images is the background image. (2) The information providing apparatus according to (1).

  According to the configuration described in (2), an image formed by pixels having a relatively large motion vector can be extracted as a background image. In many cases, the target of the leading actor is continuously located in the center of the entire image, and the background image tends to move more than the target. According to the configuration described in (2), it is possible to efficiently extract a background image using such a tendency.

  (3) The background image extraction unit is configured to determine that an image including the pixel corresponding to the motion vector substantially the same as the motion vector of the pixel in the vicinity of at least one corner of the entire image is a background image. The information providing apparatus according to any one of (1) and (2), wherein

  According to the configuration described in (3), it is possible to extract the background image based on the movement of the pixels near the corner having a high probability of being the background image. The target is rarely positioned near the corner of the entire image, whereas the background image tends to be positioned near the corner of the entire image. According to the configuration described in (3), it is possible to extract a background image with high certainty using such a tendency.

  (4) The matching unit is configured to cause the user terminal to display sightseeing spot information associated with a specific background object as the related information. (1) to (3) The information provision apparatus in any one.

  According to the configuration described in (4), for example, the information providing apparatus can cause the user terminal to display tourist spot information regarding a background object in an entire image (image frame) transmitted in a computer network.

  (5) Any one of (1) to (3), wherein the matching unit is configured to display an advertisement associated with a specific background object on the user terminal as the related information. The information providing device described in 1.

  According to the configuration described in (5), for example, the information providing apparatus can cause the user terminal to display an advertisement related to the background object in the entire image (image frame) transmitted in the computer network.

  (6) Image information in which an information providing device that provides information in response to a request from a user acquires moving image information indicating a moving image composed of a plurality of whole images (image frames) from inside or outside the information providing device. An acquisition step; an optical flow analysis step for analyzing an optical flow of each pixel of the entire image; and a motion vector indicating a motion of each pixel with respect to each pixel based on an analysis result in the optical flow analysis step. A labeling process step associated with each of the pixels, a background image extraction step of extracting a background image which is a partial image constituting a part of the whole image based on the motion vector, and based on at least one of the background images A background object specifying step for specifying a background object, and the background object An association step for associating the video information with the video information, and while the specific background object in the video transmitted in response to a request from the user terminal is displayed on the display screen of the user terminal, the specific background A matching step of transmitting related information associated with the object to the user terminal for display.

  According to the invention described in (6), similarly to the invention described in (1), it is possible to display information related to the background object in the transmitted whole image (image frame) on the user terminal.

  (7) A program for causing a computer to function as an information providing apparatus that provides information in response to a request from a user, wherein the information providing apparatus indicates moving picture information indicating a moving picture composed of a plurality of whole images (image frames). Image information acquisition step acquired from inside or outside, optical flow analysis step of analyzing the optical flow of each pixel of the entire image, and for each pixel based on the analysis result in the optical flow analysis step A labeling processing step that associates a motion vector indicating the motion of the pixel with each of the pixels, and a background image extraction step of extracting a background image that is a partial image constituting a part of the entire image based on the motion vector; A background object is identified based on at least one of the background images A scene object specifying step, an associating step of associating the background object with the moving image information, and the specific background object in the moving image transmitted in response to a request from the user terminal is displayed on the display screen of the user terminal And a matching step of causing the user terminal to transmit and display related information associated with the specific background object.

  According to the invention described in (7), similarly to the invention described in (1), it is possible to display information related to the background object in the transmitted whole image (image frame) on the user terminal.

Embodiment of the present invention
FIG. 1 is a diagram showing a schematic configuration and the like of an information providing apparatus 10 (hereinafter referred to as “apparatus 10”) according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing the user terminal 5 and the like. FIG. 3 is a diagram showing an example of a moving image displayed on the user terminal 5 or the like. As illustrated in FIG. 1, the device 10 includes an image dictionary generation device 20, a search device 40, and a background object matching device 100. The device 10 can communicate with a plurality of user terminals 5, 5, user terminals 7, and the like via a communication line such as the Internet line 9.

  The image dictionary generation device 20 can be configured by a server that generates a search dictionary that associates a partial image with content such as a moving image including the partial image. The search key of the image dictionary generation device 20 is not a character string but a partial image. Hereinafter, the partial image input for searching is referred to as “search image”. Thus, the search image is not a whole image but a partial image. For example, an image including all of the road 202, the sparrow 204, the tree 206, and Mt. Fuji 210 illustrated in FIG. 3A is referred to as an entire image (image frame). Each image such as Mt. Fuji 210 is called a partial image. A plurality of whole images (image frames) constitute a content such as a moving image. In FIG. 3, the sparrow 204 is the leading role, and is always located near the center of the entire image in the moving image. On the other hand, the road 202, the tree 206, and Mt. Fuji 210 are background images.

  In response to the search image received from the user terminal 5 or the like, the search device 40 extracts a link (URL or address) to the content from the image search DB 34 (see FIG. 4) of the image dictionary generation device 20, and the user terminal 5. It can be configured by a server for transmitting to the network. Note that the search device 40 may transmit related information in addition to the link to the content. The background object matching apparatus 100 displays the tourist information associated with a specific background image included in the moving image information transmitted to the user terminal 5 or the like on the user terminal 5 or the like only while the background image is displayed. It is the composition.

  As shown in FIG. 2A, the user terminal 5 can be configured by a mobile phone, for example, and includes a liquid crystal screen 5a, an operation button group 5b, an enter button 5c, and a move button 5d. In addition to the general function, the movement button 5d is also used to move the cursor to a specific partial image in the entire image displayed on the liquid crystal screen 5a. In addition to a general function, the determination button 5c is also used to determine a partial image on which the cursor is moved by the movement button 5d as a search image. As shown in FIG. 2B, the user terminal 7 can be configured by, for example, a notebook personal computer, and includes a liquid crystal screen 7a, an operation button group 7b, and a mouse 7c. Various information is displayed on the liquid crystal screens 5a and 7a. For example, moving images shown in FIGS. 3A to 3C are also displayed. The images of scenes 1 to 3 shown in FIG. 3 are continuously displayed to form a series of moving images.

[Image Dictionary Generation Device 20]
FIG. 4 is a schematic block diagram illustrating the configuration of the image dictionary generation device 20.
As illustrated in FIG. 4, the image dictionary generation device 20 includes a content acquisition unit 21. The content acquisition unit 21 is a configuration for acquiring (receiving) various contents from the outside of the device 10 via the Internet line 9 and functions as an image information acquisition unit. The content acquisition unit 21 includes a computer program and hardware such as a CPU (Central Processing Unit) that controls the apparatus 10 based on the program, and includes an input unit and an output unit. Each unit described below is similarly configured by a computer program and hardware, and has an input unit and an output unit.

  The content includes text data and image data. The image data includes moving image data. The image dictionary generation device 20 includes a content clustering unit 22. The content clustering unit 22 is configured to classify content acquired from the outside into clusters (subsets) and store them in a content DB (database) 30 in a state where the classification method is machine-learned by so-called supervised learning. That is, the content clustering unit 22 is an automatic classifier. Each content has a unique address (or URL). The content clustering unit 22 described above functions as an image information storage unit. The cluster is, for example, “a set of mountain climbing related contents”, “a set of travel related contents”, “a set of music related contents”, or the like.

  As illustrated in FIG. 4, the image dictionary generation device 20 includes a partial image acquisition unit 23. The partial image acquisition unit 23 has a configuration for acquiring, for example, an image of the sparrow 204 in FIG. 3A (hereinafter, also simply referred to as “sparrow 204”. The same applies to other partial images). For example, the partial image acquisition unit 23 automatically extracts a partial image from content stored in the content DB 30. For example, the partial image acquisition unit 23 displays the image of FIG. 3A on the user terminal 7, and when the user of the user terminal 7 places the cursor on the sparrow 204 with the mouse 7 c and clicks, the partial image acquisition unit 23 transmits via the Internet line 9. The whole image and the partial image (for example, sparrow 204) included in the whole image may be received.

  As illustrated in FIG. 4, the image dictionary generation device 20 includes a partial image clustering unit 24. The partial image clustering unit 24 is configured to perform machine learning of the classification method by so-called supervised learning, classify content acquired from the outside into clusters (subsets), and store the contents in the partial image DB 32. That is, the partial image clustering unit 24 is an automatic classifier and functions as a partial image storage unit. The cluster is, for example, “a collection of mountains”, “a collection of tourist attractions”, “a collection of musical instruments”, or the like.

  As illustrated in FIG. 4, the image dictionary generation device 20 includes a feature adding unit 25. The feature adding unit 25 is a configuration for analyzing the features of the acquired partial image and generating feature information, and functions as a feature information generating unit. The feature adding unit 25 extracts, for example, the contour of the partial image and the number of used colors as features. The contour is extracted by edge processing. With these characteristics, even when the viewing angle of the same partial image (for example, Mt. Fuji) is different, it can be determined that the partial images are the same.

  As illustrated in FIG. 4, the image dictionary generation device 20 includes a tag adding unit 26. The tag providing unit 26 is configured to record the feature information extracted from the partial image as a tag (or index) in association with a link between the tag and the content stored in the content DB 30 in the image table. The image table is stored in the image search DB 34.

  FIG. 5 is a diagram illustrating an example of an image table. As shown in FIG. 5, in the image table, a moving image ID (also referred to as an image ID or content ID), feature information, clustering ID, content address, in-scene position, in-image (image frame) position (coordinates), motion A vector and an advertisement ID are associated with each other. “Movie ID” is, for example, identification information of the entire movie shown in FIG.

  In the present embodiment, feature information is used as a search key. “Feature information” is information indicating the features of the partial image. The feature information indicates, for example, the outline of Mt. Fuji and the number of colors used in the Mt. Fuji (5 colors in the example of FIG. 5). “Clustering ID” is identification information of a cluster (subset) to which each partial image belongs. “Link to content” is, for example, the address or URL of the entire moving image of FIG. “In-scene position” is, for example, information indicating the temporal position of each scene in the entire moving image of FIG. “In-image position (coordinates)” is, for example, information indicating coordinates at which the central portion of Mt. Fuji 210 is located in the entire image of FIG. The “motion vector” is information indicating the motion of the partial image in the entire image. “Advertisement ID” is identification information of a banner advertisement corresponding to each partial image. Unlike this embodiment, “sightseeing spot information ID” indicating sightseeing spot information (tourist spot information) may be entered in the image table instead of “advertisement ID”.

  FIG. 6 is a diagram showing an advertisement table and a background object table stored in the image search DB 34. In the advertisement table, advertisement IDs, places such as Mt. Fuji, and tags are stored in association with each other. The “location” and “tag” in the advertisement table are text information that can be freely specified by the advertiser. “Location” is also a type of tag, and “tag” is a tag other than “location”. For example, for a specific advertisement ID, “place” is “Mt. Fuji”, and “tag” is “3776 meters”, “Japan's best”, “national park”, “sunrise”. As described above, at least one “location” and “tag” are added to one advertisement ID.

  In the background object table, the feature information and the location and tag assigned by the user (video distributor) are stored in association with each other. “Place” and “tag” in the background object table are text information that can be freely given to a partial image in a moving image by a user (moving image distributor). “Location” is also a type of tag, and “tag” is a tag other than “location”. For example, for “Mt. Fuji” as a partial image, “place” is “Mt. Fuji”, and “tag” is “3776 meters”, “Japan's best”, “national park”, “sunrise”. In this way, at least one “location” and “tag” are added to one partial image.

  Thereby, for example, when the advertiser designates a specific advertisement ID and a tag “3776 meters”, the advertisement ID and the feature information can be associated with each other using the tag as a key, and an image table can be formed. Further, when the advertiser designates a specific advertisement ID and a tag “Japan's best”, the advertisement ID and the feature information can be associated with each other using the tag as a key. In this way, a specific advertisement ID and a specific background object can be associated with each other using various tags specified by the advertiser. In this way, the image table of FIG. 5 can be generated.

  FIG. 7 is a conceptual diagram illustrating an example of a partial image dictionary. The partial image dictionary is a part of the image table described above. As shown in FIG. 7, the partial image dictionary is configured by associating at least one content address with one piece of feature information. The content may be a video. The image search DB 34 is commonly used by each device included in the device 10.

[Searching device 40]
The device 10 refers to the image search DB 34 based on the search image received from the user terminal 5 or the like by the search device 40 and returns the URL of the content to the user terminal 5 or the like. The search device 40 may transmit information related to the search image as text information in addition to the content URL.

  FIG. 8 is a block diagram illustrating a schematic configuration of the search device 40. As illustrated in FIG. 8A, the search device 40 includes a query reception unit 41, a search processing unit 42, a search result generation unit 43, a search result transmission unit 44, and a search log recording unit 45. The query receiving unit 41 is configured to receive a search image from the user terminal 5 or the like. The search processing unit 42 is configured to search the image search DB 34 for tag IDs of partial images having characteristics that correlate with the search image. The search result generation unit 43 is configured to generate a search result for transmission to the user terminal 5 or the like. The search result transmission unit 44 is configured to transmit the search result to the user terminal 5 and the like, and functions as a related information transmission unit. The search log recording unit 45 is configured to generate a record in which each partial image is searched.

  As illustrated in FIG. 8B, the search processing unit 42 includes an image acquisition unit 42a, a feature detection unit 42b, an image search unit 42c, and a related information extraction unit 42d. The image acquisition unit 42a functions as a partial image receiving unit that receives a partial image as a search key. The feature detection unit 42b has a configuration for detecting features such as the contour and the number of colors of the received partial image. The image search unit 42c functions as an identification information extraction unit that extracts a tag ID. The related information extraction unit 42d is configured to extract related information related to the partial image searched based on the search image from the content DB 30. For example, when the apparatus 10 receives a photo of a dish from the user terminal 5 as a search image, the related information extraction unit 42d extracts the recipe for the dish from the content DB 30. The related information extracted by the related information extraction unit 42d is transmitted to the user terminal 5 or the like by the search result transmission unit 44 described above.

  FIG. 9 is a diagram illustrating an example of a search result displayed on the liquid crystal screen 5a of the user terminal 5 based on the data transmitted by the search result transmission unit 44 (see FIG. 8A). As shown in FIG. 9, for example, if the search image is “A type car”, at least one address of the home page where the A type car is displayed is displayed together with the image of the A type car. Yes. At this time, related information such as the performance such as the displacement of the A type automobile may be displayed at the same time.

  As will be described later, when the user of the user terminal 5 clicks on a specific address, the device 10 extracts and transmits a moving image from the content DB 30. At this time, the background image included in the moving image is displayed. A corresponding banner advertisement is also transmitted and displayed on the terminal 5 or the like.

[Background Object Matching Device 100]
FIG. 10 is a block diagram showing a schematic configuration and the like of the background object matching device 100 (hereinafter referred to as “matching device 100”). 11 and 12 are diagrams illustrating an example of how a background image is extracted. 13 and 14 are diagrams showing an example of how the background object is specified.

  As shown in FIG. 10, the matching apparatus 100 includes an image acquisition unit 102, an image display unit 104, an optical flow analysis unit 106, a labeling processing unit 108, a background image extraction unit 110, a background object specification unit 112, an association unit 114, An information matching unit 116 and a related information DB 120 are included. The image acquisition unit 102 is configured to automatically acquire moving image information from, for example, the content DB 30 (see FIG. 4). Note that the image acquisition unit 102 may receive the moving image information from the outside of the device 10. The image acquisition unit 102 is an example of an image information acquisition unit.

  The image display unit 104 has a liquid crystal display screen 104a (hereinafter, referred to as “display screen 104a”) for displaying the moving image acquired by the image acquisition unit 102, and has a size of, for example, horizontal 1280 pixels and vertical 1024 pixels. It is. An image is displayed on the display screen 104a by changing the color and brightness of each pixel of the image display unit 104.

  The optical flow analysis unit 106 is a configuration for analyzing the optical flow of each pixel of the entire image, and is an example of an optical flow analysis unit. Here, the optical flow is a vector indicating in what direction and how far a certain point or figure in the image moves at the next moment. In other words, the optical flow is a velocity field at each point on the image, and the extraction method is disclosed in, for example, Japanese Patent Application Laid-Open No. 2006-98119.

  For example, when the entire image displayed on the display screen 104a of the apparatus 10 changes from the state shown in FIG. 11A as shown in FIG. 11B, the main target 205 is the display screen 104a of the image display unit 104. It continues to be located at the center of the center. In contrast, Mount Fuji 210 moves in the direction of arrow X1 on the display screen 104a. That is, the target 205 actually moves in the direction of the arrow X2, but the target 205 hardly moves on the display screen 104a. On the other hand, Mt. Fuji 210, which is a background image, moves larger than the target 205, which is the leading role. In FIGS. 11 (a) and 11 (b), Mt. Fuji 210 is not completely displayed, and a part thereof is displayed. In the present embodiment, an image other than the leading role is referred to as a background image, regardless of whether the entire outline appears or only a part appears. The distinction between the main character image and the background image is defined by the magnitude of a motion vector described later. Further, the background image is not always displayed on the display screen 104a. For this reason, the entire background image is referred to as a “background object”. For example, in the case of Mt. Fuji 210, the “麓” portion is a background image, but the background object is “Mt. Fuji”.

  The optical flow analysis unit 106 analyzes, for example, the optical flow when changing from the entire image (frame) in FIG. 11A to the next entire image in FIG. 11B for each pixel.

  The labeling processing unit 108 is a configuration for associating a motion vector indicating the motion of each pixel with each pixel based on the analysis result of the optical flow analysis unit 106, and is an example of a labeling processing unit.

  The labeling processing unit 108, for example, as shown in FIG. 12A, an image displayed on the display screen 104a, motion vectors a (a1, a2...), B (b1, b2) indicating the movement of each pixel. ...) Are associated with each pixel. In the image of FIG. 12A, the pixels at the start of movement are associated with each other so that they become the bases of the motion vectors a and b.

  The background image extraction unit 110 is configured to extract a background image based on the motion vector a1 associated with each pixel, and is an example of a background image extraction unit. The background image extraction unit 110 determines that an image formed by pixels associated with a motion vector a that is relatively larger than the motion vector b is a background image. Here, “relatively large” means that the size of the motion vector is twice or more.

  Note that unlike the present embodiment, the background image extraction unit 110 determines that an image including a pixel corresponding to a motion vector substantially the same as a pixel near at least one corner of the entire image is a background image. That is, the background image extraction unit 110 includes an image formed by pixels associated with a relatively large motion vector and an image including pixels corresponding to substantially the same motion vector as pixels near at least one corner of the entire image. May be determined to be a background image. Here, the “near corner” is, for example, one pixel at the corner (referred to as “corner pixel”) and a pixel located within a radius of 20 pixels from the corner pixel. “Substantially the same” means, for example, a match of 90 percent (%) or more.

  In FIG. 12B, only the motion vector a determined to correspond to the background image is displayed. The background image extraction unit 110 extracts a background image as shown in FIG. 12C by continuously forming a plurality of base portions of the motion vector a determined to correspond to the background image.

  The background object specifying unit 112 is a configuration for specifying a background object based on at least one background image, and is an example of a background object specifying unit. The background object specifying unit 112 detects the features (the number of outlines and colors) of the background image extracted by the background image extracting unit 110 described above, and to the extent specified in advance as the features of the partial images stored in the image search DB 34. If they match, identify the background object. The background object specifying unit 112 is configured to specify a background object when the feature of the background image and the feature information stored in the image search DB 34 coincide with each other to a predetermined degree, for example, 90% or more. For example, if the feature information of the entire image of Mt. Fuji 210 is stored in the image search DB 34 and the entire image of Mt. Fuji 210 is displayed in one background image, the background object is specified if the features match 90% or more. be able to.

  On the other hand, when only a part of Mt. Fuji 210 is displayed on the background image, the background object specifying unit 112 combines the plurality of background images to specify the background object. In the example of FIG. 13, a complete Mt. Fuji 210 is generated by combining two incomplete Mt. Fujis 210, and the Mt. Fuji 210 as a background object is specified. In the example of FIG. 14, FIGS. 14A to 14D are continuous image frames, and each background image is a portion indicated by a solid line. In this case, as shown in FIG. 13 (e), a complete Mt. Fuji 210 is generated by adding the difference portions (shaded portions in the figure) that do not appear in the previous image frame in consideration of the coordinates. Fuji mountain 210 as a background object can be specified

  The associating unit 114 is a configuration for associating the background object and the moving image stored in the content DB. Specifically, as shown in FIG. 5, the moving image ID including the background object is associated with the feature information of the background object.

  The related information matching unit 116 is related information associated with a specific background object while the specific background object exists in the moving image transmitted in response to the request from the user terminal 5 or the like. This is a configuration for displaying a banner advertisement on a user terminal, and is an example of a matching unit. Data for displaying the banner advertisement is stored in the advertisement DB 120 in association with the ID. If the above-described background object can be specified, the feature information can be specified, and the advertisement ID can be specified. And if advertisement ID can be specified, the banner advertisement corresponding to a specific background object can be extracted from advertisement DB120.

  When the related information matching unit 116 transmits the moving image information stored in the content DB 30 to the user terminal 5 or the like, the related information matching unit 116 transmits a banner advertisement corresponding to the background object included in the moving image and displays it on the terminal 5 or the like. It is the structure for making it. FIG. 15 is a diagram illustrating an example of a screen that the related information matching unit 116 displays on the user terminal 5 or the like. As shown in FIG. 15, for example, an advertisement banner “Mt. Fuji Hotel” is displayed on the display screen of the user terminal 5 or the like while the Mt. Fuji 210 as a background object exists. Unlike the present embodiment, as shown in FIG. 16, the related information matching unit 116 displays, for example, “Mt. Fuji” while the background object Mt. Fuji 210 exists on the display screen of the user terminal 5 or the like. Tourist spot information (information relating to sightseeing) may be displayed.

  The above is the configuration of the apparatus 10. As described above, the apparatus 10 can generate a motion vector indicating the motion of each pixel, extract a background image, and further specify a background object. For example, when only a part of a background object that is a large mountain is displayed in each entire image, the background object may not be specified by only one background image. In this regard, since the apparatus 10 includes the background object specifying unit, the background object can be specified based on two or more background images even when the background object cannot be specified with one background image. it can. And while the specific background object in the moving image transmitted in response to the request from the user terminal 5 or the like is displayed on the display screen of the terminal 5 or the like, the device 10 relates information related to the specific object image. (Banner advertisement and sightseeing spot information) can be transmitted to the user terminal 5 and displayed. In this way, information regarding the background object in the entire image (image frame) transmitted in the computer network can be displayed to the user.

  In addition, since the apparatus 10 is configured to determine that an image formed by pixels having a relatively large motion vector is a background image, the image formed by pixels having a relatively large motion vector. Can be extracted as a background image. The target is often located in the center of the entire image, and the background image tends to move more than the target. The apparatus 10 can extract a background image efficiently using such a tendency.

  In addition, when the apparatus 10 is configured to determine that an image including pixels corresponding to substantially the same motion vector as pixels near at least one corner of the entire image is a background image, the apparatus 10 is established as a background image. A background image can be extracted based on the movement of pixels in the vicinity of a high corner. The target is rarely positioned near the corner of the entire image, whereas the background image tends to be positioned near the corner of the entire image. The apparatus 10 can extract a background image with high certainty using such a tendency.

  Furthermore, since the apparatus 10 is configured to display advertisements or sightseeing information associated with a specific background object on the user terminal 5 or the like as related information, for example, an entire image (frame) transmitted in a computer network. ) Can be sent to the user for display of tourist information or advertisements related to background objects.

  Hereinafter, an operation example of the apparatus 10 will be described mainly using FIG. FIG. 17 is a schematic flowchart showing an operation example of the apparatus 10.

  First, the apparatus 10 receives moving image information from an external server through the Internet line 9 (step S1 in FIG. 17). This step S1 is an example of an image information receiving step. Then, the optical flow of each pixel is analyzed (step S2), and a motion vector is generated (step S3). Step S2 described above is an example of an optical flow analysis step, and step S3 is an example of a labeling process step.

  Subsequently, the apparatus 10 extracts a background image from the moving image (step S4) and specifies a background object (step S5). Step S4 described above is an example of a background image extraction step, and step S5 is an example of a background object specifying step. Subsequently, the apparatus 10 associates the background object with the moving image information (step S6). Step S6 is an example of an association step.

  Then, when transmitting a moving image in response to a request from the user terminal 5 or the like, the device 10 extracts a banner advertisement corresponding to the background object included in the moving image from the advertisement DB 120 (step ST7), and the user terminal together with the moving image. The advertisement banner corresponding to the background object is displayed only while the specific background object is displayed on the moving image (step S8). Steps S7 and S8 are an example of a matching step.

[Programs and computer-readable recording media]
A computer program for causing a computer to execute the image information acquisition step, the optical flow analysis step, the labeling processing step, the background image extraction step, the background object specifying step, the matching step, and the like of the above-described operation example can be used. Moreover, it can also be set as the computer-readable recording medium which recorded such a computer program etc.

[Hardware Configuration of Information Providing Device 10]
FIG. 18 is a diagram illustrating a hardware configuration example of the device 10. The apparatus 10 includes a CPU (Central Processing Unit) 1010 constituting each control unit (in a multiprocessor configuration, a plurality of CPUs such as a CPU 1012 may be added), a bus line 1005, a communication I / F 1040, a main memory 1050, a BIOS. (Basic Input Output System) 1060, a USB port 1090, an I / O controller 1070, input means such as a keyboard and mouse 1100, and a display device 1022.

  Storage means such as a tape drive 1072, a hard disk 1074, an optical disk drive 1076, and a semiconductor memory 1078 can be connected to the I / O controller 1070.

  The BIOS 1060 stores a boot program executed by the CPU 1010 when the apparatus 10 is started up, a program depending on the hardware of the apparatus 10, and the like.

  The hard disk 1074 constituting the storage unit stores various programs for the function of the device 10 and programs for executing the functions of the present invention, and various databases can be configured as necessary.

  As the optical disc drive 1076, for example, a DVD-ROM drive, a CD-ROM drive, a DVD-RAM drive, or a CD-RAM drive can be used. In this case, the optical disk 1077 corresponding to each drive is used. A program or data can be read from the optical disk 1077 by the optical disk drive 1076 and provided to the main memory 1050 or the hard disk 1074 via the I / O controller 1070. Similarly, the tape medium 1071 corresponding to the tape drive 1072 can be used mainly for backup.

  The program provided to the apparatus 10 is provided by being stored in a recording medium such as the hard disk 1074, the optical disk 1077, or a memory card. This program may be installed in the apparatus 10 and executed by being read from the recording medium via the I / O controller 1070 or downloaded via the communication I / F 1040.

The aforementioned program may be stored in an internal or external storage medium. Here, in addition to the hard disk 1074, the optical disk 1077, or the memory card, a magneto-optical recording medium such as an MD, or a tape medium 1071 can be used as a storage medium constituting the storage unit. Further, a storage device such as a hard disk 1074 or an optical disk library provided in a server system connected to a dedicated communication line or the Internet may be used as a recording medium, and the program may be provided to the information processing apparatus 1000 via the communication line.

  Here, the display device 1022 displays a screen for accepting data input to the server administrator, or displays a screen of a result of arithmetic processing by the device 10, and is a cathode ray tube display device (CRT) or a liquid crystal display device. (LCD) and other display devices.

  Here, the input means accepts input by the server administrator, and may be constituted by a keyboard, a mouse 1100, and the like.

  The communication I / F 1040 is a network adapter that enables the information processing apparatus 1000 to be connected to a terminal via a dedicated network or a public network. The communication I / F 1040 may include a modem, a cable modem, and an Ethernet (registered trademark) adapter.

  In the above example, the apparatus 10 has been mainly described. However, the functions described above can also be realized by installing a program in a computer and causing the computer to operate as the apparatus 10. Therefore, the functions realized by the information processing apparatus 1000 described as an embodiment in the present invention are executed by executing the above-described method by the computer, or by introducing the above-described program into the computer and executing it. Is also feasible.

  In the case where the apparatus 10 according to the embodiment of the present invention or the method used for controlling these apparatuses is realized by a program on a computer, a storage medium for storing the program is electronic, magnetic, optical, It can be an electromagnetic, infrared or semiconductor system (or apparatus or equipment) or a propagation medium. Examples of this storage medium include semiconductor or solid state storage devices, magnetic tape, removable computer readable media examples include semiconductor or solid state storage devices, magnetic tape, removable flexible disks, random Includes access memory (RAM), read only memory (ROM), rigid magnetic disk and optical disk. Examples of optical disks at the present time include compact disk read only memory (CD-ROM), compact disk read / write (CD-R / W) and DVD.

  As mentioned above, although embodiment of this invention was described, it only showed the specific example and does not specifically limit this invention. Further, the effects described in the embodiments of the present invention only list the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to the effects described in the embodiments of the present invention.

It is a figure which shows schematic structure etc. of the information provision apparatus which concerns on the 1st Embodiment of this invention. It is the schematic which shows a user terminal. It is a figure which shows an example of the moving image displayed on a user terminal. It is a schematic block diagram which shows the structure of an image dictionary production | generation apparatus. It is a figure which shows an example of an image table. It is a figure which shows an example of an advertisement table etc. It is a conceptual diagram which shows an example of a partial image dictionary. It is a block diagram which shows schematic structure of a search device. It is a figure which shows an example of the search result displayed on the liquid crystal screen of a user terminal. It is a figure which shows an example of a background object matching apparatus. It is a figure which shows an example of a mode that a background image is extracted. It is a figure which shows an example of a mode that a background image is extracted. It is a figure which shows an example of a mode that a background object is specified. It is a figure which shows an example of a mode that a background object is specified. It is a figure which shows the example of a screen which a related information matching part displays on a user terminal. It is a figure which shows the example of a screen which a related information matching part displays on a user terminal. It is a schematic flowchart which shows the operation example of an apparatus. It is a figure which shows the hardware structural example of an information provision apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Information provision apparatus 20 Image dictionary production | generation apparatus 21 Content acquisition part 22 Content clustering part 23 Partial image acquisition part 24 Partial image clustering part 25 Feature provision part 26 Tag provision part 40 Search apparatus 41 Query reception part 42 Search processing part 42a Image acquisition part 42b Feature detection unit 43c Image search unit 42d Related information extraction unit 43 Search result generation unit 44 Search result transmission unit 45 Search log recording unit 100 Background object matching generation device 102 Image acquisition unit 104 Image display unit 104
106 Optical Flow Analysis Unit 108 Labeling Processing Unit 110 Background Image Extraction Unit 112 Background Object Identification Unit 114 Association Unit 116 Related Information Matching Unit
120 Advertising DB

Claims (7)

An information providing device that provides information in response to a request from a user,
An image information acquisition unit that acquires moving image information indicating a moving image composed of a plurality of whole images (image frames) from the inside or the outside of the information providing device;
An optical flow analysis unit that analyzes the optical flow of each pixel of the entire image;
A labeling processing unit for associating each pixel with a motion vector indicating the motion of each pixel based on the analysis result of the optical flow analysis unit;
A background image extraction unit that extracts a background image that is a partial image constituting a part of the entire image based on the motion vector;
A background object specifying unit for specifying a background object based on at least one of the background images;
An association unit associating the background object with the video information;
While the specific background object in the video to be transmitted in response to a request from the user terminal is displayed on the display screen of the user terminal, related information associated with the specific background object is displayed on the user terminal. Matching part to be displayed on
An information providing apparatus comprising:   The background image extraction unit is configured to determine that an image formed by the pixels having a relatively large motion vector among the plurality of partial images is the background image. The information providing apparatus according to claim 1.   The background image extraction unit is configured to determine that an image including the pixel corresponding to the motion vector substantially the same as the motion vector of the pixel in the vicinity of at least one corner of the entire image is a background image. The information providing apparatus according to claim 1, wherein the information providing apparatus is an information providing apparatus.   The said matching part is the structure which displays the sightseeing spot information linked | related with the said specific background object as said related information on the said user terminal, The one in any one of Claim 1 thru | or 3 characterized by the above-mentioned. The information providing apparatus according to the description.   The said matching part is a structure which displays the advertisement linked | related with the said specific background object as the said related information on the said user terminal, The Claim 1 thru | or 3 characterized by the above-mentioned. Information providing device. An image information acquisition step in which an information providing device that provides information in response to a request from a user acquires moving image information indicating a moving image composed of a plurality of whole images (image frames) from inside or outside the information providing device; ,
An optical flow analyzing step in which the information providing apparatus analyzes an optical flow of each pixel of the entire image;
A labeling processing step in which the information providing apparatus associates a motion vector indicating a motion of each pixel with respect to each pixel based on an analysis result in the optical flow analysis step;
A background image extracting step in which the information providing apparatus extracts a background image that is a partial image constituting a part of the entire image based on the motion vector;
A background object specifying step in which the information providing device specifies a background object based on at least one of the background images;
The information providing apparatus associates the background object with the video information; and
While the specific background object in the moving image transmitted in response to the request from the user terminal is displayed on the display screen of the user terminal , the information providing apparatus is associated with the specific background object. A matching step of transmitting related information to the user terminal for display;
Information providing method characterized by comprising A program that causes a computer to function as an information providing device that provides information in response to a request from a user,
An image information acquisition step of acquiring moving image information indicating a moving image composed of a plurality of whole images (image frames) from inside or outside the information providing device;
An optical flow analysis step of analyzing the optical flow of each pixel of the entire image;
A labeling processing step for associating a motion vector indicating a motion of each pixel with respect to each pixel based on an analysis result in the optical flow analysis step;
A background image extraction step of extracting a background image which is a partial image constituting a part of the whole image based on the motion vector;
A background object specifying step of specifying a background object based on at least one of the background images;
Associating the background object with the video information;
While the specific background object in the video to be transmitted in response to a request from the user terminal is displayed on the display screen of the user terminal, related information associated with the specific background object is displayed on the user terminal. Matching steps to send and display to,
An information providing program for causing a computer to execute.

Images