JP2016046730A - Viewer attention information provision system, space-time marker setting device and program for the same, and information provision device and program for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a viewer with useful information related to part which is in a predetermined image distributed to the viewer and to which the viewer pays attention.SOLUTION: A viewer attention information provision system 10 detects part to which a viewer viewing temporally changing image pays attention, to provide the viewer with information corresponding to the attention part. The system 10 comprises: a space-time marker setting device 25 that identifies an existence range of a space-time volume V obtained by arranging, in time series, ranges of the same object existing in the image and sets a space-time marker obtained by adding accompanying information on the object to the space-time volume V; and an information provision device 27 that identifies, on the basis of a result of the detection of the attention part, a space-time marker of the object to which the viewer pays attention and provides the viewer with accompanying information in the space-time marker.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a viewer attention information providing system, a spatio-temporal marker setting device, and a program thereof for providing various information related to a viewer's attention portion watching an image that changes over time to viewers and other third parties. And an information providing apparatus and a program thereof.

  The present applicant has already applied for a patent on a degree-of-interest detection system for detecting the degree of attention of which part of the image the viewer is consciously viewing when a predetermined image is distributed to the viewer. (See Patent Document 1). This attention level detection system includes a configuration in which statistical data relating to which part of an image is noticed at which time among images viewed by a plurality of viewers is provided to the client side.

JP 2012-196279 A

  By the way, for various businesses related to distribution images, such as business operators who distribute images, advertisers and advertising agencies of products etc. appearing in the images, the statistical data of viewers is based on the audience rating and advertisement of the images. It can be usefully used for verification of effects. Therefore, it is necessary to further improve the accuracy and value of the statistical data. For that purpose, more viewers are asked to detect the attention level by the attention level detection system, and the statistical data of more viewers is collected. There is a need. However, from the viewer's standpoint, even if they provide their own information about the area of interest in the image to others, there is no merit to them unless they receive a reward for money, etc. It is difficult to collect more viewer statistics.

  The present invention has been devised by paying attention to such problems, and its purpose is to view data of a portion of interest in which the viewer is interested or interested in a predetermined image distributed to the viewer. Viewer attention information providing system, spatio-temporal marker setting device, and the same, which can provide viewers with useful information related to the part in the image that the viewer pays attention to, It is to provide a program, an information providing apparatus, and the program.

  In order to achieve the above object, the viewer attention information providing system according to the present invention mainly detects the attention portion of the viewer who is watching an image that changes over time, thereby obtaining information corresponding to the attention portion. A system provided to the viewer, which identifies the existence range of a spatio-temporal volume in which a range of the same object existing in the image is arranged in time series, and adds accompanying information of the object to the spatio-temporal volume And a spatio-temporal marker setting device for setting the spatio-temporal marker, and the spatio-temporal marker of the target object that the viewer is interested in based on the detection result of the target portion. The information providing device provided to the viewer is provided.

  According to the present invention, among objects existing in an image distributed to a viewer from a predetermined business operator, an object to which the viewer pays attention while comprehensively considering the time during which the viewer views the image. Is identified, and accompanying information such as product information regarding the object is provided to the viewer. For this reason, the viewer who desires to provide the accompanying information from the operator of the system according to the present invention needs to provide the operator with information on his or her attention portion existing in the distribution image instead. The operator can obtain a large amount of data of the attention portion in which the viewer is interested in the distribution image. As a result, it is possible to improve the reliability of time-series statistical data related to the degree of attention and attention information about each part in the distribution image, and to increase the evaluation and value of the statistical data.

The block diagram showing the whole viewer attention information provision system concerning this embodiment. The conceptual diagram for demonstrating a spatiotemporal volume and a spatiotemporal marker. The block diagram showing the structure of a spatiotemporal marker setting apparatus and an information provision apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing the overall configuration of the viewer attention information providing system according to the present embodiment. In this figure, the viewer attention information providing system 10 detects a portion of interest that each viewer pays attention to in a video (image) distributed to many viewers over time, and based on the detection result. This is a system that provides various types of information that are useful to each viewer and a business operator other than the viewer related to the video content.

  The viewer attention information providing system 10 includes a viewer-side computer unit 11 disposed on the viewer side, a provider-side computer unit 12 disposed on the provider side, and the computer units 11 and 12. An information processing unit 15 that is connected to each other via a network 14 such as the Internet, LAN, or broadcast and functions as a server for each of the computer units 11 and 12 is provided.

  The viewer-side computer unit 11 displays a computer main body 17 that performs information transmission / reception processing with the information processing unit 15 and internal information processing, and video and character information received from the information processing unit 15. A display device 18 such as a display, and a line-of-sight measurement device 20 that acquires the line-of-sight information of the viewer who is viewing the video displayed on the display device 18 are provided.

  Here, the line-of-sight measurement device 20 is a known device capable of measuring at least the position of the gazing point, which is a part where the viewer's line of sight is poured in the video displayed on the display device 18. In this line-of-sight measurement device 20, for each frame at a predetermined time in the video displayed on the display device 18, line-of-sight data is measured using the point-of-gaze coordinates representing the position of the point of interest in the frame as the line-of-sight information. The line-of-sight data is transmitted to the information processing unit 15 through the computer main body 17 in correspondence with the image information of the frame when the data is obtained. The configuration of the line-of-sight measurement device 20 is not an essential part of the present invention, and thus detailed description thereof is omitted.

  The provider side computer unit 12 displays a computer main body 22 that performs information transmission / reception processing with the information processing unit 15 and internal information processing, and video, character information, and the like received from the information processing unit 15. And a display device 23 such as a display.

  The information processing unit 15 is configured by one or a plurality of computers including an arithmetic processing unit such as a CPU and a storage device such as a memory and a hard disk, and a program for causing the computer to function as the following units is installed. Yes.

  The information processing unit 15 includes an image distribution device 24 that distributes video to the viewer-side computer unit 11, and one or more target products as one or more target objects that are displayed in the video distributed to each viewer. Obtained by using a spatio-temporal marker based on the spatio-temporal marker setting device 25 that embeds one or a plurality of spatio-temporal markers having predetermined accompanying information in video information that is a set of all image information It has a function as an information providing device 27 that provides various information to the computer main bodies 17 and 22 on the viewer side and the provider side.

  As shown in FIG. 2, the spatio-temporal marker has a time axis (t in the figure) on the space axis (xy axis in the figure) in the screen of the frame F which is the image data of each time constituting the video. In a three-dimensional space (hereinafter simply referred to as “time-space”) represented by the coordinates of three orthogonal axes to which the axis is added), a spatio-temporal volume V, which is an area occupied by one target product, is added to the accompanying information described later. Are related. The space-time volume V is a four-dimensional space obtained by adding a time axis to three-dimensional image information when a three-dimensional image is distributed by the image distribution device 24.

  As shown in FIG. 3, the spatiotemporal marker setting device 25 includes a range specifying means 29 for specifying the existence range of the spatiotemporal volume V of each target product in the spatiotemporal, and each spatiotemporal volume in which the existence range is specified. For V, the type is specified by the type specifying unit 30 for specifying the type of the target product, the information database 31 in which the product information as the accompanying information is stored for each type of the target product, and the type specifying unit 30. For each target product, a spatio-temporal marker generating unit 32 that extracts product information from the information database 31 to generate a spatio-temporal marker, and a spatio-temporal marker storage unit 33 that stores the spatio-temporal marker embedded in video information. It has.

  The range specifying unit 29 uses a known image processing technique such as an image recognition and tracking technique to specify the existence range of the spatiotemporal volume V of each target product in the spatiotemporal as follows. That is, whether or not the same image area as the image area of the target product existing in the frame F at a certain time exists in each frame F at times before and after that time, that is, the same in each frame F within a predetermined time. The presence / absence of the target product is determined by a technique such as image matching. As a result, if the target product is reflected somewhere in each frame F at a certain time, the in-screen coordinates (xy coordinates in FIG. 2) of the target product in each frame and the time t of each frame F Thus, the existence range of the target product in the space-time is specified three-dimensionally as a space-time volume.

  In the type specifying unit 30, for example, the corresponding product image stored in advance together with the type of product such as a product name such as XX (brand name) of XX company, and each time the existence range is specified by the range specifying unit 29. The image of the target product in the spatial volume V is compared, and the type of the product such as the product name of the target product is specified for each spatiotemporal volume V by a technique such as image matching. Note that the operator may directly input a product name or the like to the image of the target product in each spatiotemporal volume V. In this case, the type of the product and the spatiotemporal volume V are stored in association with each other.

  In the information database 31, for each type of target product, for example, advertisement information from the manufacturer or seller of the product, company information of those people, related information such as URL of the homepage, etc. Is remembered as

  In the spatio-temporal marker generating means 32, product information corresponding to the type of the target product of each spatio-temporal volume V specified by the type specifying means 30 is extracted from the information database 31, and the product information is converted into the corresponding spatio-temporal volume. By adding in association with V, a spatiotemporal marker for each spatiotemporal volume V of each target product is generated.

  In the spatiotemporal marker storage means 33, each spatiotemporal marker generated by the spatiotemporal marker generation means 32 is stored after being embedded in the video information.

  As shown in FIG. 3, the information providing device 27 extracts the product information using the spatiotemporal marker of the spatiotemporal marker setting device 25 based on the line-of-sight data transmitted from the viewer-side computer unit 11. Accompanying information extracting means 35, accompanying information providing means 36 for providing accompanying information extracted by the accompanying information extracting means 35 to the viewer-side computer unit 11, and line-of-sight data acquired from each viewer-side computer unit 11 in the system Are integrated and aggregated to perform predetermined statistical processing, and use the spatiotemporal marker of the spatiotemporal marker setting device 25 to obtain data relating to the attention level of the product information of each spatiotemporal marker for all the aggregated viewers Information statistical management means 38 and statistical information providing means 39 for providing the data to the business entity computer unit 12 are provided.

  The accompanying information extracting means 35 includes an individual totaling processing unit 42 that totals line-of-sight data for each viewer and performs a predetermined statistical process, and a spatio-temporal marker setting based on the totaling result in the individual totaling processing unit 42 A target marker that identifies a spatiotemporal marker that the target viewer is paying attention from among the spatiotemporal markers stored in the device 25 and extracts product information provided to the viewer from the spatiotemporal marker And a specifying unit 43.

  In the individual totalization processing unit 42, the following likelihood distribution data is created for each viewer for each viewer who has viewed the video distributed to each viewer-side computer unit 11. That is, the gaze point determined to have a high degree of attention is voted for each frame F based on the line-of-sight data obtained over time for each viewer, and time-series likelihood distribution data that becomes a probability distribution of the degree of attention is generated in the frame F. Created every time. Note that, in determining the gazing point and creating the likelihood distribution data, for example, a known method described in Japanese Patent Application Laid-Open No. 2012-196279 is used. As long as the gaze rate and the like can be determined and calculated, various methods can be employed. Since this method is not an essential part of the present invention, a detailed description thereof will be omitted.

  In the attention marker specifying unit 43, for each viewer, the time of each target product is calculated from the time series data (time series likelihood distribution data) of the attention degree in each part in the video obtained by the individual totalization processing unit 42. The degree of attention is aggregated for each spatial volume V, and a spatio-temporal volume V having an attention degree equal to or greater than a predetermined threshold set in advance is specified from each spatio-temporal volume V. Are extracted as self-focused product information that is of interest to the viewer.

  In the accompanying information providing means 36, the self-attention product information extracted by the attention marker specifying unit 43 is transmitted to the viewer-side computer unit 11 held by the corresponding viewer. The self-interested product information received by the viewer-side computer unit 11 can be recognized by the viewer through the display device 18 as information accompanying the target portion of the video viewed by the viewer. For example, when the video content is a drama, when a viewer looks at the performer's clothes appearing in the drama for a certain amount of time, the product name related to the clothes is displayed at a predetermined timing. In addition, merchandise information such as a store handling the merchandise, price information, and popularity is displayed as accompanying information on the display device 18 of the viewer.

  Here, the self-attention product information transmitted from the accompanying information providing unit 36 is displayed at a predetermined timing, for example, while viewing the distribution image or after viewing the distribution image by the viewer's screen operation. Can be displayed. Further, the product information can be selectively received from the information providing device 27 by the setting in the viewer-side computer unit 11. In addition, when all the self-attention product information is provided to the viewer, it becomes possible to grasp the potential preference of the viewer himself who cannot be consciously known.

  The accompanying information statistical management means 38 integrates and aggregates the line-of-sight data of each viewer in the system, and stores the integrated statistical processing unit 45 that performs predetermined statistical processing and the spatio-temporal marker setting device 25. The spatial marker is provided with a degree-of-interest evaluation unit 46 that evaluates the degree of attention to the entire viewer based on the processing result in the integrated total processing unit 45.

  In the integrated tabulation processing unit 45, each viewing / listening transmitted from each viewer-side computer unit 11 existing in the system is performed by the same method as the processing in the individual tabulation processing unit 42 of the accompanying information extracting means 35 described above. The gaze data of the viewers are collectively voted for each frame F of the video, and the likelihood distribution data for the gazing point, that is, the gazing point distribution data for each frame F for all viewers is generated for each frame F of the video.

  The attention level evaluation unit 46 compares each of the spatio-temporal markers with the spatio-temporal marker generated by the spatio-temporal marker setting device 25 from the gaze point distribution data of all viewers obtained by the integrated tabulation processing unit 45. With respect to the product information, attention level data relating to the attention level targeting all viewers is generated. That is, here, the gaze point distribution data of all viewers is aggregated for each spatio-temporal volume V of each target product, and the degree of attention corresponding to the aggregate value of the gaze data of all viewers is calculated for each product information. Attention level data consisting of numerical values and / or characters to be expressed is obtained.

  In the statistical information providing means 39, attention level data of each product information intended for viewers in this system is transmitted to the business unit computer unit 12 owned by a business such as a sponsor or an advertising agency. . Accordingly, the business operator can quantitatively evaluate the quality of the advertisement effect on the viewer of the product information by numerical values.

  The spatiotemporal marker setting device 25 is not limited to the above-described configuration, and the following configuration can also be adopted. For example, a configuration in which a spatiotemporal marker is embedded in video information using the line-of-sight measurement device 20 is also possible. In this case, the video delivered to the viewer is presented to the worker, and the worker watches the target product in the video to which additional information such as product information is to be added. At this time, the range specifying unit 29 specifies the existence range of the spatio-temporal volume V for each target product, and the operator gazes at the measurement result of the gaze information of the worker obtained by the gaze measurement device 20. The spatiotemporal volume V of the target product is identified. In the spatiotemporal marker generating means 32, the worker newly inputs product information regarding the spatiotemporal volume V of the specified target product, or the worker selects the corresponding product information from the information database 31. Thus, a spatiotemporal marker associated with the product information is generated, and the spatiotemporal marker is embedded and stored in the video information by the spatiotemporal marker storage means 33. According to this, spatio-temporal markers can be simultaneously generated by a plurality of workers in parallel, by which accompanying information can be added to the volumes of different target products while a plurality of workers are viewing images simultaneously. it can.

  In addition, a spatio-temporal marker generated individually for each target product by a plurality of workers for one video is transmitted from the computer terminal of each worker to the information processing unit 15 via a network such as the Internet or a LAN. A configuration in which the space-time markers are collectively managed by the information processing unit 15 is also possible.

  Furthermore, some of the above-described functions of the information processing unit 15 can be provided in a client computer that provides information using the functions. For example, the spatiotemporal marker setting device 25 may transmit the spatiotemporal marker to the computer main body 17 of the viewer-side computer unit 11, and the computer main body 17 may be provided with the accompanying information extracting means 35 and the accompanying information providing means 36. . In this case, the accompanying information extracted by the processing in the accompanying information extracting means 35 of the computer main body 17 is displayed on the display device 18 by the accompanying information providing means 36 of the computer main body 17.

  In addition, the configuration of each part of the apparatus in the present invention is not limited to the illustrated configuration example, and various modifications are possible as long as substantially the same operation is achieved.

DESCRIPTION OF SYMBOLS 10 Viewer attention information provision system 20 Gaze measurement apparatus 25 Spatio-temporal marker setting apparatus 27 Information provision apparatus 29 Range identification means 30 Type identification means 31 Information database 32 Spatio-temporal marker generation means 35 Accompanying information extraction means 36 Accompanying information provision means 38 Accompanying Information statistics management means 39 Statistical information provision means 42 Individual total processing section 43 Marker-of-interest specifying section V Space-time volume

Claims (8)

In a system for providing information corresponding to the target portion to the viewer by detecting the target portion of the viewer who is viewing an image that changes over time,
When specifying a space-time volume existence range in which ranges of the same object existing in the image are arranged in time series, and setting a space-time marker formed by adding accompanying information of the object to the space-time volume A space marker setting device; and an information providing device that identifies the spatio-temporal marker of an object of interest to a viewer based on a detection result of the attention portion and provides accompanying information in the spatio-temporal marker to the viewer A viewer attention providing system characterized by comprising:   The spatio-temporal marker setting device corresponds to range specifying means for specifying the existence range of the spatio-temporal volume for each of a plurality of objects existing in the image, and the spatio-temporal volume corresponding to each object. The viewer attention information providing system according to claim 1, further comprising a spatiotemporal marker generating unit that generates the spatiotemporal marker for each of the objects by adding accompanying information. The information providing apparatus includes an accompanying information extracting unit that extracts the accompanying information,
The accompanying information extraction means includes an individual aggregation processing unit that aggregates viewer's line-of-sight information serving as an index of the attention portion for each viewer and performs predetermined statistical processing, and an aggregation result in the individual aggregation processing unit, A target marker specifying unit that specifies a spatiotemporal marker that is being watched by a target viewer and that extracts the accompanying information provided to the viewer from the spatiotemporal marker. The viewer attention information providing system according to 1 or 2.   The information providing device integrates, aggregates, and performs statistical processing on the viewer's line-of-sight information, which is an index of the attention portion, for each viewer in the system, so that the spatio-temporal marker of all the viewers aggregated is calculated. 2. An accompanying information statistical management means for obtaining data related to the degree of attention of the accompanying information, and a statistical information providing means for transmitting data obtained by the accompanying information statistical management means to a predetermined business operator. The viewer attention information providing system according to 2 or 3. A spatio-temporal marker setting device that sets a spatiotemporal marker in which accompanying information of the target is added to a spatiotemporal volume in which ranges of the same target existing in an image that changes with time are arranged in time series. ,
A range specifying means for specifying the existence range of the spatio-temporal volume of each of a plurality of objects existing in the image, and adding corresponding incidental information to the spatio-temporal volume corresponding to each object, A spatiotemporal marker setting device comprising a spatiotemporal marker generating means for generating the spatiotemporal marker for each object. A computer of a spatiotemporal marker setting device for setting a spatiotemporal marker in which accompanying information of the target is added to a spatiotemporal volume in which ranges of the same target existing in an image that changes with time are arranged in time series A program for functioning,
A range specifying means for specifying the existence range of the spatio-temporal volume of each of a plurality of objects existing in the image, and adding corresponding incidental information to the spatio-temporal volume corresponding to each object, A program for a spatiotemporal marker setting device, which causes the computer to function as a spatiotemporal marker generating means for generating the spatiotemporal marker for each object. In the apparatus for providing information corresponding to the target portion to the viewer based on the detection result of the target portion of the viewer watching the image that changes over time,
From the detection result of the target portion of the spatiotemporal marker in which the accompanying information of the target object is added to the spatiotemporal volume in which the range of the same target object existing in the image is arranged in time series, the viewer pays attention. Specifying the spatio-temporal marker of the target object, and extracting the accompanying information in the spatio-temporal marker, and providing the accompanying information that provides the viewer with the accompanying information extracted by the accompanying information extracting means And an information providing device. In a program for causing a computer of an apparatus to provide information corresponding to a target part to the viewer based on a detection result of the target part of the viewer who is viewing an image that changes over time,
From the detection result of the target portion of the spatiotemporal marker in which the accompanying information of the target object is added to the spatiotemporal volume in which the range of the same target object existing in the image is arranged in time series, the viewer pays attention. Specifying the spatio-temporal marker of the target object, and extracting the accompanying information in the spatio-temporal marker, and providing the accompanying information that provides the viewer with the accompanying information extracted by the accompanying information extracting means A program for an information providing apparatus that causes the computer to function as means.

Images