JP4782033B2 - Video viewing history acquisition device and program - Google Patents

Description

The present invention relates to a video viewing history acquisition apparatus and program, particularly relates to a video viewing history acquisition apparatus and a program for individuals to obtain a history of images viewed at the personal computer.

  Due to the widespread use of broadband, an increase in storage capacity, and the widespread use of digital photography equipment, various services using video, such as video sharing sites and video blogs, have become widely used. In addition, with the advent of personal computers with built-in TV tuners (hereinafter referred to as PCs), the number of users who watch TV using PCs is increasing. Thus, there is a need to refer again to videos that have been viewed in the past in a situation where the opportunity for users to handle videos increases.

There is an example in which various files (documents, images, videos, music data, etc.) handled on a PC are examined as a conventional approach for the need to refer to information referred to in the past again. These conventional techniques accumulate the user's operation history before and after the reference time of the file and Web page referred to on the PC and the Web page, and utilize them together with the file name (for example, Non-Patent Document 1, 2).
Morita et al., "Desktop search System based on the Action-Oriented algorism", APSITT 2005. Ryo Osawa et al., “Ode Desk: Information Recall Support Tool Based on User Operation History”, 47th Programming Symposium of Information Processing Society of Japan

  When applying the related art to a video file search on a PC, the file name, the playback date of the stored file, and the like can be used at the time of the search, but there are the following problems.

・ The file cannot be obtained:
In addition to video files that exist on a PC storage device (HDD, memory, etc.), videos viewed on a PC include videos that are viewed on video blogs and video sharing sites, as an example that has increased in recent years. And TV broadcast video. That is, although the user is certainly referring to the video, they may not be stored as video files on the PC. For this reason, it may be difficult to obtain a file name or a playback history.

・ Problems that the playback history is difficult to obtain:
Video file format (for example, AVI, MPEG1, WMV) and playback procedure (for example, Windows Media Player (registered trademark), Quick Time Player (registered trademark), Real Player Flash (registered trademark), Macromedia Flash Player (registered trademark)) ) Is wide-ranging. For this reason, it is not easy to comprehensively acquire the playback history of a video file whose entity exists on the PC.

-Problems related to user's memory cannot be determined by file and playback history alone:
User's memory can be an important clue when trying to re-reference previously referenced video. However, it is difficult to obtain these from the playback history of the video file. For example, even if the playback history of a video file can be acquired from the log of the playback means, another operation (for example, scroll operation or page switching) may be performed on the user web browser during that time. Attention may not be directed to the video being displayed. As described above, there are cases where the log output from the video reproduction means and the browsing history of the Web page cannot always be effective information during the search.

  Due to the above problems, it can be said that it is difficult to efficiently search for video files that have been seen in the past with the prior art.

The present invention has been made in view of the above points, and pays attention to the fact that the user's memory is an important clue when searching for videos that have been viewed in the past. It is obtained in a manner independent of its object to provide a video viewing history preparative TokuSo location and program capable.

  FIG. 1 is a diagram for explaining the principle of the present invention.

This onset Akira is a video viewing history acquisition method to get the history of the video you watched in the personal computer,
The screen image acquisition means acquires the screen image of the active window that is the operation target among the windows displayed on the screen of the personal computer together with time information at a fixed time interval Δt, and acquires the screen image stored in the storage means Step (step 1);
A viewing determination step (step 2) in which the viewing determination means classifies into a screen image at the time of video viewing and a screen image other than that using the screen image read from the storage means;
The viewing history storage means obtains the video viewing start time and end time from the time information of the screen image determined to be during video viewing in the viewing determination step, and together with the screen image corresponding to the section between the start time and the end time And a viewing history storing step (step 3) for storing in the viewing history database ,
In the viewing determination step (step 2),
From the color difference between the screen image It at the time t acquired in the screen image acquisition step and the screen image It + 1 at the next time t + Δt, a binary image BIt is generated with a pixel having a color change being 1 and a pixel having no color change being 0,
For each pixel of the binary image BIt, whether or not the pixel (i, j, t) has the same tendency as the neighboring 8 pixel with respect to the presence or absence of a color change is indicated by the difference between the value and the value of the neighboring 8 pixel. Obtaining an index value v (i, j, t);
The sum C (t) of the number of pixels having a value indicating that the index value v (i, j, t) has the same tendency is totaled over the binary image BIt according to the presence or absence of a color change,
For each presence / absence of color change, a ratio CR (t) in the total number of pixels of the sum C (t) is obtained as a first feature amount,
Using CR (t + 1) obtained in the same manner as the ratio CR (t), a difference ΔCR (t) between CR (t) and CR (t + 1) is obtained as a second feature amount for each color presence / absence,
Condition determination is performed on the first feature amount CR (t) and the second feature amount ΔCR (t) based on a predetermined threshold value, and a screen image at the time of video viewing and other screen images are determined. Classify.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 1 ) is a video viewing history acquisition device for acquiring a history of videos viewed on a personal computer,
A screen image acquisition unit 10 that acquires a screen image of an active window that is an operation target among windows displayed on a screen of a personal computer together with time information at a predetermined time interval Δt, and stores the screen image in a storage unit;
Using the screen image acquired by the screen image acquisition means 10 to classify the image into a screen image at the time of video viewing and a screen image other than that;
From the time information of the screen image determined to be at the time of viewing the video by the viewing determining means 20, the start time and end time of viewing the video are acquired, and the viewing history database 220 together with the screen image corresponding to the section between the start time and the end time. includes a viewing history storage means 30 to store, into,
The viewing determination means 20
From the color difference between the screen image It at time t acquired by the screen image acquisition means 10 and the screen image It + 1 at the next time t + Δt, a binary image BIt is generated with 1 as a pixel having a color change and 0 as a pixel having no color change. Means,
For each pixel of the binary image BIt, whether or not the pixel (i, j, t) has the same tendency as the neighboring 8 pixel with respect to the presence or absence of a color change is indicated by the difference between the value and the value of the neighboring 8 pixel. Means for obtaining an index value v (i, j, t);
Means for totalizing C (t) of the number of pixels having a value indicating that the index value v (i, j, t) has the same tendency over the binary image BIt according to the presence or absence of a color change;
Means for determining, as a first feature amount, a ratio CR (t) in the total number of pixels of the total C (t) for each color change;
Means for obtaining a difference ΔCR (t) between CR (t) and CR (t + 1) as a second feature amount, using CR (t + 1) obtained in the same manner as the ratio CR (t), for each color presence / absence;
Condition determination is performed on the first feature amount CR (t) and the second feature amount ΔCR (t) based on a predetermined threshold value, and a screen image at the time of video viewing and other screen images are determined. Means for classifying .

The present invention (Claim 2 ) is a program that causes a computer to function as the video viewing history acquisition apparatus according to Claim 1 .

  As described above, according to the present invention, the video section is extracted based on the feature amount of the screen image of the window, and thereby data indicating the time when the video is viewed and the content of the video (when the video is played back). Screen image) can be obtained. As a result, any video playback means that displays video on the screen of a personal computer can acquire a video viewing history regardless of the type of video playback means. In addition, it is possible to acquire information on the video actually displayed on the foreground screen.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the following description, “window” refers to an independent small screen such as a browser or folder window on a PC screen. The “active window” refers to a window that is currently an operation target and is displayed in the forefront among the windows displayed on the screen.

[First Embodiment]
In the present embodiment, an example is shown in which a user views a video existing on the Internet on a browser using a PC.

  FIG. 3 shows the configuration of the video viewing history acquisition apparatus according to the first embodiment of the present invention.

  The video viewing history acquisition apparatus shown in the figure includes a browser image acquisition unit 10, a viewing determination unit 20, a history registration processing unit 100 including a viewing history storage unit 30, a storage unit 210, and a viewing history database 220.

  The storage unit 210 has an area for storing the browser screen image and time information acquired by the browser image acquisition unit 10 and an area for storing the label information generated by the viewing determination unit 20.

  The viewing history database 220 stores viewing history information generated from the label information and time information in the viewing history storage unit 30.

  The operation in the above configuration will be described below.

  FIG. 4 is a flowchart of the operation outline in the first embodiment of the present invention.

  Step 11) The browser image acquisition unit 10 acquires a screen image of the browser at a predetermined time interval using a screen capture tool, and stores the screen image and time information in the storage unit 210 together with the acquired time information.

  Step 12) Next, the viewing determination unit 20 reads two screen images from the storage unit 210 based on the time information, and classifies them into screen images at the time of viewing the video and other screen images using the color change characteristics. Then, a label (viewing / other than viewing) is assigned and stored in the storage unit 210.

  Step 13) The viewing history storage unit 30 acquires the time information of the screen image determined to be at the time of video viewing (with the “viewing” label) from the storage unit 210, and the video viewing start time from the time information. The end time is acquired and stored as a history in the viewing history database 220 together with the screen image corresponding to the section.

  Next, the operation of the viewing determination step of step 12 will be described in detail.

  FIG. 5 is a flowchart of the viewing determination step (S12) in the first embodiment of the present invention. The viewing determination step includes a feature amount calculation step (S121) and a feature amount determination step (S122).

Step 121) In the viewing determination unit 20, a feature amount 1 generation step shown in FIG. 6 is performed as a feature amount for classifying the screen image acquired in Step 11 into a screen image at the time of viewing the video and other screen images. Two feature amounts (C _m (t), ΔCR _m (t)) are calculated by the processing and the feature amount 2 generation step shown in FIG.

  FIG. 6 is a flowchart of the feature quantity 1 generation step according to the first embodiment of the present invention.

Step 1211) Feature 1 generation step:
Step 12111) in the screen image obtained in step 11, the color difference is calculated for each pixel in the screen image I _{t + 1} which has been acquired screen image I _t and time t + △ t obtained at time t, those for which the color change A binary image BI _t is generated by setting 1 or 0 to 0.

Step 12112) Pixel value P (i, j, t) of (i, j) in the binary image BI _t generated in Step 12111 (where i represents a pixel number in the x-axis direction, i = 0, 1, 2,..., I-1, j represents pixel numbers in the y-axis direction, j = 0, 1, 2,..., J-1, t is time, and P (i, j, t) is 1 or 0. The pixel number N (1, t) where P (i, j, t) = 1 and the pixel number N (0, t) where P (i, j, t) = 0. Are counted and output.

Step 12113) Among the pixels included in the binary image BI _t output in Step 12111, any P (i, j, t) (i = 1, 2, 3) included in a range excluding only one surrounding pixel. ,..., I−2, j = 1, 2,..., J−2) (see FIG. 7 for the range of i, j), P (i−1, r−1, t) to P (i + 1) , j + 1, t) represented by P (i, extracting j, t) near 8 pixels, the index based on the color difference of the neighboring eight pixels and its center pixel _{v m (i, j, t} ) Is obtained by the following algorithm.

次に上記で求めたｖ_ｍ（ｉ，ｊ，ｔ）（ただし、ｍ＝０または１）を、以下の式により２値画像全体にわたって積分し、その値Ｃ_ｍ（ｔ）を出力する。

Next, v _m (i, j, t) (where m = 0 or 1) obtained above is integrated over the entire binary image by the following equation, and its value C _m (t) is output.

ステップ１２１１４） ステップ１２１１２で出力されるＮ（ｍ，ｔ）の値及びステップ１２１１３で出力されたＣ_ｍ（ｔ）の値を入力し、次の式で与えられる特徴量１ＣＲ_ｍ（ｔ）を出力する。

Step 12114) The value of N (m, t) output in step 12112 and the value of C _m (t) output in step 12113 are input, and the feature value 1CR _m (t) given by the following equation is output. To do.

この特徴量は、動きのある部分（１）または、動きのない部分（０）の集中度を表す指標である。映像視聴時は画面を基本的に動かさない。このため、映像視聴時は、動いている部分が映像の表示されている部分に集中する。そのため、映像視聴時においては、この値は大きくなる傾向がある。ＰＣにおけるアクティブウィンドウの画面イメージ（約６時間半分 ２秒間隔）から計算した映像視聴時及び映像視聴時以外の特徴量１の分布を図８、図９に示す。

This feature amount is an index representing the degree of concentration of a portion (1) with motion or a portion (0) without motion. The screen is not basically moved when viewing the video. For this reason, when viewing the video, the moving part is concentrated on the part where the video is displayed. Therefore, this value tends to increase during video viewing. 8 and 9 show the distribution of the feature amount 1 calculated during the viewing of the video and the time other than the viewing of the video calculated from the screen image of the active window on the PC (approximately 6 hours and half 2 seconds).

  Next, the feature amount 2 generation step will be described.

  FIG. 10 is a flowchart of the feature quantity 2 generation step (S1212) in the first embodiment of the present invention.

Step 1212) Feature quantity 2 generation step:
Step 12121) In the above step 1211, it is determined whether or not the generation of the feature quantity 1 has been completed for the times t and t + 1. If not, the process returns to the step 1211. If completed, the process proceeds to step 12122.

Step 12122) In this step, the outputs CR _m (t) and CR _m (t + 1) of Step 1211 at the times t and t + 1 are input, and the feature value 2ΔCR _m (t) given by the following equation is stored in the memory ( (Not shown).

映像視聴時は、動いている部分、動いていない部分がほぼ同じであること、また、前述したように特徴量１が大きい値で安定している傾向があることから、差分は小さくなる傾向がある。ＰＣにおけるアクティブウィンドウの画面イメージ（約６時間半分 ２秒間隔）から計算した映像視聴時および映像視聴時以外の特徴量２の分布を図１１、図１２に示す。

During video viewing, the moving part and the non-moving part are almost the same, and since the feature amount 1 tends to be stable at a large value as described above, the difference tends to be small. is there. FIGS. 11 and 12 show the distribution of the feature amount 2 calculated during the viewing of the video and the time other than the viewing of the video calculated from the screen image of the active window on the PC (approximately 6 hours and half 2 seconds).

Step 122) Feature amount determination step:
FIG. 13 is a flowchart of the feature amount determination step in the first embodiment of the present invention.

In step 1221) viewing determination unit 20, it is generated in step 121 above, the memory (amount characteristics stored in not _{shown) F n (CR 0 (t} ), CR 1 (t), △ CR 0 (t ), ΔCR ₁ (t)) determines whether or not a preset threshold condition is satisfied. If the condition is satisfied, the process proceeds to step 1222, and if not, the process proceeds to step 1225. Note that the threshold value is determined in advance using a screen image for a learning sample in which a label is added to the video viewing section in advance.

  Step 1222) It is determined whether or not the condition determination has been completed for all feature amounts. If completed, the process proceeds to Step 1224. If not completed, the process proceeds to Step 1223.

  Step 1223) Set n = n + 1 and return to Step 1221.

To step 1224) Screen image _{I t,} grants label ( "viewing") representing the time of image viewing, and stores in the storage unit 210.

Step 1225) A label ("non-viewing") other than the time of video viewing is _assigned to the screen image It and stored in the storage unit 210.

  Next, step 13 (viewing history storage step) in FIG. 4 will be described in detail.

  FIG. 14 is a flowchart of the viewing history storing step (S13) in the first embodiment of the present invention.

  Step 131) The viewing history storage unit 30 acquires the start time and the end time of video viewing from the determination result (label information) in Step 123 and the time information of the screen image.

  Step 132) The screen image corresponding to the section is stored in the viewing history database 220 together with the start time and end time acquired in Step 131.

  Next, the detailed operation of the time information acquisition step (S131) in FIG. 14 will be described.

  FIG. 15 is a flowchart of the time information acquisition step (S131) in the first embodiment of the present invention.

Step 1311) viewing history storage unit 30 obtains the time from the storage unit 210 is stored in the label information and the step 11 of the applied screen image I _t at step 12 above.

  Step 1312) It is determined whether the label of the screen image It is “viewing”. If so, the process proceeds to Step 1314, and if not, the process proceeds to Step 1313.

  Step 1313) The time t is set to t = t + 1, and the process proceeds to Step 1311.

  Step 1314) The time of the screen image It is set as the start time.

Step 1315) to get the label and time of screen image _{I t} from the storage unit 210.

  Step 1316) It is determined whether the label of the acquired screen image It is “viewing”. If so, the process proceeds to Step 1317, and if not, the process proceeds to Step 1318.

  Step 1317) The time t is set to t = t + 1, and the process proceeds to Step 1315.

Step 1318) When the label of the screen image It is “viewing” in step 1316, the time of the screen image It _-1 is set as the viewing end time.

Step 132) viewing history storage unit 30, stored along with the acquired start time and end time in step 131, the screen image _{I t (t = T s ~T} e) storage unit 210 as viewing history corresponding to the video section To do.

[Second Embodiment]
In this embodiment, an example will be described in which a video is searched using data acquired in the first embodiment and stored in the viewing history database 220.

  FIG. 16 shows the configuration of the video viewing history acquisition apparatus according to the second embodiment of the present invention.

  The video viewing history acquisition apparatus shown in the figure has a configuration in which a search unit 300 for searching a video viewing history is added to the configuration of FIG.

  FIG. 17 is a flowchart of search processing according to the second embodiment of the present invention.

  Step 21) First, the search unit 300 acquires time information serving as a search condition designated by the user via the input unit 301.

  Step 22) The viewing history information (viewing start time, viewing end time, screen image) that matches the search conditions at the specified time is acquired from the viewing history database 220.

  Step 23) The search result acquired in Step 22 is output to the display means 302 and presented to the user.

[Third Embodiment]
In this embodiment, in addition to the data acquired in the first embodiment, four tool information of a key operation monitoring tool, a mouse operation monitoring tool, a clipboard monitoring tool, and a process monitoring tool and display means (not shown) are displayed. An example of generating a viewing history using text information of a displayed page will be described.

  FIG. 18 is a flowchart of the operation in the third embodiment of the present invention.

  Step 31) Similar to the first embodiment, the browser image acquisition unit 10 acquires viewing start time and end time information and a screen image of the section.

  Step 32) Based on the time information of the viewing section acquired in Step 31, the viewing determination unit 20 acquires the above four tools and page information from the PC that is used as the video viewing history acquisition device, and the viewing section Associate with. Specifically, the key and mouse operation performed in the viewing section, the information on the clipboard, the name of the activated application, and the text information of the displayed page are set as related information of the viewing section.

  Step 33) The information acquired in Step 31 and the tool and page information associated with the viewing section in Step 32 are stored in the viewing history database 220.

  By creating the database as described above, it is possible to search the viewing section by using the keyword and the application name that has been started at the same time in addition to the time information in the second embodiment.

[Other embodiments]
In the first embodiment, the case of viewing on a browser has been described, but other video playback means can be applied as long as they are displayed on the screen of a personal computer.

  In the third embodiment, information related to the viewing section is used as the related information. However, by expanding the range and adding information performed within a certain period of time before and after, the information is started before and after. It is possible to search from the application, clipboard information and page information.

  As described above, according to the present invention, the video viewing section can be extracted based on the screen image information of the window. Thus, if there is a feature of being displayed on the screen of the PC, playback means, a player (for example, Windows Media Player (registered trademark), Quick Time Player (registered trademark), Real Player Flash (registered trademark), Regardless of Macromedia Flash Player (registered trademark), viewing time and screen information of the window at that time can be acquired.

  Further, by using information that is actually viewed by the user, such as a screen image, it is possible to acquire information of a video that is being actively watched, not simply being played back in the background. Thereby, when searching for videos viewed in the past, it is possible to utilize the storage of screen information when viewed. As a result, it is possible to efficiently re-reference previously viewed videos.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  In the present invention, since it is possible to extract a video viewing section based on the screen image information of a window, a playback means or player (for example, Windows Media Player (registered trademark)) characterized by being displayed on a PC screen. , Quick Time Player (registered trademark), Real Player Flash (registered trademark), Macromedia Flash Player (registered trademark)).

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a block diagram of the video viewing history acquisition apparatus in the 1st Embodiment of this invention. It is a flowchart of the operation | movement outline | summary in the 1st Embodiment of this invention. It is a flowchart of the viewing determination step (S12) in the first embodiment of the present invention. It is a flowchart of the feature-value 1 production | generation step (S1211) in the 1st Embodiment of this invention. It is a pixel to be processed in step 1213 of the feature quantity 1 generation step in the first embodiment of the present invention. The distribution of the feature 1CR _m in the first embodiment of the present invention (at the time of image viewing). The distribution of the feature 1CR _m in the first embodiment of the present invention (except when image viewing). It is a flowchart of the feature-value 2 production | generation step (S1212) in the 1st Embodiment of this invention. A first distribution of the feature quantity 2 △ CR _m in the embodiment of the present invention (at the time of image viewing). 6 is a distribution of feature amount 2ΔCR _m (except when viewing video) in the first embodiment of the present invention. It is a flowchart of the feature-value determination step (S122) in the 1st Embodiment of this invention. It is a flowchart of the viewing history preservation | save step (S13) in the 1st Embodiment of this invention. It is a flowchart of the time information acquisition step (S131) in the 1st Embodiment of this invention. It is a block diagram of the video viewing apparatus at the time of the search in the 2nd Embodiment of this invention. It is a flowchart of the search process in the 2nd Embodiment of this invention. It is a flowchart of the operation | movement in the 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Screen image acquisition means, Browser image acquisition part 20 Viewing determination means, Viewing determination part 30 Viewing history preservation | save means, Viewing history preservation | save part 100 History registration process part 210 Storage part 220 Viewing history database 300 Search part 301 Input means 302 Display means

Claims (2)

A video viewing history acquisition device for acquiring a history of videos viewed on a personal computer,
A screen image acquisition unit that acquires a screen image of an active window that is an operation target among windows displayed on the screen of the personal computer together with time information at a predetermined time interval Δt, and stores the screen image in a storage unit;
Using the screen image read from the storage means, viewing determination means for classifying into a screen image at the time of video viewing and a screen image other than that,
From the time information of the screen image determined to be at the time of video viewing by the viewing determination means, the video viewing start time and end time are acquired, and together with the screen image corresponding to the start time and end time interval, the viewing history database Viewing history storage means for storing;
Have
The viewing determination means includes
From the color difference between the screen image It acquired at the time t acquired by the screen image acquisition means and the screen image It + 1 at the next time t + Δt, a binary image BIt is generated in which a pixel having a color change is 1 and a pixel having no color change is 0. Means,
For each pixel of the binary image BIt, whether or not the pixel (i, j, t) has the same tendency as the neighboring 8 pixel with respect to the presence or absence of a color change is indicated by the difference between the value and the value of the neighboring 8 pixel. Means for obtaining an index value v (i, j, t);
Means for totalizing C (t) of the number of pixels having a value indicating that the index value v (i, j, t) has the same tendency over the binary image BIt according to the presence or absence of a color change;
Means for determining, as a first feature amount, a ratio CR (t) of the total C (t) in the total number of pixels for each color presence / absence;
Means for obtaining a difference ΔCR (t) between CR (t) and CR (t + 1) as a second feature amount using CR (t + 1) obtained in the same manner as the ratio CR (t), depending on whether or not there is a color change; ,
Condition determination is performed on the first feature amount CR (t) and the second feature amount ΔCR (t) based on a predetermined threshold, and a screen image at the time of video viewing and other screen images Means to classify
Video viewing history acquisition apparatus characterized by comprising. Computer
A program that functions as the video viewing history acquisition device according to claim 1 .

Images