JP2011520324A - Method and apparatus for media content playback and playback control in web page - Google Patents

Abstract

  A method and apparatus for controlling playback and playback of media content in a web page is proposed. In one embodiment, a web page that uses a media player and media content is loaded, and an operator associated with the media player is identified by processing an image of the web page. This is a method for presenting media content in a web page that presents.

Description

  The present invention relates to web crawling, and more particularly to a method and apparatus for media content playback and playback control in a web page.

  As the WWW (World Wide Web) spreads and the bandwidth for the public advances, the distribution of media content using web pages is also increasing. Media contents provided to the web page viewer, such as audio and video, are provided to the web page viewer in various formats and various media players. Examples of media players currently used include Windows (registered trademark) MediaPlayer (trademark), Quicktime (registered trademark), RealPlayer (registered trademark), FlashPlayer (registered trademark), and the like. In these media players, various types of operators (controls) are provided so that a web page viewer can operate the reproduction of moving images. Some media players provide different versions for each type of web browser application.

US Pat. No. 6,353,929 US Pat. No. 6,519,648 US Pat. No. 6,535,880 US Pat. No. 6,714,933 US Pat. No. 6,721,741 US Pat. No. 6,725,222 US Pat. No. 6,725,275 US Pat. No. 7,082,426 US Pat. No. 7,162,696 US Pat. No. 7,800,801 US Pat. No. 7,281,034 US Patent Application Publication No. 2002/0056089 US Patent Application Publication No. 2002/0091764 US Patent Application Publication No. 2003/0066070 US Patent Application Publication No. 2004/0003102 US Patent Application Publication No. 2004/0177096 US Patent Application Publication No. 2004/0254956 US Patent Application Publication No. 2004/0254958 US Patent Application Publication No. 2004/0267812 US Patent Application Publication No. 2006/0026162 US Patent Application Publication No. 2006/0041589 US Patent Application Publication No. 2006/0242192 US Patent Application Publication No. 2006/0271977 US Patent Application Publication No. 2007/0073758 US Patent Application Publication No. 2007/0124110 US Patent Application Publication No. 2007/0150612 US Patent Application Publication No. 2007/0168543 US Patent Application Publication No. 2007/0277088 US Patent Application Publication No. 2007/0294252 US Patent Application Publication No. 2008/0046562 US Patent Application Publication No. 2008/0046738 US Patent Application Publication No. 2008/0120420 US Patent Application Publication No. 2008/0140712 US Patent Application Publication No. 2008/0141162 International Publication No. WO2006 / 058075 Pamphlet International Publication No. WO2008 / 021459 Pamphlet International Publication No. WO 98/27497 Pamphlet

BUSHER et al., "Generating and Using Gaze-Based Document Annotations," CHI 2008 Proceedings-Works in Progress, April 5-10, 2008, Florence, Italy, pages 3045-3050. FU et al., "Detecting Phishing Web Pages with Visual Similarity Assessment Based on Earth Mover's Distance (EMD)," IEEE Transactions on Dependable and Secure Computing, Vol.3, No.4, October-December 2006, 11 pages.

  An object of the present invention is to provide a method and apparatus for reproducing and monitoring media content in a web page.

  One embodiment of the present invention is a web crawler for media content playback and playback control in a web page, which includes a web browser, an image generator, an image analyzer, a media player controller, and a signature generator.

  In this embodiment, first, the web browser loads a web page that uses the media player, and sends information indicating the layout of the page to the image generator. The image generator generates a web page image, that is, an image that will appear when the web page is displayed on a monitor screen or the like. The image generator identifies the media player to be used and the media content to be played by the media player by analyzing the web page image using an image processing technique. At the time of player identification, the image analyzer also identifies an operator associated with the media player, that is, an operator such as a playback button serving as a means for controlling media content reproduction for a web page viewer. The image analyzer further identifies the media content display area and the media player status. There are various states that a media player can take. Here, it is identified whether the media content is being played back or the media content data buffering is in progress.

  The media player controller sends information representing the media player operation element and the media player state to the web browser, and controls the media player operating there. For example, by giving a playback command to the web browser, the media player is caused to execute the same operation as that when the web page viewer clicks the play button. The signature generator then uses the media information to generate a digital signature for the media content. The media content identifier identifies some or all of the media content in the web page by referring to a database that associates digital signatures with known media content using the generated digital signature.

  Another embodiment of the present invention is a web page media content playback / playback monitoring method executed using the web crawler or the like according to the above-described embodiment. In this method, the web crawler loads a web page that uses one or more types of media players and media content, generates an image of the web page, and further analyzes the image to obtain the media content and The position and function of the media player operator for the user are identified. Once the media player, media content, and user controls are identified, the web crawler generates a digital signature for the media content (starting playback of the media content first if not automatically played). To do. The web crawler also monitors the media player status during media content playback. Media player statuses that can be monitored are: “Media content playing” “Media content playback paused” “Media content playback stopped” “Media content playback ended” “Media content data buffering” “Media player error occurred” And so on.

It is a figure which shows the example of the web page which uses media content and a media player. It is a figure which shows another example of the web page which uses media content and a media player. It is a figure which shows in detail the media player and media content in FIG. 1A. It is a block diagram which shows an example of the web crawler used for reproduction | regeneration and reproduction | regeneration monitoring of the media content in a web page. It is a figure which shows the media player at the time of media content reproduction waiting. It is a figure which shows the media player in media content reproduction | regeneration. It is a figure which shows the media player after completion | finish of media content reproduction. It is a figure which shows the media player in media content data buffering. It is a flowchart which shows the example of the media content reproduction | regeneration and reproduction | regeneration monitoring procedure in a web page. 1 is a diagram illustrating an example of a processing system capable of implementing the methods and apparatuses illustrated herein. FIG.

  Hereinafter, a system according to an embodiment of the present invention will be described. In this description, some of the constituent members are realized by software, and the software is executed on corresponding hardware. However, such a device configuration is merely an example, and the present invention can be implemented in other forms. can do. For example, some or all of the components described as hardware or software can be realized by using only dedicated hardware, only dedicated software, or only dedicated firmware. It can also be realized in combination.

  In addition, in this specification, the method and apparatus according to the embodiment are described using an example in which the media content to be played back and the media player whose status is monitored are single. However, it should be noted that these methods and apparatuses can be applied to web pages that use a plurality of media players and media contents.

  FIG. 1A shows a web page 100 using a media player 102 and media content 104 as an example. The page 100 is displayed when the page 100 is downloaded from a website visited by a user following a URL (uniform resource locator) to a computer, for example, the processing system shown in FIG. Normally, a computer language such as HTML (hypertext mark-up language) or XML (extensible mark-up language) is used for the description of the page 100, and a web browser is used for downloading and displaying the page 100. Known web browsers include Microsoft (registered trademark) Internet Explorer (registered trademark), Mozilla (registered trademark) Firefox (registered trademark), Apple (registered trademark) Safari (registered trademark), and the like.

  The media content 104 can be included, for example, as a description representing the content 104 in the description regarding the web page 100. In this case, the content 104 is downloaded together with other content constituting the page 100 and displayed by the web browser. Further, the location of the content 104 may be described in the description about the page 100. In this case, the content 104 is downloaded from a location different from the page 100. Furthermore, the media player 102 can also be downloaded from the page 100 and used. However, normally, a suitable media player among the media players already installed on the computer is used as the player 102. For example, a suitable browser plug-in is called by a web browser and used. The types of media content that can be played by such a player 102 are usually limited to those created in accordance with a specific type of data format. That is, although it is not possible to create a media player that can reproduce media contents of many or all data formats, the media content data format that can be reproduced usually differs for each type of media player. Since the types of media players used are generally different for each website and each web page, depending on what kind of media player is used for each site or page, The types of media player controls for users (for example, web viewers) displayed on the site or page in association with individual media players are also different. In addition, some web pages start playing media content once the web page is loaded, even without user interaction or instructions from the user.

  FIG. 1B shows a web page 106 using media player 108 and media content 110 as another example. The player 108 in the figure is different from the media player 102, and the data format of the content 110 is the same as, similar to, or different from that of the media content 104. Differences between the players 102 and 108 are particularly apparent in the controls. For example, some of the media player controls for users provided by the player 108 are functionally similar to the controls for the player 102, while some others are provided by the player 102. It is not. Generally speaking, the method and apparatus according to the embodiments of the present invention identify an operator for a user provided by one or a plurality of types of players 102 and 108 in use on the pages 100 and 106, and the identified operation. The reproduction of the contents 104 and 110 related to the individual players 102 and 108 is controlled using the child.

  FIG. 2 shows details of the media player 102 and media content 104 used in the web page 100 shown in FIG. 1A. The player 102 is provided with a play button 204 and a play / pause button 206 in addition to a media content display area 202 in which a moving image or the like of the content 104 is displayed. The playback button 204 is displayed when the playback of the content 104 can be started (before the playback is started), and is not displayed when the content 104 is played back. The playback / pause button 206 displays a playback symbol when the content 104 is not played back (when playback is paused or stopped), and is not shown when the content 104 is played back or buffered. A pause symbol is displayed.

  The media player 102 is further provided with a progress bar 208. This bar 208 is a bar for notifying the user of the progress of the content reproduction, and the time, the number of frames, etc. based on the beginning (left end of the bar 208) and end (right end of the bar 208) of the media content 104 are displayed. The above progress bar indicator 210 is configured to point to. In this example, a countdown timer 212 that displays the remaining playback time when the content 104 is played back at the desired speed is also used. A count-up timer that displays the playback time of the content 104 based on the start or end of the content 104 can be used instead of or together with the remaining time indicator. Further, since the player 102 is provided with a volume operator 214, the audio output level can be adjusted when the content 104 including audio is reproduced.

  In the media player 102, only the play / pause button 204, play button 206, progress bar 208, progress bar indicator 210, countdown timer 212, and volume operator 214 are provided as the controls. Can provide different numbers and types of media player controls and can be used for playback control of the media content 104. For example, a pause button, stop button, play / stop button, media display size operator, fast forward button, rewind button, elapsed time indicator, countdown indicator, elapsed time / total time indicator, etc. It can be used together with or instead of the operators 204 to 214.

  FIG. 3 shows an example block configuration 300 of a web crawler for media content playback and playback monitoring within a web page. The web crawler 300 includes a web browser 302, an image generator 304, an image analyzer 306, a media player controller 308, and a signature generator 310, and is connected to the WWW 312 on the Internet so that web pages can be received from various websites. ing. For example, such a crawler 300 can be realized in the form of a software module executed on a certain processing platform (such as the processing system shown in FIG. 9).

  In order to execute and monitor media content reproduction on a web page, for example, reproduction of media content 104 on the web page 100 shown in FIG. 1, the web browser 302 displays web page information describing the page 100. Receive and load via WWW 312. The web page information to be loaded describes the page 100 using a code or computer language such as HTML code, XML code, Javascript (registered trademark) code or the like. As the browser 302, commercial or custom web browser software capable of executing display based on the web page information such as Internet Explorer (registered trademark) can be used.

  When displaying the web page 100 including the media content 104, the web browser 302 executes and activates a media player associated with the content 104 on the page 100, for example, the media player 102 shown in FIG. The operation of reproducing the content 104 (actuating the player 102) can be realized by the operation of the browser 302 itself, a browser plug-in (a program that controls a specific function by interacting with the host application), or a combination thereof. Browser 302 further provides web page information to image generator 304 and image analyzer 306, and media player information, media content information, or both, to generator 304. The web page information from the WWW 312 may be received by the browser 302 and the analyzer 306 in parallel.

  The image generator 304 uses the web page information sent from the web browser 302 to generate a web page image, that is, an image that will be seen by the user when the web page is displayed on the monitor. The image generator 304 generates such web page images over the number corresponding to the refresh rate of a display for a media playback device such as a computer monitor or a television receiver. By appropriately setting the number of images to be generated and the generation speed, it is possible to suitably detect media contents and operators in the image analyzer 306 and generate a digital signature in the signature generator 310.

  The image analyzer 306 receives the web page image generated by the image generator 304 and analyzes it using an image processing technique, so that information about the web page 100, particularly the media player 102 used in the page 100, is displayed. (For example, a page 100 that uses one type of player 102 to play the media content 104 for simplicity of discussion, but as described elsewhere herein, The methods and apparatus of embodiments of the invention can also be applied to web pages that use multiple players 102). Specifically, the analyzer 306 analyzes a web page image indicating the initial state of the page 100 loaded by the web browser 302, and thereby determines the presence / absence and position of the player 102. For this determination, the analyzer 306 attempts to identify the media content display area 202 and controls the player 102 such as a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume, and the like. Attempts to identify operators, progress bars, media display size operators, fast forward buttons, rewind buttons, elapsed time indicators, countdown indicators, elapsed time / total time indicators, etc. All the operators illustrated here may be set as identification targets, or only certain specific controls (groups) such as the play / pause button 206, stop button, elapsed time / total time indicator, and the like may be set as identification targets. .

  The image analyzer 306 uses image matching with a hint or edge detection as the image processing technique. The purpose of image matching is to identify the media player 102 and the media content display area 202, and the purpose of edge detection is to identify the media player operator. As the hint, any information that can be used for identifying the player 102 and the area 202 by the analyzer 306 can be used. For example, a hint may be derived from a code such as HTML included in the received web page information, a known coordinate value, a comparison result between consecutive images, related network traffic, or the like. An example of related network traffic is traffic generated when a plug-in is called by the web browser 302 (“Rectangle 20, 20, 100, 100”, etc.).

に示す。この式（Ｅｑ．１）中、Ｉはウェブページ画像、Ｔはテンプレート、Ｒは結果を表しており、総和の範囲はｘ’＝０，…，ｗ−１及びｙ’＝０，…，ｈ−１の範囲、即ちテンプレート及び画像パッチの全幅ｗ及び全高ｈに亘る範囲とされている。こうした画像処理を実行できるソフトウェアはＯｐｅｎＣＶライブラリから入手することができる。なお、ここで述べた手法に代え又はそれと共に、それ以外の画像処理技術、手法乃至アルゴリズムを用いることもできる。 The hint is used to focus on a target area, for example, 202 when identifying the media player 102 and the media content display area 202. An example of the algorithm is

Shown in In this formula (Eq.1), I is a web page image, T is a template, R is a result, and the sum ranges are x ′ = 0,..., W−1 and y ′ = 0,. -1 range, that is, a range extending over the entire width w and height h of the template and the image patch. Software capable of executing such image processing can be obtained from the OpenCV library. Note that other image processing techniques, methods or algorithms can be used instead of or in addition to the methods described here.

  The image analyzer 306 not only identifies the position of the media player 102, the position of the media content display area 202, and the position and function of the media player operator, but also identifies the state of the player 102. For example, the media player status such as “media content playback” “media content playback paused” “media content playback stopped” “media content playback end” “media content data buffering” “media player error occurred” Identify.

  Among them, the media player state “media content playback paused” is a state where the media player 102 is not actively playing the media content 104 and cancels the state when the state is canceled (pause button). This is a state in which the reproduction of the content 104 is resumed from almost the same place as when is pressed.

  The media player state of “Media content playback stopped” is a state in which the playback of the content 104 is resumed from the beginning when the media player 102 is not actively playing back the media content 104 when the state is canceled. That is.

  The media player state of “media content playback end” is a state where the media player 102 is stopped at the end of the content 104 among the states where the media player 102 is not actively playing back the media content 104.

  Since the image analyzer 306 has such a function, the web crawler 300 determines whether or not the media content 104 is being played back, starts or stops playback of the content 104, and finishes playback of the content 104. Alternatively, it is possible to detect the pause or the buffering of the data, and it is possible to detect problems or errors that occur during the reproduction of the content 104.

  The media player controller 308 receives operation information for the user related to the media player 102 from the image analyzer 306. The operator information for the user includes, for example, the function of the operator (play / pause, etc.), the position in the web page (x, y coordinate values, etc.), various usage conditions (playback of the media content 104 by the player 102). Timing etc.) is included. In order to control the player 102, the controller 308 gives a command, for example, a command simulating a user input such as a mouse click at a specific position and a press of a specific key to the web browser 302. The browser 302 regards the command as a user input and executes it. Therefore, the controller 308 can instruct the browser 302 by such a command, for example, a mouse click command, to recognize that the inside of the area identified as the playback button 206 by the analyzer 306 is clicked. Playback can be started. Other user inputs, gestures, and commands can be generated by the controller 308 in a pseudo manner or can be executed by the browser 302.

  The signature generator 310 receives the media information and generates a digital signature for the media content 104 being played. This digital signature generation is executed by an arbitrary algorithm as long as one or a plurality of types of numerical values that can identify the content 104 is generated based on audio, video, or both included in the content 104, for example. can do. Media content identifier 314 identifies content 104 based on the digital signature so generated. This identification is performed, for example, by referencing a database that associates the reference digital signature with the library of reference media content with at least a portion of the received digital signature. That is, the identifier 314 obtains the similarity by comparing the received digital signature with the reference digital signature. If any of the obtained similarities exceeds a predetermined threshold value, the similarity for the content 104 is identified as the highest.

等の再生シンボルが表示されていたエリアに今では

等の一時停止シンボルが表示されているか否かをアナライザ３０６で判別することで行う。そして、コンテンツ１０４の再生が自動的に開始されることの検出は、ユーザその他の情報源からクローラ３００への入力に基づく推定で行う。 Also, some web pages that use the media player start playback of media content by the player when the page is fully or partially loaded on the web browser (the player is set as such). is there. The image analyzer 306 detects that playback of the media content 104 has started, for example, detects motion in the media content display area 202, detects sound, and detects that the play / pause button 204 is in the pause position. Or by using a prior knowledge that the page 100 is a page for automatically starting the reproduction of the content 104. The motion in the area 202 is detected by analyzing a plurality of web page images and detecting a change in the area 202, for example. For example, the web crawler 300 detects the audio format output from the web browser 302 to an audio device (not shown) (virtual or real). The detection that the play / pause button 204 is at the pause position is, for example, until then.

In the area where playback symbols such as

The analyzer 306 discriminates whether or not a pause symbol such as is displayed. Then, detection that the reproduction of the content 104 is automatically started is performed by estimation based on an input from the user or other information sources to the crawler 300.

  On the contrary, when the reproduction of the media content 104 does not automatically start or when the reproduction is stopped or paused, the web crawler 300 executes steps for reproducing the content 104. Specifically, after an operator of the media player 102 is identified, media player operator information and playback information are sent from the image analyzer 306 to the media player controller 308, and commands based on these information are sent from the controller 308 to the web browser. By sending it to 302, the reproduction of the content 104 is started.

  This media content reproduction will eventually end. The image analyzer 306 indicates that the reproduction has ended, a reproduction symbol is displayed on the reproduction / pause button 204, the reproduction button 206 is visible, and the progress bar indicator 210 is at the end of the progress bar 208 (for example, the right end). ) In the media content display area 202 in which the voice has not been detected for a predetermined time, the countdown timer 212 is 0:00, and the elapsed time indicator and the total time indicator have the same value. Detection is performed by discriminating one or more of the conditions such as not being seen. When the analyzer 306 detects that the reproduction has ended, the analyzer 306 sends a signal indicating that to the signature generator 310. In response, generator 310 completes the execution of the steps for generating a digital signature for that media content 104.

  Media content playback may be paused. It is only paused when a command to that effect is issued from the media player controller 308. For this reason, the web crawler 300 defines a pause variable so that the value changes when playback of the media content 104 is paused by a command from the controller 308 and when the pause is released. In that case, refer to the value. Note that the playback of the content 104 is paused by a command from the controller 308 because, for example, the signature generator 310 can generate only one signature for media content at a time, but a plurality of media on the web page 100. This is a case where there are players 102 and 106.

  Media content playback may also be interrupted until the data buffering for that media content 104 has settled. The data is buffered so that the streaming media playback is not interrupted. The buffering in the media player is usually executed in consideration of the connection speed, total playback time, remaining playback time, and the like. The length of the buffering is determined by estimating the extent to which the media can be played back before the end of playback can catch up with the downloaded data. For example, if the media content 104 to be played is a 60 second media clip, and the media player 102 can download the 55 seconds behind the content 104 within 55 seconds, the player 102 buffers the media content 102 before playing it. What is necessary is just to set the length to be 5 seconds. However, there may be a delay in the data connection for obtaining the content 104. In such a case, before reaching the end of the content 104, all the data permitted by the player 102 will be reproduced. In this case, the player 102 cannot resume the reproduction of the content 104 without buffering the data of the content 104 to some extent. The player 102 notifies the user that he is in such a state by displaying the word “Buffering” (buffering) in the media content display area 202 (see FIG. 7 described later).

  The image analyzer 306 indicates that the word “Buffering” is displayed in the media content display area 202 or the operator area, the sound has been interrupted for a predetermined time or more, and the end of the progress bar 208 has been reached. However, the progress bar indicator 210 has not been moved for a predetermined time, the data buffering is in progress, for example, displayed on the website using a pop-up window, etc. By detecting a plurality of types, it is detected that data buffering related to the media content 104 is in progress.

  Note that when data buffering is in progress and the media content 104 is not played back, the audio that makes up the content 104 is not played back, so the audio output of the web browser 302 is monitored within the web crawler 300. Thus, it is possible to acquire additional information about the state of the media player 102 and check whether or not buffering of data related to the content 104 is in progress. Similarly, the progress bar indicator 210 is monitored and its movement is detected by comparing, with the image analyzer 306, a plurality of web page images (substantially) continuously sent from the image generator 304. Can do. Further, among the displays made on the web page 100 and the player 102, one type or a plurality of types of display indicating that data buffering related to the content 104 is in progress, for example, a display using a pop-up window is examined. In this way, it is possible to determine the data buffering state of the content 104 during reproduction.

  Media content playback may be stopped. The web crawler 300 indicates that a command to press the stop button is issued from the media player controller 308, that the progress bar indicator 210 has not been moved for a predetermined time or more, and that the progress bar indicator 210 is in progress. It is detected by detecting one or more of being at the end of 208 and the voice output of the web browser 302 being interrupted for a predetermined time or more.

  For example, the web crawler 300 detects that the playback of the media content 104 has been successfully started but has not ended even after the expected time has elapsed, thereby causing a failure in the playback. Detect what is happening.

  Note that the example shown in FIG. 3 is merely an example. Those skilled in the art (so-called persons skilled in the art) will appreciate that other configurations are possible and beneficial. For example, the present invention can be implemented by executing a software application capable of identifying and monitoring media content on a web page on a personal computer (PC) owned by a media research panelist. In this form, one or more panelists using the PC can refer to the result of identification / monitoring by the software.

  FIG. 4 shows an example of the media player 400 waiting for the start of playback of the media content 402. The content 402 in the figure is a 2 minute (2:00) long video clip with audio. The player 400 includes a play / pause button 404, a stop button 406, a volume operator 408, a rewind button 410, a fast-forward button 412, an elapsed time indicator 414, a total time indicator 416, a progress bar 418, A progress bar indicator 420 is provided. The present invention can be implemented with other configurations, for example, a configuration in which the elapsed time indicator 414 and the total time indicator 416 are replaced with the remaining time indicator 212 in FIG.

  As described above, the image analyzer 306 detects that the playback of the media content 402 has not started by monitoring the playback / pause button 404, the elapsed time indicator 414, and the progress bar indicator 420. In this figure, a playback symbol is displayed on the playback / pause button 404, and the display by the elapsed time indicator 414 is “0:00”, that is, a display indicating that the content 402 is (almost) the beginning. The progress bar indicator 420 is located at the beginning (left end) of the progress bar 418.

  FIG. 5 shows an example of a media player 400 that is playing the media content 402. A player 400 in the figure is configured to be able to reproduce the content 402, and is accompanied by operators 404 to 420 similar to those in FIG. In the illustrated example, the reproduction of the content 402 is in progress, and one minute (1:00) has elapsed since the start. The image analyzer 306 detects that the content 402 is being played back by the player 400, detects that a pause symbol is displayed on the play / pause button 404, which detects a difference between images constituting the content 402. , Detecting that the time displayed on the elapsed time indicator 414 is not 0:00 and not equal to the total playback time, detect a change in the position of the progress bar indicator 420, execute any combination thereof, It detects by the method of etc. A method for detecting an audio output related to the content 402 may be used in combination with one or more of the above-described manipulator-based reproduction state detection methods.

  FIG. 6 shows an example of the media player 400 that has finished playing the media content 400. The player 400 is associated with the content 402 and is accompanied by user-oriented controls 404 to 420 similar to those shown in FIG. The image analyzer 306 detects that the reproduction of the content 402 by the player 400 has been completed, and a reproduction / pause button 404 that detects that no change between images is seen in the content 402, the progress bar indicator, or both. Detect that the playback symbol is displayed. Detect that the elapsed time indicator 414 and the total time indicator 416 display the same time. Detect that the audio output has been interrupted for a predetermined time. Detecting through analysis on individual or series of web page images, such as performing any combination thereof.

  FIG. 7 shows an example of the media player 400 that is buffering data related to the media content 402. The player 400 is associated with the content 402 and is accompanied with user-oriented controls 404 to 420. Since buffering of the data related to the content 402 or a part thereof is in progress, the player 400 displays the alert 422 including the word “Buffering” indicating that buffering is being performed. The part to be displayed may be on the player 400 or a part thereof, or may be another place in the web page 100. As shown in the figure, since the buffering progress is added to the alert 422, the user can watch the progress display, and the buffering of data by the player 400 is finished and the reproduction of the content 402 is started or resumed. You can know the approximate time to get ready. The image analyzer 306 detects the word “Buffering” or similar alert 422 from the player 400 or the web page 100 that the player 400 is proceeding with data buffering for the content 402, and the audio output is interrupted for a predetermined time. Other indications from the website that data buffering by the player 400 is in progress, detecting that the position of the progress bar indicator 420 has not changed over time. It detects by the method of detecting, performing those arbitrary combinations, etc.

  FIG. 8 shows an example flow 800 of a procedure for playing back and playing back the media content 104 referenced on the web page 100. The procedure 800 in this figure can be implemented by various types of processors, controllers or processing devices. For example, the procedure 800 of this figure is described by a group of instruction codes, the codes are stored / loaded in a tangible medium such as a flash memory or a random access memory (RAM), and a corresponding processor such as FIG. 9 is referred to. It can be executed by a processor 912 in a processor system 900 described later. In addition, the procedure 800 in this figure includes an ASIC, a PLD (programmable logic device), a FPLD (field programmable logic device), a discrete logic circuit, hardware, firmware, manual operation, and any combination thereof (for example, firmware, Software, hardware, or a combination thereof). In the following description, the procedure 800 of this figure is referred to, but various other procedures that can realize the web crawler 300 shown in FIG. 3 can be proposed. For example, the execution order of the steps is changed, the steps are changed, deleted, divided or combined, or any combination thereof. In addition, the procedure of this figure can be executed in parallel or sequentially using a plurality of processing threads, processors, devices, circuits, and the like.

  In this procedure 800, first, the web crawler 300 loads one of one or a plurality of web pages registered in the web page list (step 802). This list is a list of web pages that designate or refer to one or more types of media players and media content to be identified. What is loaded is one of such web pages, for example, a web page 100 designating or referring to media player 102 or 108 and media content 104 or 110 as shown in FIGS. 1A, 1B and 2. The crawler 300 uses the image generator 304 that is a part of the crawler 300 to generate a web page image, that is, an image that will appear when the web page 100 is displayed for a user on a display device or the like (step 804).

、一時停止

、停止

、早送り

、巻き戻し

、ボリューム

、プログレスバー、プログレスバー指示子、再生タイマ等のシンボルを識別する。アナライザ３０６は、更に、それら操作子の付近にある画像エリアをサーチすることでそのメディアエリアを識別する。これに代え又はこれと共に、ウェブクローラ３００の指示でメディアを再生させ、それにより得られる複数枚のウェブページ画像をアナライザ３０６で相互比較することで、画像のうち変化が生じているエリアを検出する、という手法を使用してもよい。本発明は、本明細書で例示する画像処理技術に限らず、他の画像処理技術乃至手法又はそれに従い修正した画像処理技術乃至手法で実施することもできる。 The image analyzer 306 analyzes the web page 100 based on the image, thereby identifying the media area exemplified as the media content display area 202 in FIG. 2 and all the user operators associated therewith (step 806). . The image analyzer 306 detects a pattern that matches a symbol known as a user-oriented operator in accordance with an edge detection algorithm, so that such a user-oriented operator, for example, a reproduction is reproduced.

,pause

,Stop

Fast forward

,Rewind

,volume

Identify symbols such as progress bar, progress bar indicator, and playback timer. The analyzer 306 further identifies the media area by searching for an image area in the vicinity of these controls. In place of this, or in addition to this, the media is played according to the instruction of the web crawler 300, and a plurality of web page images obtained thereby are compared with each other by the analyzer 306, thereby detecting an area where a change has occurred in the images. The method of, may be used. The present invention is not limited to the image processing technique exemplified in the present specification, and may be implemented by other image processing techniques or techniques or image processing techniques or techniques modified in accordance therewith.

  When a plurality of media players are used on a certain web page 100 and different identification target media contents are associated with each player, the image analyzer 306 displays the media of each of the plurality of types of players. The content display area 202 and the operator are identified. In order to avoid conflicts or conflicts when the signature generator is generating digital signatures for those players, the media player controller 308 instructs the web browser to stop one or more of those players. Give a directive.

  When the media content display area 202 and the operator for the user are identified (step 806), the image analyzer 306 determines whether or not the reproduction of the media content 104 has already been started (step 808). For example, when the reproduction of the content 104 is started by the instruction of the web crawler 300 as described above, the analyzer 306 determines that the reproduction has been started. Conversely, if the crawler 300 has not issued an instruction to play the content 104, that is, an instruction pretending to press the play button, the analyzer 306 determines whether or not the playback of the content 104 has started. Preliminary knowledge regarding searching, detecting audio, checking whether the play / pause button 204 is in the pause position, or whether the web page 100 is a page for automatically playing the content 104 as described above. It is determined by using. If it is determined that the reproduction of the content 102 by the media player 102 has not been started, the crawler 300 gives a command to the web browser 302 using the built-in media player controller 308 to start the reproduction of the content 104 ( Step 810).

  If playback of media content 104 has already begun, image analyzer 306 analyzes the series of web page images and provides media information to signature generator 310. Generator 310 generates a digital signature for content 104 based on the received media information (step 812). For example, when the content 104 is composed of audio and moving images, the generated digital signature is based on one or both of the audio and moving images.

  The image analyzer 306 monitors various states of the media player 102 used in parallel with analyzing a series of web page images and supplying media information to the signature generator 310. Based on the monitoring result, the analyzer 306 first determines whether or not the reproduction of the media content 104 is stopped (step 814). For example, the word “Buffering” is displayed in the media content display area 202 or the operator area, the sound is interrupted for a predetermined time, and the progress bar indicator 210 has not moved for a predetermined time, The playback symbol is displayed on the playback / pause button 204, the playback button 206 is visible, the progress bar indicator 210 is at the end of the progress bar 208, for example, the right end, and the countdown timer 212 is set to 0: “00” is displayed, the elapsed time indicator 414 and the total time indicator 416 indicate the same value, the motion in the area 202 is not seen, and the reproduction of the content 104 is stopped. When it detects that the sign is provided by the website, To determined that the reproduction of the content 104 is stopped. On the other hand, if the reproduction of the content 104 has not stopped, the process shown in step 812 is executed. That is, web page image analysis by the analyzer 306 and media information supply to the generator 310 are re-executed.

  When the reproduction of the media content 104 is stopped, the image analyzer 306 determines whether data buffering by the media player 102 is in progress for the content 104 based on the monitoring result (step 816). For example, the word “Buffering” is displayed in the media content display area 202 or the operation element area, the sound is interrupted for a predetermined time, and the progress bar indicator 210 is at a position other than the end of the progress bar 208. The determination is made by detecting that it has been stationary for a certain amount of time, other indications that data buffering is in progress, from the website, any combination, etc. . If the analyzer 306 determines that data buffering by the player 102 is in progress for the content 104, the operation of the analyzer 306 returns to step 812 to determine whether data buffering for the content 104 is in progress. The determination is re-executed.

  Conversely, if it is determined that data buffering by the media player 102 is not in progress or buffering of data related to the media content 104 has ended, the image analyzer 306 finishes playing the content 104 by the player 102. It is determined whether or not (step 818). If the reproduction of the content 104 by the player 102 has not been completed, the process returns to step 816, and a process for generating a digital signature for the content 104 is executed again.

  When the reproduction of the media content 104 is completed, the web crawler 300 identifies the content 104 by referring to the digital signature database with the digital signature of the content 104 (step 820). Alternatively, a method of storing the digital signature in the web crawler for later reference or transmitting the digital signature to the outside for identification may be used. In the illustrated web crawler 300, the media content identifier 316 shown in FIG.

  The web crawler 300 further determines whether other media players are used on the web page 100 (step 822). If there is still a media player related to the media content for which the digital signature has not been generated, the process returns to step 808 to determine whether or not the media player 102 has already started playing the media content. If there is no media player related to such media content, the process proceeds to step 824, where it is determined whether there is no web page remaining on the web page list. If there are web pages remaining, web crawler 300 selects the next web page from the list (step 826) and returns to step 802 to load the web page. If no web page remains on the web page list (step 824), the process is terminated.

  FIG. 9 illustrates an example processing system 900 that can implement the web crawler 300 illustrated in FIG. 3 and perform the procedure 800 illustrated in FIG. In this processing system 900, a processor 902 to which a memory such as a RAM 904, a ROM (read only memory) 906, and a flash memory 908 is connected is connected to an interface such as a bus 912, and other members are further connected to the bus 912. It has a configuration. In the figure, connected to the bus 912 are members such as an input device 914, a display device 916, a mass storage device 918, a removable storage device drive 920, and a network adapter 922. The drive 920 uses a removable storage medium 924 suitable for the drive 920 among magnetic recording media, optical recording media, and the like. A network adapter 922 connects the system 900 to an external network 926.

  Such a processing system 900 can be realized by an information processing apparatus such as an existing desktop PC, notebook PC, or workstation. The processor 902 includes a microprocessor represented by the Intel (registered trademark) Pentium (registered trademark) family, the Intel (registered trademark) Itanium (registered trademark) family, the Intel (registered trademark) XScale (trademark) family, and the like. Various types of processing units can be used. As the memories 904, 906 and 908 connected to the processor 902, any type of memory device can be used as long as it has a storage capacity suitable for the storage performance required for the system 900. For example, the nonvolatile memory used as the flash memory 908 may be of a type that can be accessed and erased in units of blocks.

  As the input device 914, a keyboard, a mouse, a touch screen, a track pad, a barcode scanner, or the like can be used. Others can be used as long as they can be used to supply information from the user to the processor 902.

  As the display device 916, an LCD (Liquid Crystal Display) monitor, a CRT (Cathode Ray Tube) monitor, or the like can be used. Any other device may be used as long as it can function as an interface between the processor 902 and the user. The display device 916 in the figure also includes various hardware necessary for the cooperation between the display screen and the processor 902.

  As the mass storage device 918, a magnetic recording medium, an optical recording medium, a solid storage medium, or the like that is readable to the processor 902 such as a conventional hard disk drive can be used.

  As the removable storage device drive 920, for example, an optical drive such as a CD-R (compact disk-recordable) drive, a CD-RW (compact disk-rewritable) drive, or a DVD (digital versatile disk) drive can be used. Alternatively, other types of drives such as a magnetic medium drive, a solid-state USB (universal serial bus) memory drive, and the like can be used. As the removable storage medium 924, one that is paired with the drive 920, that is, one that can be used with the drive 920 is selected. For example, if the drive 920 is an optical drive, the medium 924 is a corresponding one of optical disks such as a CD-R disk, a CD-RW disk, and a DVD disk. On the other hand, if the drive 920 is a magnetic medium drive, the medium 924 is a corresponding one of magnetic recording media such as diskettes.

  As the network adapter 922, for example, an Ethernet (registered trademark) adapter, a wireless local area network (LAN) adapter, a telephone modem, or the like is used. Any other type of device can be used as long as it can perform communication via a network between the processing system 900 and another processing system. The external network 926 is various networks that can communicate with the processing system 900, such as a LAN, a wide area network (WAN), and a wireless communication network. Examples include the Internet, intranets, ad hoc networks, and the like.

  The present invention has been described above by taking the system including software or firmware and hardware that executes the software as an example. However, since this configuration is only an example, the present invention is not limited to a system having such a configuration. Please keep this in mind. For example, some or all of the hardware components and software components described above may be implemented exclusively in hardware, implemented exclusively in software, implemented exclusively in firmware, or any of hardware, firmware, and software. It can also be realized by a combination of That is, as can be readily appreciated by those skilled in the art, the systems, methods and media illustrated herein are not the only forms that can implement the systems, methods and media of the present invention. Therefore, the technical scope of the present invention is not limited to the method, apparatus and medium exemplified in the present application, but all methods and apparatus which fall literally or equivalently within the scope of the appended claims. And media.

Claims (81)

Loading a web page using a media player and media content;
Identifying an operator associated with the media player by processing an image of the web page;
Presenting the media content using the operator;
A method for presenting media content in a web page.   The media content presentation method in a web page according to claim 1, further comprising a step of identifying the position of the media content in the web page by processing the image.   2. A method for presenting media content in a web page, wherein the media content is presented according to the position.   The media content presentation method in a web page according to claim 1, further comprising a step of determining the state of the media player by processing the image.   5. The method of presenting media content in a web page according to claim 4, wherein one of the states is media content playback, media content playback paused, media content playback stopped, media content playback end, media content data. A method for presenting media content in a web page during buffering or media player error occurrence.   The method for presenting media content in a web page according to claim 1, further comprising: generating a digital signature for the media content by processing the media content.   7. The method for presenting media content in a web page according to claim 6, wherein the above-described processing for the media content includes processing for one or a plurality of images constituting the media content. Method.   2. The method for presenting media content in a web page according to claim 1, wherein the media content presentation using the operation element includes a presentation according to a state of the operation element.   The method for presenting media content in a web page according to claim 1, further comprising the step of generating one or more images of the web page.   The method for presenting media content in a web page according to claim 1, wherein the above-described processing for the image includes edge detection.   The media content presentation method according to claim 1, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, A method of presenting media content in a web page, which is a progress bar, media display size operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   The method for presenting media content in a web page according to claim 1, further comprising a step of starting reproduction of the media content.   2. The method for presenting media content in a web page according to claim 1, wherein the media content includes audio, video, or both.   The method for presenting media content in a web page according to claim 1, further comprising the step of identifying the media content based on a digital signature.   The method for presenting media content in a web page according to claim 1, further comprising the step of determining whether or not reproduction of the media content has started by processing an image of the web page. . An image generator that generates images based on web pages that use media content;
An image analyzer for identifying a media player and an operator associated therewith based on the image, and further providing information necessary for presenting the media content using the operator;
An in-web page media content presentation device comprising:   The media content presentation apparatus in a web page according to claim 16, further comprising a web browser for loading the web page.   18. The media content presentation apparatus in a web page according to claim 17, further comprising a media player controller that sends a command to the web browser that the operation element is activated.   The media content presentation device in a web page according to claim 16, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, A media content presentation device in a web page which is a progress bar, media display size operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   The media content presentation device in a web page according to claim 16, wherein the image analyzer identifies a state of the media player related to the media content.   17. The media content presentation apparatus in a web page according to claim 16, further comprising a signature generator that generates a digital signature based on the media content. Loading a web page using a media player and media content;
Identifying an operator associated with the media player by processing an image of the web page;
Presenting the media content using the operator;
A medium in which machine-readable instructions for causing a machine to execute are stored at the time of manufacture.   23. The medium of claim 22, wherein the machine readable instructions group causes the machine to identify the location of the media content on the web page by processing the image.   23. The medium of claim 22, wherein the machine readable instructions group cause the machine to process the media content and generate a digital signature for the media content.   23. The medium according to claim 22, wherein the machine-readable instruction group causes the machine to determine the state of the media player by processing an image of the web page.   26. The medium of claim 25, wherein one of the states is media content playback, media content playback paused, media content playback stopped, media content playback end, media content data buffering or media player The medium where the error is occurring.   23. The medium according to claim 22, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, a progress bar, and a media display size. A medium that is an operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   23. The medium of claim 22, wherein the machine readable instructions group causes the machine to start playing the media content.   24. The medium of claim 22, wherein the media content includes audio, video, or both.   23. The medium of claim 22, wherein the machine readable instructions group cause the machine to identify the media content based on a digital signature.   23. The medium according to claim 22, wherein the group of machine readable instructions causes the machine to determine whether or not the reproduction of the media content has started by processing the image of the web page. Loading a web page that uses a media player;
Detecting an operator associated with the media player by processing an image of the web page;
Detecting the state of the media player by processing the image;
Starting the reproduction of the media content by the media player based on the detection result of the operation element, the state or both;
A method of reproducing media content in a web page having:   33. The method for reproducing media content in a web page according to claim 32, wherein the state is detected based on the operation element.   33. The method for reproducing media content in a web page according to claim 32, further comprising a step of detecting a type of media player used in the web page.   33. The method for reproducing media content in a web page according to claim 32, further comprising the step of generating a digital signature based on the media content.   33. The method for reproducing media content in a web page according to claim 32, further comprising the step of determining whether or not playback of the media content has started.   33. The method for reproducing media content in a web page according to claim 32, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, A method of reproducing media content in a web page, which is a progress bar, media display size operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   33. The media content playback method in a web page according to claim 32, wherein one of the above states is media content playback, media content playback paused, media content playback stopped, media content playback end, media content data A method of reproducing media content in a web page during buffering or media player error occurrence.   33. The method for reproducing media content in a web page according to claim 32, wherein the media content includes audio, video, or both. A web browser that loads a web page and activates a media player for use on the web page;
An image generator for generating an image of the web page;
An image analyzer that identifies the media player and its media content display area and controls based on the image;
An in-web page media content playback and playback control device.   41. The media content playback / playback control apparatus for web pages according to claim 40, wherein the media content playback / playback control device for web pages comprises a media player controller for sending a command related to the operation element to a web browser.   41. The media content reproduction / playback control apparatus for web pages according to claim 40, comprising a media player controller for controlling the media player in accordance with information relating to the operation element.   41. The media content playback and playback control apparatus according to claim 40, wherein one of the operators is a playback button, a pause button, a stop button, a playback / stop button, a playback / pause button, a volume. Media content playback / playback control device in a web page, which is an operator, progress bar, media display size operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   41. The in-web page media content playback / playback control apparatus according to claim 40, wherein the image analyzer identifies the state of the media player related to the media content.   45. The media content playback and playback control apparatus according to claim 44, wherein one of the media player states is media content playback, media content playback paused, media content playback stopped, media content A media content playback / playback control device in a web page that is being played back, media content data buffering or media player error is occurring Loading a web page;
Detecting the type of media player used in the web page;
Detecting one or more operators associated with the media player;
Starting playback of media content by the media player;
Detecting the state of the media player during playback of the media content;
A medium in which machine-readable instructions for causing a machine to execute are stored at the time of manufacture.   48. The medium of claim 46, wherein the machine readable instructions group cause the machine to generate a digital signature based on the media content.   48. The medium of claim 46, wherein the machine readable instructions group cause the machine to determine whether playback of the media content has begun.   47. The medium according to claim 46, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, a progress bar, and a media display size. A medium that is an operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.   50. The medium of claim 49, wherein one of the states of the media player is media content playback, media content playback paused, media content playback stopped, media content playback end, media content data buffering. Or a medium in which a media player error is occurring.   48. The medium of claim 46, wherein the media content includes audio, video, or both. A method for controlling a plurality of media players each associated with media content comprising:
Generating an image of a web page;
Detecting first and second media players and first and second media content by processing the image;
Detecting at least one operator associated with each of the first and second media players by processing the image;
Playing back the first media content;
Playing back the second media content;
Having a method.   53. The method of claim 52, wherein the first and second media content are not played simultaneously.   53. The method of claim 52, comprising the step of stopping playback of the second media content in accordance with the first media content.   53. The method of claim 52, comprising generating a digital signature based on the first or second media content.   53. The method of claim 52, comprising loading the web page.   53. The method of claim 52, wherein the first and second media players are associated with the web page.   53. A method according to claim 52, comprising the step of detecting the state of the first or second media player by analyzing the image.   59. The method of claim 58, wherein one of the states is media content playback, media content playback paused, media content playback stopped, media content playback end, media content data buffering or media player. The method in which an error is occurring.   53. The method of claim 52, wherein the media content includes audio, video, or both.   53. The method according to claim 52, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, a progress bar, and a media display size. A method that is an operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator. Generating an image of a web page;
Detecting the first and second media players and the first and second media contents by analyzing the image;
Detecting at least one operator associated with each of the first and second media players by analyzing the image;
Playing back the first media content;
Playing back the second media content;
Media manufactured to control a plurality of media players each associated with a media content.   64. A medium according to claim 62, wherein said first and second media contents are not reproduced simultaneously.   64. The medium of claim 62, wherein the machine readable instructions group cause the machine to stop playing the second media content.   63. The medium of claim 62, wherein the machine readable instructions group cause the machine to generate a digital signature based on the first or second media content.   64. The medium of claim 62, wherein the group of machine readable instructions causes the machine to load the web page.   64. The medium of claim 62, wherein the first and second media players are associated with the web page.   64. The medium of claim 62, wherein the machine readable instructions group causes the machine to detect the state of the first or second media player by analyzing the image.   69. The medium of claim 68, wherein one of the states is during media content playback, media content playback paused, media content playback stopped, media content playback ended, media content data buffering or media player The medium where the error is occurring.   64. The medium of claim 62, wherein the media content includes audio, video or both.   63. The medium according to claim 62, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, a progress bar, and a media display size. A medium that is an operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator. An apparatus for controlling a plurality of media players each associated with a media content,
An image generator that generates images of web pages;
An image analyzer that detects the first and second media players and the first and second media contents by analyzing the image and detects at least one operator associated with each of the first and second media players. When,
A media player controller for reproducing the first and second media contents;
A device comprising:   75. The apparatus of claim 72, wherein the first and second media content are not played simultaneously.   75. The apparatus of claim 72, wherein the media player controller stops playback of the second media content.   75. The apparatus of claim 72, comprising a signature generator that generates a digital signature based on the first or second media content.   75. The apparatus of claim 72, comprising a web browser that loads the web page.   75. The apparatus of claim 72, wherein the first and second media players are associated with the web page.   75. The apparatus of claim 72, wherein the image analyzer detects the state of the first or second media player by analyzing the image.   79. The apparatus of claim 78, wherein one of the states is during media content playback, media content playback paused, media content playback stopped, media content playback ended, media content data buffering or media player A device that is experiencing an error.   75. The apparatus of claim 72, wherein the media content includes audio, video, or both.   75. The apparatus according to claim 72, wherein one of the operators is a play button, a pause button, a stop button, a play / stop button, a play / pause button, a volume operator, a progress bar, and a media display size. A device that is an operator, fast forward button, rewind button, elapsed time indicator, countdown indicator or elapsed time / total time indicator.

Images