JP2015233182A - Moving image information acquisition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a moving image information acquisition system which can acquire information on a moving image under reproduction, only by capturing the moving image under reproduction.SOLUTION: A moving image information acquisition system includes a mobile terminal with a camera, a first server (a matching server 104), and a second server (a video server 106). Retrieval data is transmitted to the first server using one scene of a moving image which is acquired by capturing with the mobile terminal to make an inquiry (S608). The first server, on the basis of a retrieval result, makes an inquiry with the second server (S610) to acquire moving image data (S612), and transmits information on the moving image which was inquired from the mobile terminal (S613).

Description

  The present invention relates to a moving image information acquisition system, and more particularly to a technique for searching and acquiring information related to a moving image from a scene of the moving image.

  Accompanying the widespread use of mobile terminals such as smartphones, it has become common to acquire images and the like using a camera provided in the mobile terminal. Using such a function, for example, conventionally, there is known a technique of searching for, acquiring, and using information related to an image taken using a camera or the like (Patent Document 1 or the like).

JP 2013-222406 A

  Conventionally, there is a request for acquiring information regarding a moving image being played back. As the information on the moving image, for example, if the moving image is a television broadcast, information on the television program, information on an article or a person reflected in the broadcast content, and the like can be given.

  Usually, when such information is acquired, it is often acquired using an Internet search or the like. In this case, information that the user knows about the moving image and its contents is a PC (personal computer) or a portable terminal. It is necessary to input to.

  However, such an input takes time and problems, and there arises a problem that the search result is not sufficient when the user does not sufficiently grasp information that can be input.

  In view of this, the present invention provides a technique that enables acquisition of information relating to a moving image that is being reproduced simply by shooting the moving image that is being reproduced.

In order to solve the above problems, the present invention has the following configuration. That is, a video information acquisition system including a mobile terminal equipped with a camera, a first server, and a second server,
The portable terminal is
Recognizing means for recognizing the moving image being played by the playback device via the camera;
An acquisition means for acquiring an image from the recognized moving image being reproduced via the camera;
First generation means for generating search data using the image acquired by the acquisition means;
Means for transmitting the search data generated by the first generation means to the first server, and acquiring moving image data from the first server as a response;
The first server is
Second generating means for generating search data using an image that constitutes the same moving image as the moving image to be played by the playback device;
Search data storage means for associating the search data generated by the second generation means with the information about the image used for generating the search data in the first storage unit;
Receiving means for receiving search data from the portable terminal;
Search means for searching search data matching the search data received by the receiving means from the first storage unit, and acquiring information associated with the matching search data;
Using the information acquired by the search means to make an inquiry to the second server, and acquiring the moving image data as a response;
Transmitting means for transmitting the moving image data acquired by the acquiring means to the mobile terminal;
Have
The second server is
Moving image data storage means for acquiring the same moving image as the moving image to be played back by the playback device and storing it in a second storage unit;
When an inquiry is received from the first server, moving image data corresponding to the information specified in the inquiry is specified from the second storage unit, and the specified moving image data is used as a response to the inquiry. Response means for transmitting to one server.

  According to the present invention, it is possible to easily acquire information relating to a moving image being reproduced.

The figure which shows the example of the whole structure of the system which concerns on 1st embodiment. The figure which shows the hardware structural example of a portable terminal and various servers. The figure which shows the software structural example of the portable terminal and various servers which concern on 1st embodiment. The figure for demonstrating the moving image data which concerns on this invention. The figure which shows the structural example of the moving image data which concern on 1st embodiment. The figure which shows the process sequence which concerns on 1st embodiment. The flowchart of the process in the portable terminal which concerns on 1st embodiment. The flowchart of the process using the moving image data which concerns on 1st embodiment. The figure which shows the example of the whole structure of the system which concerns on 2nd embodiment. The figure which shows the software structural example of the various servers which concern on 2nd embodiment. The figure which shows the process sequence which concerns on 2nd embodiment.

<First embodiment>
Embodiments according to the present invention will be described below with reference to the drawings. In addition, the structure shown below is an example and is not limited to this. In the present embodiment, the moving image being reproduced will be described using an example of television broadcasting, but the object as a moving image is not limited to this. Details will be described later.

[System configuration]
FIG. 1 is a diagram showing an example of the overall configuration of a moving picture information acquisition system according to the present invention. The system according to the present embodiment includes a mobile terminal 101, a television 102, a broadcast station 103, a matching server 104, an index server 105, a video server 106, and a service server 107. The portable terminal 101 is connected to the Internet 108 via a wireless network, and thereby connected to the various servers connected to the Internet 108 so as to be communicable.

  The mobile terminal 101 corresponds to, for example, a smartphone or a tablet terminal provided with a camera. The camera is capable of capturing a still image or a moving image according to an operation instruction from the user of the mobile terminal 101. The mobile terminal 101 also has a function of executing recognition processing for one or more frames constituting the moving image while capturing a moving image, and obtaining a unique value for the recognized frame.

  The television 102 is a device that can reproduce a moving image according to a television broadcast radio wave transmitted from the broadcast station 103. In the present embodiment, a description will be given of a television 102 that receives and reproduces television broadcast radio waves as an example of a device that reproduces moving images. However, the present invention is not limited to this, and for example, moving image distribution via the Internet 108 Alternatively, it may be a device that plays back video recorded on a DVD or the like.

  The Internet 108 is connected to the mobile terminal 101 and various servers regardless of wired / wireless, and serves as a communication path between the devices.

  In FIG. 1, only one mobile terminal and various servers are shown, but more devices may be connected so as to communicate with each other. In addition, various servers may be physically distributed by a plurality of information processing devices, and one server may be responsible for the functions of two or more servers.

  In the present embodiment, for example, television broadcasting is described. Therefore, it is assumed that there are as many broadcasting channels as the number of broadcasting stations, and broadcasting radio waves transmitted from each broadcasting station are transmitted to the television 102 (one broadcasting station is reproduced). It is assumed that the index server 105 and the video server 106 have received the moving image corresponding to the radio wave).

  The service server 107 provides various services in response to requests from the mobile terminal 101. Details of the functions of the various servers will be described later together with the processing flow.

[Hardware configuration]
FIG. 2 is a diagram illustrating an example of the hardware configuration of the mobile terminal 101 and various servers according to the present embodiment. Note that a conventional television 102 is used, and a description thereof is omitted here.

  FIG. 2A shows an example of the hardware configuration of the mobile terminal 101 according to the present embodiment. A CPU (Central Processing Unit) 201 controls the entire processing in the mobile terminal 101. For example, by reading and executing a program or data stored in a ROM (Read Only Memory) 203 or a storage 204 as a storage unit, each process described later as what the mobile terminal 101 performs is realized. A RAM (Random Access Memory) 202 is a storage unit that is accessed from the CPU 201 and is used as a storage location for programs and data read from the ROM 203 and the storage 204, and data generated in the process being executed. The ROM 203 is a nonvolatile storage unit. The storage 204 is a storage unit for storing a program, data, and the like for causing the CPU 201 to execute each process described later as performed by the mobile terminal 101.

  The display unit 205 is a display unit for displaying a captured image captured by the capturing unit 206 and a UI (User Interface) for the user. The display unit 205 is configured by, for example, a touch panel and can accept an instruction from the user. The imaging unit 206 is configured with, for example, a camera, and the user of the mobile terminal 101 can capture an image of the surrounding environment as necessary. In addition, when the photographing unit 206 is pointed at some object or the like, an image in a direction in which the photographing unit 206 faces is displayed on the display unit 205. The transmission / reception unit 207 is used when transmitting / receiving data to / from an external device (for example, various servers) via the network (Internet 108). These functional units are all connected to a common bus 208, thereby enabling data exchange.

  FIG. 2B shows an example of the hardware configuration of various servers (information processing apparatuses) according to this embodiment. Note that the matching server 104 according to the present embodiment does not include the I / F 221 and the television broadcast receiving unit 223. The CPU 211 controls the entire processing in the information processing apparatus. For example, the processing according to the present invention performed by various servers is realized by reading out and executing the program and data according to the present invention stored in the ROM 213 or the like as the storage unit. The RAM 212 is a storage unit that is accessed by the CPU 211 and used as a storage location for temporary processes such as programs and data read from the ROM 213 and data generated during the process being executed. The ROM 213 is a nonvolatile storage unit.

  The input I / F 214 is an interface for controlling the input device 215. The input device 215 is a device for receiving an input instruction from a user, and corresponds to a mouse, a keyboard, or the like. The display I / F 216 is an interface for controlling the display unit 217. The display unit 217 is a display for presenting various types of information to the user, and includes a CRT, a liquid crystal display, or the like. The external memory I / F 218 is an interface for controlling the external memory 219. The external memory 219 is a storage unit such as an HDD and stores various data. The network I / F 220 is an interface for connecting to a wired / wireless network (Internet 108) in order to communicate with an external device.

  The I / F 221 receives data related to the television broadcast from the television broadcast receiver 223. The television broadcast receiving unit 223 receives radio waves and data related to television broadcasts transmitted from the broadcast station 103, converts them into a predetermined data format, and provides them to the server. The function of the TV broadcast receiving unit 223 is equivalent to the function of the TV 102, and includes, for example, a tuner, a converter, and the like. Conversion from broadcast radio waves to video signals (NTSC, etc.), and moving image data to MP4, etc. Conversion to etc. Furthermore, the moving image data may be distributed to the server using a protocol such as RTSP (Real Time Streaming Protocol). Here, a known method is used, and detailed description thereof is omitted. Each element constituting the server is connected by a system bus 222, and data can be exchanged within the server.

  The internal structures of the mobile terminal and the server are not limited to this, and may include other components as necessary.

Software configuration
FIG. 3 is a diagram showing examples of software configurations of the mobile terminal 101 and various servers according to the present invention. Note that the software configuration shown below is an example, and may include other components.

  FIG. 3A shows an example of the software configuration of the mobile terminal 101. In the mobile terminal 101, an application (hereinafter simply referred to as “app”) 302 operates under the control of an OS (Operating System) 301, and processing of the mobile terminal 101 according to the present embodiment is performed by the application 302. And Note that the mobile terminal 101 may further include apps having other functions, and various apps can be arbitrarily installed / uninstalled.

  The application 302 includes a moving image recognition unit 303, an image extraction unit 304, a search data generation unit 305, a data transmission / reception unit 306, and a data processing unit 307. The moving image recognition unit 303 recognizes that a moving image is being played when the camera, which is the photographing unit 206 included in the mobile terminal 101, is held over the television 102 on which the moving image (here, TV broadcast) is played, A plurality of images of the moving image being reproduced are acquired via the camera. The image extraction unit 304 extracts one or a plurality of images from the images acquired by the moving image recognition unit 303. A plurality of images to be extracted here may be acquired at a predetermined frame rate, or a single image may be acquired. When a predetermined frame rate is used here, the frame rate is determined based on the relationship with the processing of the video server 106 described later.

  The search data generation unit 305 uses the image extracted by the image extraction unit 304 to generate video information search data (hereinafter referred to as “search data”). In this embodiment, FingerPrint generated by a hash function is used as the search data. However, if the method is to obtain a unique value for the image extracted by the image extraction unit 304 as FingerPrint, a hash function is used. It is not limited to the method used. Further, the hash function method is not particularly limited. The data transmission / reception unit 306 transmits various data to various servers according to the purpose, and receives the response. In the present embodiment, the data transmission / reception unit 306 transmits the search data generated by the search data generation unit 305 to the matching server 104 and acquires moving image data as a response. The data processing unit 307 performs processing using data received from various servers. An example of the processing here will be described later using a flowchart.

  FIG. 3B shows an example of the software configuration of the matching server 104. The data transmission / reception unit 311 receives a request from the mobile terminal 101 and returns a response. Specifically, a request including search data is received, and moving image data searched using the search data is returned. The matching processing unit 312 uses the search data included in the request from the mobile terminal 101 to search for the corresponding data from the data stored in the database (for example, the external memory 219) by the search data storage unit 313. To do. Then, the matching processing unit 312 transmits the search result to the moving image data inquiry unit 314.

  The search data storage unit 313 sequentially receives search data generated by the index server 105 using an image extracted from a moving image of a television broadcast, and stores and accumulates the data in a database. The moving image data inquiry unit 314 uses the search result acquired from the matching processing unit 312 to inquire the video server 106 about moving image data corresponding to the search result. Then, the moving image data inquiry unit 314 acquires moving image data as the response. Thereafter, the moving image data inquiry unit 314 transmits the acquired moving image data to the data transmission / reception unit 311 and causes the requesting mobile terminal 101 to transmit the acquired moving image data.

  FIG. 3C shows an example of the software configuration of the index server 105. The moving image data receiving unit 321 receives moving image data (television broadcast data) transmitted from the broadcasting station 103 and received by the television broadcast receiving unit 223. At this time, when there are a plurality of broadcasting stations as shown in FIG. 1, the moving image data receiving unit 321 receives the information uniquely indicating the broadcasting station that has distributed the data and moving image data. A configuration example of the moving image data according to the present embodiment will be described later with reference to FIG. Then, the moving image data reception unit 321 transmits the received various data to the image extraction unit 322.

  The image extraction unit 322 extracts a frame (image) from the moving image data acquired by the moving image data reception unit 321. A plurality of frames to be extracted here are acquired at a predetermined frame rate (predetermined time interval). Here, it is assumed that the predetermined frame rate is higher than the frame rate when a plurality of images are extracted on the mobile terminal 101 side. Then, the image extraction unit 322 transmits the extracted frame to the search data generation unit 323 in association with the information acquired from the moving image data reception unit 321 corresponding to the extracted frame.

  The search data generation unit 323 generates search data using the image extracted by the image extraction unit 322. The generation process here uses the same method as the search data generation unit 305 of the mobile terminal 101. Note that the search data generation unit 232 is not limited to the method using the hash function as long as it is a method for obtaining a unique value as FingerPrint for the image extracted by the image extraction unit 322. Then, the search data generation unit 323 associates the generated search data with information corresponding to the image received from the image extraction unit 322, and transmits the data to the search data providing unit 324. The search data providing unit 324 sequentially transmits the information received from the search data generation unit 323 to the matching server 104. The information transmitted here is stored in the database by the search data storage unit 313 of the matching server 104.

  FIG. 3D shows an example of the software configuration of the video server 106. The moving image data receiving unit 331 receives moving image data (television broadcast data) transmitted from the broadcasting station 103 and received by the television broadcast receiving unit 223. At this time, when there are a plurality of broadcast stations as shown in FIG. 1, the moving image data receiving unit 331 receives information uniquely indicating the broadcast station that distributed the data and moving image data. In this embodiment, it is assumed that the configuration is the same as that of the moving image data receiving unit 321 of the index server 105. Then, the moving image data reception unit 331 transmits the received various data to the moving image data storage unit 332.

  The moving image data storage unit 332 sequentially stores the data transmitted from the moving image data receiving unit 331 in the database. In response to a request from the matching server 104, the moving image data providing unit 333 identifies corresponding moving image data from the moving image data stored in the database and sends back the corresponding moving image data. A specific processing flow here will be described later.

  Note that the search data storage unit 313 of the matching server 104 and the moving image data storage unit 332 of the video server 106 may delete the data accumulated in the database after a predetermined time has elapsed. In this case, the corresponding information in each database (for example, data broadcast in the same time zone) is deleted so as to be consistent (so that only one data does not remain). Further, the amount of data stored for each broadcasting station may be varied. The database in which each server stores data is not limited to the storage unit in the server, and a separate storage device may be provided and stored and accumulated therein. In this case, each server may be configured to directly retrieve and acquire target data from the storage device instead of inquiring of other servers.

[Process flow]
A processing flow according to the present embodiment will be described.

(Description of video data)
First, moving image data handled in this embodiment will be described. FIG. 4 shows the concept of moving image data according to the present embodiment. In FIG. 4, the horizontal axis indicates time, and the arrangement of moving image data images (frames) at a certain time is shown. FIG. 4A shows continuous frames constituting a moving image distributed from the broadcast station 103. For example, in terrestrial digital broadcasting, it becomes 29.97 fps (frame per second).

  FIG. 4B is a series of frames in which the television broadcast receiving unit 223 of the video server 106 receives and acquires a moving image distributed from the broadcast station 103. In the present embodiment, all the broadcast data is acquired and held, and the same configuration as in FIG. When there are a plurality of broadcasting stations 103, data corresponding to each broadcasting station in the same time zone is held together with information indicating the time at which the broadcasting station was received (or distributed) and the broadcasting station.

  FIG. 4C shows a frame that is extracted by the image extraction unit 322 after the television broadcast reception unit 223 of the index server 105 receives the moving image distributed from the broadcast station 103. The frame extraction interval (fps) extracted here is determined by the accuracy of moving image detection, the accuracy of search, the balance with the processing load, or the like. Here, an example is shown in which frames are acquired at a rate of one third of the original data.

  FIG. 4D shows an image that is recognized and extracted by the application 302 (image extraction unit 304) of the mobile terminal 101 by holding the camera over the television 102 on which a moving image is played. Here, only one image is acquired. As described above, a plurality of images may be extracted at a predetermined extraction interval. In this case, it is assumed that the interval is longer than the extraction interval on the index server 105 side shown in FIG. This suppresses a decrease in search accuracy on the matching server 104 side.

  In (a) to (c) of FIG. 4, since each server receives data distributed directly from a broadcasting station, in addition to images, moving image information associated therewith is also acquired. On the other hand, since the mobile terminal 101 recognizes the reproduced moving image and acquires the image, only the image data is acquired.

(Data structure)
FIG. 5 is a diagram illustrating an example of a data configuration according to the present embodiment. The moving image data according to the present embodiment includes a plurality of images and moving image information defined at a predetermined frame rate. The moving image information according to the present embodiment has a channel ID, a channel name, and a moving image content name for uniquely identifying a broadcasting station (channel) that performs television broadcasting. Note that the information included in the moving image information is not limited to this, and may include information related to content such as performers and sponsors of the program, for example. Each image (frame) included in the moving image data has time information (time stamp). In the example of the television broadcast as in the present embodiment, the broadcast time is shown. In the case of a movie or the like, the time from the start of the movie may be indicated. Further, the information for each image is not limited to the time stamp, and for example, it may further include information such as a caption indicating a group of images.

(Processing sequence)
FIG. 6 shows a processing sequence between the apparatuses according to the present embodiment. The processing according to this processing sequence is realized by the CPU of each processing subject reading and executing the program according to the present embodiment stored in the storage unit such as the ROM.

  In S601, the video server 106 receives the moving image data distributed from the broadcast station 103 and records it in the database. The moving image data acquired here includes moving image information in addition to images as shown in FIG.

  In S <b> 602, the index server 105 receives the moving image data distributed from the broadcast station 103. The moving image data received here is the same as that received by the video server 106 in S601.

  In S603, the index server 105 extracts images at predetermined time intervals from the received moving image data. Then, the index server 105 generates search data (FingerPrint) for the extracted image using a predetermined hash function. The generation method related to FingerPrint used as an example in the present embodiment can be applied with a conventional technique, and detailed description thereof is omitted here.

  In step S <b> 604, the index server 105 transmits the generated FingerPrint and moving image information corresponding to the image that is the basis of the FingerPrint to the matching server 104.

  In S605, matching server 104 associates FingerPrint received from index server 105 with the corresponding moving image information, and sequentially accumulates them in the database.

  On the other hand, on the mobile terminal 101 side, the user activates the application 302 and holds the camera as the photographing unit 206 over the television 102. As a result, in S606, the mobile terminal 101 recognizes the moving image being played back (broadcast) on the television 102. Here, the moving image being reproduced by the television is the same as the moving image data received by the video server 106 or the index server 105 in S601 or S602.

  In step S607, the mobile terminal 101 extracts an image from the recognized moving image. The number of images extracted here may be one as shown in FIG. 4, or a plurality of images may be acquired at predetermined intervals. Then, the portable terminal 101 generates search data (FingerPrint) for the extracted image using a predetermined hash function.

  In step S <b> 608, the mobile terminal 101 transmits the generated FingerPrint to the matching server 104.

  In S609, the matching server 104 searches for a FingerPrint that matches the FingerPrint received from the mobile terminal 101 from among the FingerPrints accumulated in S605. If the matching FingerPrint is held in the matching server 104, the moving image information stored in the database in association with the matching FingerPrint is acquired. For example, in the case where the video information is a television program distributed from a broadcasting station as in the present embodiment, the channel ID (identifier) uniquely indicating the broadcasting station 103 and the time when the image (frame) is distributed ( Time stamp). Note that the moving image information may include other information.

  In S610, matching server 104 makes an inquiry to video server 106 using the specified moving image information.

  In S611, the video server 106 identifies corresponding moving image data from the accumulated moving images based on the channel ID and the time stamp included in the inquiry from the matching server 104. For example, if the channel ID and time stamp included in the inquiry from the matching server 104 are “X” and “T”, respectively, the frame delivered by the broadcast station indicated by the channel ID “X” at the time T is shown. Identified. Then, the video server 106 specifies moving image data (consecutive frames) for a predetermined time from the specified frame, and receives data including the specified frame and the specified moving image data from the matching server 104. It is determined as the video data for the inquiry. Here, the moving image data to be returned may be all moving image data in the time period corresponding to the inquiry from the matching server 104 among the data accumulated in the database by the video server 106 or the inquiry from the matching server 104. It is possible to extract a part of a frame group in a time zone according to the predetermined frame rate as a target. Note that the “predetermined time” is defined in advance, and for example, it may be moving image data that goes back one minute from the specified time.

  In S612, the video server 106 returns the moving image data specified in S611 to the matching server 104.

  In S613, the matching server 104 transmits the moving image data received from the video server 106 as a response to the request (FingerPrint) transmitted from the mobile terminal 101 in S608.

  In S614, the mobile terminal 101 receives the moving image data transmitted from the matching server 104 in S613, and performs processing using the received moving image data. An example of the processing here will be described later. Then, this processing flow ends.

[Handling of mobile devices]
FIG. 7 shows a flowchart of processing performed in the mobile terminal 101 according to the present embodiment. This flow corresponds to the processing of the mobile terminal 101 shown in FIG. This processing flow is realized by the CPU 201 reading and executing a program stored in the storage 204 or the like that is a storage unit.

  In step S <b> 701, the mobile terminal 101 activates the application 302 and directs the camera to the television 102 that is reproducing the moving image by the user, so that the moving image being reproduced (distributed by the broadcasting station 103 in this example). TV program).

  In S702, the mobile terminal 101 extracts an image from the recognized moving image at an arbitrary timing. In S703, the mobile terminal 101 generates search data (FingerPrint) from the extracted image using a predetermined method. In step S <b> 704, the mobile terminal 101 transmits the generated FingerPrint to the matching server 104.

  In S705, the mobile terminal 101 receives moving image data as a response in S704. In step S706, the mobile terminal 101 performs processing using the received moving image data. Any processing using moving image data may be used, and for example, it may be used in a function of another application included in the mobile terminal 101. In the present embodiment, an example of the processing here will be described with reference to FIG. Then, this processing flow ends.

[Usage example of video data]
Below, an example of the process in the portable terminal 101 which acquired moving image data is demonstrated. In the present embodiment, an example in which the mobile terminal 101 uses the acquired moving image data in cooperation with the service server 107 will be described.

  FIG. 8 shows a flowchart of processing performed in the mobile terminal 101 according to the present embodiment. This flow is an example of the process in S614 of FIG. This processing flow is realized by the CPU 201 reading and executing a program stored in the storage 204 or the like that is a storage unit.

  In S801, the mobile terminal 101 acquires moving image data (a plurality of continuous frames) transmitted from the matching server 104 in S613. In S802, the mobile terminal 101 extracts one or a plurality of images (frames) from the acquired moving image data. In step S803, the mobile terminal 101 causes the display unit 205 to display the extracted image or images on the display unit 205 so that the user can select them. The display method here may be displayed depending on the function of the application, for example.

  In S804, the mobile terminal 101 receives a selection of an image from the user, and acquires information associated with the selected image from the moving image data acquired in S801. The information associated here corresponds to the moving image information described above, and includes, for example, the channel ID, broadcasting time, program name, and the like of the moving image data. These pieces of information correspond to the information received from the matching server 104 in S613 of FIG.

  In step S805, the mobile terminal 101 transmits the image selected in step S804 and the information acquired in step S804 to the service server 107. Here, the service server 107 may be, for example, a server that provides an image posting service or a server that provides a storage service, and the service content is not limited. In addition, before transmitting an image, for example, when an image editing function is provided as a function of an application included in the mobile terminal 101, the image can be transmitted after editing the selected image using the function. You may do it.

  On the service server 107 side, arbitrary service processing is executed using the image and information received from the mobile terminal 101. For example, the received image and information may be published on a web page accessible by a third party such as an electronic bulletin board, or the received image and information may be transferred to a specific or designated transfer destination. The process of transferring to Then, the service server 107 returns the result of the service process to the mobile terminal 101.

  In step S807, the mobile terminal 101 receives a response to the image and information transmitted to the service server 107 in step S806, that is, the result of service processing performed by the service server 107 using the image and information. The mobile terminal 101 may display the content of the received response on the display unit 205. Then, this processing flow ends.

  As described above, according to the present embodiment, the user can easily acquire the partial moving image and related information of the moving image by simply recognizing the moving image being played back on the playback device such as the television with the mobile terminal.

<Second Embodiment>
In the present embodiment, a configuration for acquiring moving image data and related information with a configuration different from that of the first embodiment will be described.

[System configuration]
FIG. 9 shows an example of a system configuration according to this embodiment. In this embodiment, a proxy server 901 is provided instead of the video server 106. The hardware configuration of the proxy server 901 is the same as that of the matching server 104, and does not receive moving image data. On the other hand, the index server 105 holds an image extracted from moving image data and moving image information corresponding thereto in a database (not shown) when generating search data (FingerPrint).

Software configuration
FIG. 10 is a diagram illustrating an example of the software configuration of the index server 105 and the proxy server 901 according to the present embodiment. Note that description of overlapping parts is omitted. The configurations of the mobile terminal 101, the matching server 104, and the service server 107 are the same as those in the first embodiment, and a description thereof is omitted here.

  FIG. 10A shows an example of the software configuration of the index server 105 according to the present embodiment. An image data recording unit 1001 and an image data providing unit 1002 are added to the configuration of FIG. When the image extraction unit 322 extracts an image from moving image data, the image data recording unit 1001 records the extracted image and moving image information corresponding to the extracted image in a database in association with each other. When receiving the moving image information from the proxy server 901, the image data providing unit 1002 returns image data corresponding to the moving image information among the image data recorded by the image data recording unit 1001 to the proxy server 901 as a response.

  FIG. 10B shows an example of the software configuration of the proxy server 901 according to this embodiment. The inquiry reception unit 1011 receives an inquiry including moving image information from the mobile terminal 101. When the inquiry reception unit 1011 receives the inquiry, the inquiry reception unit 1011 transmits the moving image information included in the inquiry to the image data acquisition unit 1012. When receiving the moving image information from the inquiry receiving unit 1011, the image data acquiring unit 1012 acquires image data from the index server 105 using the moving image information. The image data acquired here corresponds to the image data recorded by the image data recording unit 1001 of the index server 105. That is, the image data providing unit 1002 of the index server 105 returns image data in response to a request from the proxy server 901. When the image data response unit 1013 acquires image data from the index server 105, the image data response unit 1013 returns the image data as a response to the mobile terminal 101. Here, it is assumed that the image data response unit 1013 returns not only one image data but also a plurality of continuous image data for a predetermined time back from the image data. The predetermined time here is determined in advance as in the first embodiment.

[Processing sequence]
FIG. 11 shows a processing sequence between the apparatuses according to the present embodiment. The processing according to this processing sequence is realized by the CPU of each processing subject reading and executing the program according to the present embodiment stored in the storage unit such as the ROM.

  In S1101, the index server 105 receives the moving image data distributed from the broadcast station 103. In S1102, the index server 105 extracts images at predetermined time intervals from the received moving image data. Further, the index server 105 records the extracted image data in a database (external memory 219 or the like). Further, the index server 105 generates search data (FingerPrint) for the extracted image data using a predetermined hash function. As in the first embodiment, the generation method related to FingerPrint used as an example in the present embodiment can be applied with the conventional technique, and detailed description thereof is omitted here.

  In step S <b> 1103, the index server 105 transmits the generated FingerPrint and moving image information corresponding to the image that is the basis of the FingerPrint to the matching server 104.

  In S1104, matching server 104 associates FingerPrint received from index server 105 and moving image information corresponding to the FingerPrint, and sequentially accumulates them in the database.

  On the other hand, on the mobile terminal 101 side, the user activates the application 302 and holds the camera over the television 102. Thereby, in S1105, the mobile terminal 101 recognizes the moving image being reproduced on the television 102. Here, the moving image being reproduced (broadcast) by the television 102 is equivalent to the moving image data received by the index server 105 in S1101.

  In step S1106, the mobile terminal 101 extracts an image from the recognized moving image. The number of images extracted here may be one as shown in FIG. 4, or a plurality of images may be acquired at predetermined intervals. Then, the portable terminal 101 generates search data (FingerPrint) for the extracted image using a predetermined hash function. In step S <b> 1107, the mobile terminal 101 transmits the generated FingerPrint to the matching server 104.

  In S1108, the matching server 104 searches for a FingerPrint that matches the FingerPrint received from the mobile terminal 101, from the FingerPrint stored in S1104. If the matching FingerPrint is held in the matching server 104, the moving image information stored in the database in association with the matching FingerPrint is acquired. For example, in the case where the video information is a television program distributed from a broadcasting station as in the present embodiment, the channel ID (identifier) uniquely indicating the broadcasting station 103 and the time when the image (frame) is distributed ( Time stamp). Note that the moving image information may include other information.

  In S1109, matching server 104 transmits the specified moving image information to portable terminal 101.

  In step S1110, the mobile terminal 101 makes an inquiry to the proxy server 901 using the moving image information received from the matching server 104.

  In S1111, the proxy server 901 makes an inquiry to the index server 105 using the moving image information included in the inquiry from the mobile terminal 101.

  In S1112, the index server 105 searches the database that has stored the image data in S1102 for image data that matches the moving image information included in the inquiry from the proxy server 901.

  In step S1113, the proxy server 901 returns the image data obtained as a result of the detection to the proxy server 901 as a response.

  In S <b> 1114, the proxy server 901 receives image data as a search result from the index server 105 and transmits the received image data to the mobile terminal 101.

  In step S1115, the mobile terminal 101 performs processing using the acquired moving image data. An example of the processing here corresponds to the example described with reference to FIG. 8 in the first embodiment. Then, this processing flow ends.

  As described above, according to the present embodiment, the user can easily acquire the image data and related information of the moving image simply by causing the mobile terminal to recognize the moving image being played back on the playback device such as a television.

<Other embodiments>
In the above-described embodiment, an example in which a moving image is detected for television broadcasting has been described. However, the present invention is not limited to this. For example, for a movie or a short movie, the video data is held on the server side, and the mobile terminal recognizes that the movie or the like is being played, thereby performing the same processing as in the above embodiment. It is possible.

  In the above embodiment, the image is extracted from the moving image data at predetermined intervals. However, the present invention is not limited to this, and the image may be acquired even when the content of the image changes significantly.

  Further, although the image is acquired by a predetermined time from the time when the image time is specified, the user may be allowed to specify the acquisition time.

  The present invention also supplies a software program that implements the functions of the above-described embodiments to a system or apparatus via a network or various storage media, and a computer (or CPU or MPU included in the computer) of the system or apparatus. Etc.) may be realized by reading and executing the program. Furthermore, the method which each apparatus in each embodiment mentioned above implements is also included under the category of the present invention.

Portable terminal 101
TV 102
Broadcasting station 103
Matching server 104
Index server 105
Video server 106
Service server 107
Proxy server 901

Claims (3)

A video information acquisition system including a mobile terminal equipped with a camera, a first server, and a second server,
The portable terminal is
Recognizing means for recognizing the moving image being played by the playback device via the camera;
An acquisition means for acquiring an image from the recognized moving image being reproduced via the camera;
First generation means for generating search data using the image acquired by the acquisition means;
Means for transmitting the search data generated by the first generation means to the first server, and acquiring moving image data from the first server as a response;
The first server is
Second generating means for generating search data using an image that constitutes the same moving image as the moving image to be played by the playback device;
Search data storage means for associating the search data generated by the second generation means with the information about the image used for generating the search data in the first storage unit;
Receiving means for receiving search data from the portable terminal;
Search means for searching search data matching the search data received by the receiving means from the first storage unit, and acquiring information associated with the matching search data;
Using the information acquired by the search means to make an inquiry to the second server, and acquiring the moving image data as a response;
Transmitting means for transmitting the moving image data acquired by the acquiring means to the mobile terminal;
Have
The second server is
Moving image data storage means for acquiring the same moving image as the moving image to be played back by the playback device and storing it in a second storage unit;
When an inquiry is received from the first server, moving image data corresponding to the information specified in the inquiry is specified from the second storage unit, and the specified moving image data is used as a response to the inquiry. A moving image information acquisition system comprising: response means for transmitting to one server. The playback device is a television set;
The moving image information acquisition system according to claim 1, wherein the moving image played by the playback device is a moving image distributed as a television broadcast. A video information acquisition system including a mobile terminal equipped with a camera, a first server, and a second server,
The portable terminal is
Recognizing means for recognizing the moving image being played by the playback device via the camera;
An acquisition means for acquiring an image from the recognized moving image being reproduced via the camera;
First generation means for generating search data using the image acquired by the acquisition means;
An acquisition unit that transmits the search data generated by the first search unit to the first server, and acquires information about the image acquired by the acquisition unit as a response;
Using the information obtained by the obtaining means, inquiring the second server, and obtaining moving image data as a response;
The first server is
Second generating means for extracting a plurality of images from the same moving image as the moving image to be reproduced by the reproducing apparatus, and generating a plurality of search data using each of the extracted plurality of images;
Storage means for associating a plurality of search data generated by the second generation means with information related to a plurality of image data used for generating the search data in a storage unit;
Receiving means for receiving search data from the portable terminal;
Search means for searching for search data matching the search data received by the receiving means from the storage unit, and acquiring information associated with the matching search data;
Means for transmitting information searched by the search means to the portable terminal;
Have
The second server is
When an inquiry is received from the portable terminal, image data that matches the information specified in the inquiry and image data that continues to the matching image data are identified from the storage unit, and the portable terminal is used as moving image data. A moving image information acquisition system comprising:

Images