WO2014142301A1

WO2014142301A1 - Information supply system, server device, video display device, information supply method, and program

Info

Publication number: WO2014142301A1
Application number: PCT/JP2014/056867
Authority: WO
Inventors: 賢木下; 幸樹岡田
Original assignee: 日本電気株式会社
Priority date: 2013-03-15
Filing date: 2014-03-14
Publication date: 2014-09-18

Abstract

An information supply system (1) provided with a video display device (10) and a server device (100). The server device (100) is provided with: a feature information acquisition unit (102) for acquiring image feature information extracted from a still image corresponding to a time point designated by a viewer, within a video output to the video display device (10); an object identifying unit (104) for identifying an object in the still image on the basis of the image feature information acquired by the feature information acquisition unit (102); and a preference information generating unit (106) for generating viewer preference information (108) including object information identified by the object identifying unit (104).

Description

Information providing system, server device, moving image display device, information providing method, and program

The present invention relates to an information providing system, a server device, a moving image display device, an information providing method, and a program.

An example of a method for distributing product information included in broadcast video is described in Patent Document 1. In the information distribution method of Patent Document 1, a broadcast station broadcasts a video, related information of a product included in the video, and correspondence information indicating in which scene of the video the product is displayed. And storing at least temporarily the broadcasted product related information and correspondence information in the storage unit of the receiving device, displaying the broadcast video on the receiving video display device, and the video display device. In response to the instruction input that the viewer wants to obtain information about the product by looking at the displayed scene of the displayed product, the predetermined range area before the instruction input based on the correspondence information And displaying the product related information on the video display device.

Also, the information providing method described in Patent Document 2 extracts only the portion of interest of the user from the multimedia information and selectively provides individual information for each user. Extraction of information of interest to the user is performed based on a user profile created by inquiring the user or taking a questionnaire. The advertisement information distribution method described in Patent Document 3 distributes a program schedule table in which an information tag attached to each broadcast program is embedded to a home server for home appliance management of each home, displays it on the information terminal of each home, and views it. When the user operates the television from the program schedule table, an advertisement suitable for each viewer is distributed in real time from the operation history of the information terminal and the information tag. The advertisement guidance system described in Patent Document 4 transmits selected advertisement guidance from a call center system to a customer phone, and refers to, registers, and updates the conditions for sending the advertisement guidance to the customer phone and the content of the advertisement guidance. , And can be deleted.

An update registration method for an image-added word dictionary system described in Patent Document 5 analyzes input image data, converts it into image comparison data, searches a dictionary database, and images similar to the converted image comparison data When comparison data and related names are extracted, and similar image comparison data is extracted, it is determined whether or not the extracted name data and the input name have the same tendency. By registering the obtained image comparison data in the dictionary database as a similar pattern, the relationship between the registered image and its name is automatically organized.
Further, the content and CM distribution system described in the feature document 6 can receive the content data and the CM data from the server, accumulate the CM data, and reproduce it at a timing desired by the user. The advertisement distribution method described in Patent Document 7 automatically selects consumers with high interest in various products including services from Internet users, and distributes advertisements of the respective products.

JP 2005-94788 A JP 2000-250944 A JP 2003-331183 A JP 2004-312093 A JP 2008-40649 A JP 2009-288891 A JP 2002-109367 A

However, in the technology related to the information distribution method described above, correspondence information is prepared because information in which the appearance position of the product included in the video is associated with the related information of the product is prepared and broadcast in advance. There is a problem in that it is impossible to input an instruction to obtain information on a user's product for a video or product that is not available.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an information providing system, a server device, a moving image display device, an information providing method, and a program for accurately estimating a user's preference from a moving image viewing act. Is to provide.

The information providing system of the present invention includes:
An information providing system comprising a video display device and a server device,
The server device
Feature information acquisition means for acquiring image feature information extracted from a still image corresponding to a time point designated by the viewing user in the video output to the video display device;
Based on the image feature information acquired by the feature information acquisition means, an object specifying means for specifying an object that appears in the still image;
Preference information generating means for generating preference information of the viewing user including object information specified by the object specifying means.

The server device of the present invention
A server device connected to the video display device,
Feature information acquisition means for acquiring image feature information extracted from a still image corresponding to a time point designated by the viewing user in the video output to the video display device;
Based on the image feature information acquired by the feature information acquisition means, an object specifying means for specifying an object that appears in the still image;
Preference information generating means for generating preference information of the viewing user including object information specified by the object specifying means.

The moving image display apparatus of the present invention is
A video display device connected to the server device,
A video display means for displaying a video;
Obtaining means for obtaining a still image corresponding to a time point designated by a viewing user in the moving picture output to the moving picture display means;
A still image transmitting means for transmitting the still image to the server device;
The recommended action generated based on the still image is received from the server device, and the recommended action is sequentially accumulated, and an indicator indicating that the recommended action is provided from the server device outside the area where the moving image is output And a recommended action notification means for notifying the viewing user,
And presenting means for presenting recommended behavior information to the viewing user in response to an instruction by the viewing user to view the accumulated recommended behavior in response to the notification by the recommended behavior notification means.

The information providing method of the present invention includes:
Server device
Acquire extracted image feature information from the still image corresponding to the time point specified by the viewing user in the video output to the video display device,
Based on the acquired image feature information, identify an object that appears in the still image,
The viewing user preference information including the specified object information is generated.

The first computer program of the present invention is:
On the computer,
A procedure for acquiring extracted image feature information from a still image corresponding to a time point specified by a viewing user in a moving image output to a moving image display device,
A procedure for identifying an object shown in the still image based on the acquired image feature information;
It is a program for executing a procedure for generating preference information of the viewing user including object information specified by a procedure for specifying the object.

The second computer program of the present invention is:
In a computer equipped with a storage device,
A procedure for storing the preference information generated by the procedure for generating the preference information based on the still image obtained from the user designation for the moving image in the storage device;
A procedure for performing data mining on the stored preference information of the viewing user and classifying the viewing user into at least one field in which the viewing user is interested;
For each field, a recommended condition indicating a correspondence relationship between each of a plurality of different recommended actions and each condition pattern indicating a combination of conditions for each action of the viewing user indicated by the viewing user's preference information is stored in the memory The procedure to hold in the device,
The recommendation condition corresponding to the field into which the viewing user is classified is acquired, and the behavior pattern of the viewing user obtained from the viewing user's preference information stored in the storage device is applied to each condition pattern of the recommendation condition This is a program for executing a procedure for generating recommended action information indicating at least one recommended action among the plurality of different recommended actions.

The third computer program of the present invention is:
To a computer that can be connected to the server device,
Steps to view the video,
A procedure for acquiring a still image corresponding to a point in time specified by the viewing user in the video output in the procedure for displaying the video,
A procedure for transmitting the still image to the server device;
A procedure for receiving recommended actions generated based on the still image from the server device and sequentially storing them.
A procedure for outputting an indicator indicating that the recommended action is provided from the server device outside the area where the video is output, and notifying the viewing user,
A program for receiving a notification by the notification procedure and executing a procedure for presenting recommended behavior information to the viewing user in response to an instruction from the viewing user to browse the accumulated recommended behavior.

It should be noted that an arbitrary combination of the above-described components and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as an aspect of the present invention.

The various components of the present invention do not necessarily have to be independent of each other. A plurality of components are formed as a single member, and a single component is formed of a plurality of members. It may be that a certain component is a part of another component, a part of a certain component overlaps with a part of another component, or the like.

In addition, although a plurality of procedures are described in order in the method and computer program of the present invention, the order of description does not limit the order in which the plurality of procedures are executed. For this reason, when the method and computer program of the present invention are implemented, the order of the plurality of procedures can be changed within a range that does not hinder the contents.

Furthermore, the plurality of procedures of the method and computer program of the present invention are not limited to being executed at different timings. For this reason, another procedure may occur during the execution of a certain procedure, or some or all of the execution timing of a certain procedure and the execution timing of another procedure may overlap.

According to the present invention, an information providing system, a server device, a moving image display device, an information providing method, and a program for accurately estimating a user's preference from a moving image viewing act are provided.

The above-described object and other objects, features, and advantages will be further clarified by a preferred embodiment described below and the following drawings attached thereto.

It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the information provision system which concerns on embodiment of this invention. It is a block diagram which shows the hardware constitutions of the moving image display apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the process sequence of the information provision system which implement | achieves the information provision method of this embodiment. It is a figure for demonstrating operation in the information provision system which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a figure for demonstrating operation in the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the principal part structure of the server apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the action history information of the information provision system which concerns on embodiment of this invention. It is a figure which shows the example of the conditional expression of the recommendation action of the information provision system which concerns on embodiment of this invention. It is a flowchart which shows an example of operation | movement of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a figure for demonstrating the notification method of the recommendation action of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the structure of the information provision system which concerns on embodiment of this invention. It is a functional block diagram which shows the logical structure of the moving image display apparatus which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate.

(First embodiment)
FIG. 1 is a functional block diagram showing a configuration of an information providing system 1 according to an embodiment of the present invention.
An information providing system 1 according to an embodiment of the present invention includes a moving image display device 10 and a server device 100. The server device 100 includes a feature information acquisition unit 102 that acquires image feature information extracted from a still image corresponding to a time point specified by the viewing user in the moving image output to the moving image display device 10, and a feature Based on the image feature information acquired by the information acquisition unit 102, an object specifying unit 104 that specifies an object that appears in a still image and viewing user preference information 108 that includes the object information specified by the object specifying unit 104 are generated. A preference information generation unit 106.

The server apparatus 100 of the present embodiment is realized by cloud computing using a server computer such as a data center, but is not limited to this. Further, the server computer of the server device 100 may be configured by a virtual server or the like.

1 includes a CPU (Central Processing Unit), a memory, a program that realizes the components shown in FIG. 1 loaded in the memory, a storage unit that stores the program, and a network connection interface. This is realized by an arbitrary combination of hardware and software of an arbitrary computer (not shown). It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus. Each functional block diagram described below shows a logical block of functional units, not a configuration of hardware units.
In the following drawings, the configuration of parts not related to the essence of the present invention is omitted and is not shown.

In the server device 100 according to the present embodiment, various components of the computer are realized as various functions by the CPU executing various processing operations corresponding to the computer program.
The computer program according to the embodiment of the present invention is obtained from a still image corresponding to a time point designated by a viewing user in a moving image output to the moving image display device 10 in a computer for realizing the server device 100. , A procedure for acquiring extracted image feature information, a procedure for specifying an object appearing in a still image based on the acquired image feature information, and a viewing user's preference information including object information specified by the procedure for specifying an object It is described to execute the procedure to generate.

The computer program according to the embodiment of the present invention may be recorded on a computer-readable recording medium that constitutes the server apparatus 100. The recording medium is not particularly limited, and various forms can be considered. The program may be loaded from a recording medium into a computer memory, or downloaded to a computer through a network and loaded into the memory.

The overall configuration of the information providing system 1 according to the embodiment of the present invention will be described.
FIG. 2 is a block diagram showing a configuration example of the information providing system 1 according to the embodiment of the present invention.
In the information providing system 1 according to the embodiment of the present invention, a video / broadcaster 7 transmits broadcast waves on the ground or satellite via a moving picture distribution device such as a set top box (STB) 3. And the moving image is distributed to at least one moving image display device 10. The moving image distribution device can include, for example, a plurality of tuners that simultaneously receive a plurality of channels of broadcast signals. The video distribution device may be a TV, a home server, a recording device, or a gateway device in addition to the STB 3. The moving image display device 10 is connected to the server device 100 on the cloud 5.

The video display device 10 is, for example, a smartphone, a tablet terminal, a game machine, a television receiver, a personal computer, or a media player, and installs and executes an application program for displaying the video content on the video display device 10. can do. In FIG. 2, three moving

image display devices

10a, 10b, and 10c are shown, but the number of devices is not limited to this. Hereinafter, in particular, when it is not necessary to distinguish each device, it is simply referred to as “moving image display device 10”.
The moving image display device 10 may be in various forms such as a portable type or a stationary type.

In the present invention, the moving image content displayed on the moving image display device 10 includes, for example, moving image, sound, data, or a combination thereof, and the like, stored in real time or in advance from the moving image distribution device Or what was recorded or what combined them may be sufficient.

In the following embodiment, a configuration will be described in which the video distribution device sequentially distributes the video content received from the video / broadcasting company 7 in real time, and the video display device 10 receives and displays the video content. However, the present invention is not limited to this. A configuration may be adopted in which moving image data recorded or stored in a storage device or other recording medium accessible by the moving image distribution device is read and distributed, and the moving image display device 10 receives and displays it. Alternatively, the moving image distribution apparatus may be configured to store the content received in a streaming format from a content server or the like via a network in real time or temporarily and distribute and receive and display the moving image display apparatus 10.

Alternatively, the moving image display device 10 may be configured to read and display video data recorded or stored in an accessible storage device or other recording medium. Or the structure which the moving image display apparatus 10 accumulate | stores and displays the content directly received with the streaming format from the content server etc. via the network instead of going through a moving image delivery apparatus may be sufficient.

The storage device or recording medium is, for example, a hard disk, a USB (Universal Serial Bus) flash memory, a Blu-ray disc, a DVD (Digital Versatile Disk), a CD (Compact Disc), an MO (Magneto-Optical disk), or a magnetic tape. Includes various media.
The moving image display device 10 and the moving image distribution device such as the STB 3 or the server device 100 can be configured to communicate with each other wirelessly, wired, or a combination thereof.

FIG. 3 is a block diagram showing a hardware configuration of the moving image display apparatus 10 according to the embodiment of the present invention. This embodiment demonstrates as an example the case where the moving image display apparatus 10 is a smart phone. As shown in FIG. 3, the moving image display apparatus 10 according to the present embodiment includes a CPU 12, a memory 14 including a ROM (Read Only Memory) and a RAM (Random Access Memory), an I / O (Input and Output) 18, The wireless LAN communication unit 20, the operation unit 22, the operation reception unit 24, the display unit 26, the display control unit 28, the voice control unit 36, the speaker 32, and the microphone 34 are provided.

The CPU 12 is connected to each element of the moving image display device 10 via the bus 40 and controls the entire moving image display device 10 together with each element. The memory 14 stores programs and various application programs for operating the moving image display apparatus 10, various setting data used when these programs operate, and user data including address data and various content data. To do. Further, the memory 14 has an area for temporarily storing data, such as a work area for operating the program. The I / O 18 controls input / output to the moving image display device 10.

The memory 14 includes a ROM, a RAM, a flash memory, a disk drive, etc., and other functions having functions for storing application programs and setting data for operating the programs, temporarily stored data, user data, and the like. It may be a device.

The operation unit 22 includes operation keys, operation buttons, switches, a jog dial, a touch pad, a touch panel integrated with the display unit 26, and the like. The operation reception unit 24 receives an operation of the operation unit 22 by the user and notifies the CPU 12 of the operation. The display unit 26 includes an LED (Light Emitting Diode) display, a liquid crystal display, an organic EL (ElectroLuminescence) display, and the like. The display control unit 28 displays various screens on the display unit 26 in accordance with instructions from the CPU 12. The voice control unit 36 performs voice output from the speaker 32 and voice input from the microphone 34 in accordance with instructions from the CPU 12.

The wireless LAN communication unit 20 performs wireless LAN communication with the moving image distribution device (here, STB3) or the server device 100 on the cloud 5 via the wireless LAN antenna 21 by a method compliant with, for example, the IEEE 802.11 standard. I do. The moving image distribution device or the server device 100 can communicate with each moving image display device 10 via a wireless LAN access point device (not shown). Note that the moving image distribution apparatus may include a wireless LAN access point function.

Between the moving image display device 10 and the moving image distribution device or between the moving image display device 10 and the server device 100, wireless LAN communication access point search processing, registration processing of the moving image display device 10 to the access point, authentication processing, and the like are performed. This is performed in advance, and wireless LAN communication is possible.

Further, as described above, the moving image display apparatus 10 may include at least one tuner for directly receiving a broadcast signal of at least one channel. Alternatively, the moving image display apparatus 10 may receive content directly from a content server (not shown) or the like by the wireless LAN communication unit 20 via a network. Alternatively, the moving image display device 10 may include a media playback unit (not shown), and may play back content recorded on the media and display the content on the display unit 26.
In the present embodiment, the moving image display device 10 is configured to perform wireless LAN communication by the wireless LAN communication unit 20, but is not limited thereto.

In the moving image display device 10, a trial listening application 42 that reproduces moving image content and displays it on the display unit 26 is installed in the memory 14 in advance.
In the moving image display device 10 of the present embodiment, the CPU 12 executes the viewing application 42 to operate according to various processing procedures, and various components (for example, FIG. 21) of the moving image display device 10 are realized as various functions. The

Each component of the moving image display device 10 in each embodiment described below includes a CPU 12, a memory 14, a program (viewing application 42) that realizes the components illustrated in FIG. 21 loaded in the memory 14, and a network connection interface. This is realized by any combination of hardware and software of any computer having (I / O 18, wireless LAN communication unit 20). It will be understood by those skilled in the art that there are various modifications to the implementation method and apparatus.

In the present embodiment, the feature information acquisition unit 102 of the server device 100 of FIG. 1 can share the function between the server device 100 and the moving image display device 10 part by part. For example, as illustrated in FIG. 21, the moving image display apparatus 10 reproduces moving image content, and extracts image feature information from a reproduction unit 112 displayed on the display unit 26 and a still image corresponding to a time point designated by the user. A feature information extraction unit 114 and a transmission unit 116 that transmits image feature information extracted by the feature information extraction unit 114 may be included. The feature information acquisition unit 102 of the server device 100 can acquire the feature information by receiving the feature information transmitted from the transmission unit 116 of the moving image display device 10. As described above, the feature information acquisition unit 102 of the server device 100 may be partially realized by the feature information extraction unit 114 and the transmission unit 116 of the moving image display device 10.

The viewing application 42 of the present embodiment may be recorded on a computer-readable recording medium that constitutes the moving image display apparatus 10. The recording medium is not particularly limited, and various forms can be considered. In addition, the viewing application 42 may be loaded from the recording medium into the memory 14 of the moving image display device 10 or may be downloaded to the moving image display device 10 through a network and loaded into the memory 14.

Returning to FIG. 1, the server apparatus 100 includes a feature information acquisition unit 102, an object specification unit 104, and a preference information generation unit 106.

The feature information acquisition unit 102 acquires image feature information extracted from a still image corresponding to a time point specified by the viewing user in the moving image output to the moving image display device 10.

Specifically, in the moving image display device 10, when a moving image is output by the reproduction unit 112 (FIG. 21), when the display screen is touched by the viewing user, for example, the feature information extraction unit 114 (FIG. 21). Gets a hard copy of the still image being displayed. Alternatively, the feature information extraction unit 114 (FIG. 21) may acquire a corresponding frame image as long as it is a compressed image that is decoded and displayed by the moving image display device 10. Then, the feature information extraction unit 114 (FIG. 21) performs image analysis on the still image, extracts feature points having a data amount that can identify at least an object shown in the still image, and positions of the feature points and their positions. Feature information (image feature information) is acquired. Hereinafter, this processing is referred to as “primary analysis processing”.

Here, the designation by the viewing user means, for example, that the viewing user touches the touch panel of the moving image display device 10 on which the moving image is displayed, or operating an operation key or a switch provided on the moving image display device 10. Alternatively, the viewing user may tilt or shake the moving image display device 10. The touch operation method and the like are not particularly limited, and can be specified by various operations such as single tap, double tap, flick, drag, swipe, and pinch.

For example, when a favorite actor, a favorite costume, a place that he / she wants to visit appear on the video, the viewing user displays the target object (person, thing, place, etc.) on the screen of the video display device 10. You can touch and specify.

Then, the image feature information extracted by the primary analysis process is transmitted from the moving image display apparatus 10 to the server apparatus 100 by the transmission unit 116 (FIG. 21). In the present embodiment, the feature information acquisition unit 102 of the server device 100 acquires image feature information from the moving image display device 10 in this way.
The transmission unit 116 (FIG. 21) may transmit the still image itself from the moving image display device 10 to the server device 100 and perform an image analysis process in the server device 100.
As in the present embodiment, after the feature information extraction unit 114 (FIG. 21) of the moving image display device 10 performs the primary analysis process, the transmission unit 116 (FIG. 21) sends the image feature amount information to the server device 100. If it is set as the structure which transmits, the transmission amount to the server apparatus 100 or the image analysis processing load of the server apparatus 100 can be reduced.

How much the moving image display device 10 and the server device 100 share the primary analysis processing depends on the image analysis processing capability of the moving image display device 10 or the server device 100, the required analysis accuracy, or the moving image display device. 10 and the server apparatus 100 can be determined as appropriate based on the communication processing capability and the like.

Further, the still image to be analyzed by the moving image display device 10 may include not only the image frame displayed at the moment when the user touches the screen but also a plurality of frames before and after the image frame. Further, as described above, when the level of interest designated by the user can be indicated by the operation method, the information on the level of interest or the information on the operation method is displayed together with the information on the feature amount of the image in the moving image display device 10. To the server apparatus 100.

The object specifying unit 104 specifies an object that appears in a still image based on the image feature information acquired by the feature information acquiring unit 102.
Specifically, the object specifying unit 104 collates the feature amount of each feature point with the feature amount of each feature point of the detection target object registered in advance by pattern recognition or the like, and the feature point at least partially matches. And their relative positions can be detected.
Here, the detection target object includes, for example, an object, a person, a place (a store, a facility, a landscape), or the like.

The preference information generation unit 106 generates viewing user preference information 108 including object information specified by the object specification unit 104.
Specifically, for example, when the object specifying unit 104 specifies at least one object appearing in a still image, a word (word) indicating the object can be output as the preference information 108. For example, it may be output as text data.

As described above, when the information on the level of interest of the user is acquired from the moving image display device 10, the preference information 108 can also include information on the level of interest for the object. The preference information 108 can include, for example, date information and at least one word for each user. The preference information 108 may include information obtained from the history of the preference information 108.

Note that there may be a case where the designation of the viewing user is delayed with respect to the displayed moving image. If an object is identified based on a still image at the moment of touching, an object deviating from the user's intention can be obtained. There is a possibility. Accordingly, correction processing is appropriately performed by the moving image display device 10 or the server device 100 for such a delay in operation.

As this correction processing, for example, the speed and direction of the movement of the object in the displayed moving image, the change of the object, etc. are analyzed for the images before and after the user specified, and when there is a change beyond a certain value The object may be specified for the images before and after the designated time point. Then, the words of a plurality of identified objects may be generated as preference information 108, respectively. At this time, level information such as probability (likelihood) as the preference information 108 may be added to each word of the preference information 108.

An information providing method using the server apparatus 100 according to the embodiment of the present invention in the configuration as described above will be described below. FIG. 4 is a flowchart illustrating an example of a processing procedure of the information providing system 1 that realizes the information providing method of the present embodiment.
In the information providing method according to the embodiment of the present invention, the server device 100 extracts an image extracted from a still image corresponding to a time point specified by the viewing user in the moving image output to the moving image display device 10. Feature information is acquired (YES in step S101, step S103), an object appearing in a still image is specified based on the acquired image feature information (step S105), and viewing user preference information including the specified object information is obtained. Generate (step S107).

Specifically, for example, as shown in FIG. 5, it is assumed that an image of a woman wearing a dress is displayed on the display unit 26 of the moving image display device 10. At that time, it is assumed that the user touches the portion corresponding to the female body on the screen of the moving image display device 10. At this time, when the moving image display device 10 (for example, the feature information extraction unit 114 in FIG. 21) detects that the screen is touched by the user (YES in step S101 in FIG. 6), this processing starts.

In step S103 in FIG. 6, image feature information extracted from the still image displayed when the user touches the screen is transmitted from the moving image display device 10 to the server device 100, and the feature information acquisition unit 102 The image feature information is acquired. In step S105, based on the image feature information acquired in step S103 by the object specifying unit 104, a person “actress S” and a “cat pattern dress” worn by the person are displayed as objects to be moved to a still image. Identified.
In step S107, the preference information generation unit 106 generates "actress S" and "cat pattern dress" as the user preference information 108.

Here, the reason that the user touched the screen may be because the user was a fan of female actress S. Or the user may have liked a dress worn by a woman. In response to the user's touch operation on the screen, the server apparatus 100 estimates the user's preferences and generates both “actress S” and “cat pattern dress” as preference information 108 as described above. Can do.

As described above, according to the information providing system 1 according to the embodiment of the present invention, the viewing user is interested in the moving image output to the moving image display device 10 by an operation such as the user touching the screen. When the object is specified, the server apparatus 100 is estimated to have been specified by the user based on the image feature information of the still image corresponding to the time point specified by the user of the moving image output to the moving image display device 10. Objects can be specified.
In the information providing system 1 according to the embodiment of the present invention, it is not necessary for the server device 100 to grasp in advance the content of the moving image output to the moving image display device 10. The moving image to be processed in the information providing system 1 of the present invention may be any content that can be reproduced by the viewing application 42.
According to the present invention, user preference information is generated from the object specified in this way, so that the user preference can be accurately estimated from the video viewing action.

(Second Embodiment)
FIG. 7 is a functional block diagram showing the configuration of the information providing system 1 according to the embodiment of the present invention.
The information providing system 1 of the present embodiment is different from the above-described embodiment in that it has a configuration for acquiring designation information by the user. As shown in FIG. 7, the server device 200 of the present embodiment includes a feature information acquisition unit 102 and a preference information generation unit 106 similar to the server device 100 of the above-described embodiment of FIG. An information acquisition unit 202 and an object identification unit 204 are provided.

In the information providing system 1 according to the embodiment of the present invention, the server device 200 specifies the viewing user's designated position and designated range in a moving image still image output to the moving image display device 10 at the time designated by the viewing user. , And a designation information acquisition unit 202 that acquires designation information including any of the designation methods.
Then, the object specifying unit 204 specifies an object that appears in the still image by further considering any one of the specified position, the specified range, and the specifying method.

Specifically, the designation information acquisition unit 202 displays, for example, the coordinates of the position on the screen touched by the user as the designation information of the movie output to the movie display device 10 at the time designated by the user. Receive from device 10. Similar to the above-described embodiment, the specified information acquisition unit 202 can share a part of the functions between the server device 200 and the moving image display device 10. The moving image display device 10 may include a designation information acquisition unit (not shown) that detects the coordinates of the position on the screen touched by the user and delivers the coordinates to the server device 200.
For example, the object specifying unit 204 may specify an object that exists at a position corresponding to a specified position (coordinates) in a still image from the specified information acquired by the specified information acquiring unit 202.

Alternatively, when all objects displayed on the screen are specified, a predetermined area such as an area at the end of the screen may be specified, or a predetermined operation button may be determined in advance. The object specifying unit 204 determines from the specified information that the specified area has been specified or that a predetermined operation button has been operated, specifies all objects in the still image, and the preference information generating unit 106 selects the preference information. 108 can be generated.

Also, the user's designation operation method may have meanings such as level designation by the viewing user and range designation. The designation information acquisition unit 202 may acquire designation information including information on an operation type and an operation range designated by the viewing user from the moving image display device 10.

For example, a single tap may mean “slightly interested” and a double tap may mean “very interested”. Alternatively, when an object displayed on the screen of the moving image display device 10 is flicked, it may mean that the object is “not interested”. Or if the video shows a scene where an actor is eating at a restaurant, for example, if you ’re interested in an actor or a menu, tap the actor and food. If the user is interested, the user's intention may be indicated according to the operation type or operation range, such as swiping the entire screen with a finger. The designation information acquisition unit of the moving image display apparatus 10 may include information such as an operation type and an operation range when a user designation is detected in the designation information and transmit it to the server apparatus 200.

The object specifying unit 204 can specify an object that the user is interested in or is not interested in, considering the operation type and operation range information included in the designation information.
Here, for example, as shown in FIG. 8A, it is assumed that the user touches the position of a hat worn by a woman in the moving image displayed on the display unit 26 of the moving image display device 10. At this time, the server device 200 acquires information on the coordinates touched by the user from the moving image display device 10 together with information on the feature amount of the image displayed on the display unit 26 as designation information.

Based on the image feature information of the still image, the object specifying unit 204 can specify a plurality of objects such as “Mr. Actress”, “Cat-patterned dress”, “Hat”, etc. From this, it is possible to specify an object existing in the vicinity of the coordinates indicated by the designation information, that is, a “hat”. At this time, the brand name of “hat” may be specified by image analysis. Then, the preference information generation unit 106 generates “hat” as the preference information 108. At this time, the preference information 108 may be generated together with the brand name “hat”.

Alternatively, as shown in FIG. 8B, it is assumed that the user touches the end position in the moving image displayed on the display unit 26 of the moving image display device 10. At this time, the object specifying unit 204 may specify all the objects in the still image, for example. Then, the preference information generation unit 106 generates all the objects in the still image, that is, “actress S”, “one piece of cat pattern”, and “hat” as the preference information 108.

As described above, according to the information providing system 1 of the embodiment of the present invention, the same effect as that of the above embodiment can be obtained, and the object specified from the image can be specified by using the information related to the designation by the user. Accuracy can be improved. Further, the level of interest of the user can be estimated for each object based on the designation method.

(Third embodiment)
FIG. 9 is a functional block diagram showing a configuration of the information providing system 1 according to the embodiment of the present invention.
The information providing system 1 according to the present embodiment is different from the above embodiment in that a broadcast channel, a program name, a title, a specified time, or a time that is output to the video display device 10 at a time specified by the user This is different in that it has a configuration for acquiring designation information including an elapsed time from the start of the program when designated. As shown in FIG. 9, the server device 300 of the present embodiment includes a feature information acquisition unit 102 and a preference information generation unit 106 similar to the server device 100 of the embodiment of FIG. An information acquisition unit 302, a content specifying unit 304, and an object specifying unit 306 are provided.
In addition, the server apparatus 300 of this embodiment can also contain the structure of the server apparatus 200 of the said embodiment of FIG.

In the information providing system 1 according to the embodiment of the present invention, the server device 300 includes the content information of the moving image output to the moving image display device 10 at the time specified by the viewing user, the specified time by the viewing user, and the viewing user. The designation information acquisition unit 302 that acquires the designation information including at least one of the elapsed time from the start of the video at the time designated by the video, the content information of the video, and the object information related to the content are held. A content identification unit 304 that refers to the content information holding unit 310 and identifies content that matches the designation information acquired by the designation information acquisition unit 302 as content that has been output to the video display device 10; Prepare.
The object specifying unit 306 refers to the content information holding unit 310, acquires object information related to the content specified by the content specifying unit 304, and further considers the acquired information to determine an object that appears in a still image. Identify.

The server apparatus 300 may include a content information holding unit 310 that holds moving image content information. The content information holding unit 310 may not be provided in the server device 300, as long as the server device 300 can refer to and use information held in the content information holding unit 310.

The moving picture designation information is information relating to the moving picture displayed on the moving picture display device 10 at the time when the designated operation is performed by the viewing user. The moving picture broadcast channel, program name, number of stories, etc., or DVD And the like, the number of stories, etc., the time at which the designated operation was performed, and the elapsed time from the start of the program or title at that time. Alternatively, the moving picture designation information may include information such as the elapsed time from the time when the previous designation operation was performed, or the number of designations (frequency) within a predetermined period.

Also, the designation information acquisition unit 302 includes information similar to the information acquired by the designation information acquisition unit 202 of the above-described embodiment as the movie specification information output from the movie display device 10 at the time specified by the viewing user. You may receive designation | designated information from the moving image display apparatus 10, for example, the information of the coordinate of the position on the screen touched by the user.

The content information of the moving image appears in, for example, the channel through which the content is broadcast, the program schedule information including the program name and title of the content, the number of stories, etc., and the information on the object related to the content, for example, the content A word (word) such as an object name, a sentence (text) expressing the object, an appearance position (information such as arrangement, region, time) of the object in the content can be included. Alternatively, the content information of the video includes the name of the person who appears, the type of content (drama, movie, variety, education, documentary), and the field handled by the content (romance, war, Science etc.).

Alternatively, the content information of the moving image can include tag information (for example, at least one word) added in advance to the content and at least one object in the content, or a combination thereof.

The operation of the information providing system 1 of the present embodiment configured as described above will be described below.
FIG. 10 is a flowchart showing an example of the operation of the information providing system 1 of the present embodiment.
The operation of the information providing system 1 of the present embodiment includes steps S101 and S107 similar to the flowchart of FIG. 4 showing the operation of the above embodiment, and further includes steps S301 to S307.

First, in the present embodiment, when the moving image display apparatus 10 detects a touch on the screen by the user (YES in step S101), the process starts. The feature information acquisition unit 102 acquires specified information from the moving image display device 10 together with the image feature information from the still image displayed when the user touches the screen (step S301).
For example, it is assumed that the user is watching a movie broadcast on a certain channel on the moving image display device 10. It is assumed that the user likes an appearing actor and touches the actor. At this time, the moving image display apparatus 10 transmits to the server apparatus 300 designation information including at least one of the channel being viewed, the program name, the elapsed time from the start of broadcasting, and the touch position information together with the image feature information.

Then, the content specifying unit 304 refers to the content information holding unit 310, and the content that matches the designation information acquired in step S301 is the content that has been output to the video display device 10 when the designation is made by the user. It is specified that there is (step S303).
The content specifying unit 304 can specify, for example, a broadcast program that the user is watching on the moving image display device 10 from information such as a viewing channel and an elapsed time from the start of broadcasting.

Then, the object specifying unit 306 refers to the content information holding unit 310 and acquires information on the object related to the content specified by the content specifying unit 304 (step S305).
For example, the object specifying unit 306 can acquire information such as the title of the movie being broadcast and the actor who is appearing based on the information of the broadcast program specified by the content specifying unit 304.

Then, the object specifying unit 306 further considers the information acquired in step S305 and specifies an object that appears in the still image based on the image feature information (step S307).
For example, based on the title of the movie specified by the object specifying unit 306 and the information of the actor who is appearing, the feature amount of the image of the person that is estimated to be specified by the user, specified from the image feature information, and the appearance An actor can be specified when it is determined that the feature amount of the actor's image is similar.

Then, the preference information generation unit 106 generates viewing user preference information 108 including the specified object information (step S107). For example, the preference information generating unit 106 outputs a movie title and an actor name as the preference information 108.

As described above, according to the information providing system 1 of the embodiment of the present invention, the same effect as that of the above embodiment can be obtained, and the information related to the content specified by the user can be used to identify the image. The object identification accuracy can be improved.

(Fourth embodiment)
FIG. 11 is a functional block diagram showing a configuration of the information providing system 1 according to the embodiment of the present invention.
The information providing system 1 of this embodiment is different from the above-described embodiment in that it has a configuration for specifying a scene designated by a user in a moving image.
As shown in FIG. 11, the server device 400 of this embodiment includes a feature information acquisition unit 102, a specified information acquisition unit 302, a content specification unit 304, an object, and the same as the server device 300 of the above embodiment of FIG. 9. And a timeline information acquisition unit 402, a scene specification unit 404, and a preference information generation unit 406.
In addition, the server apparatus 400 of this embodiment can also contain the structure of the server apparatus 200 of the said embodiment of FIG.

The server apparatus 400 stores at least one of content information similar to the content information holding unit 310 of the above-described embodiment and timeline information of a plurality of scenes constituting the moving image as content information of the moving image. May be included. The content information holding unit 410 may not be provided in the server device 400, and it is only necessary that the server device 400 can refer to and use information held in the content information holding unit 410.

In the information providing system 1 according to the embodiment of the present invention, the server device 400 refers to the content information holding unit 410 that further holds timeline information of a plurality of scenes included in the moving image, and is specified by the content specifying unit 304 A timeline information acquisition unit 402 that acquires timeline information of the content that has been obtained, and a scene corresponding to the time point specified by the viewing user is identified based on the timeline information of the content acquired by the timeline information acquisition unit 402 And a scene specifying unit 404.
The preference information generation unit 406 further generates the viewing user preference information 108 based on the scene information specified by the scene specification unit 404.

In the present embodiment, the content information holding unit 410 can include timeline information of a plurality of scenes included in the moving image as the moving image content information. The timeline information is, for example, data in which a series of image frames and audio frames constituting a moving image are associated with the output time. In other words, the server device 400 can acquire the information on the image and sound included in the scene at the elapsed time from the start of the content from the timeline information. For example, it is possible to acquire music information flowing in the content and speech information such as lines from the timeline information.

Further, in the present embodiment, the content information holding unit 410 can hold the information on the object included in each frame in association with the timeline information. That is, the content information holding unit 410 can hold the timeline information in association with what kind of object appears at which time (scene) of the program. Further, the content information holding unit 410 may hold tag information (word) corresponding to an object or content as timeline information in association with time information.
Further, the content information holding unit 410 may hold, as timeline information, scenario information indicating the contents of each scene of the program in association with the time information.

For example, if the scenario information is a scene showing how parents and children are playing in an amusement park, the name of the amusement park, the location of the amusement park, the season, the surroundings showing the scene with Mt. Fuji in the background, It can include the name and type of play equipment being played, the act of going out with a parent and child, the act of playing in an amusement park, or a fun-looking state with a parent and child.

The audio information included in the scene can include, for example, dialogue included in the audio output in the content, the content of the conversation itself, words and sentences included in the audio, and music information. In addition, how to speak (friendly and scary), emotions expressed by voice, loudness, speed, voice quality, voice principal (person name, etc.), voice gender, voice age, dialect type, suspicion and mimetic words Information related to various voices, such as the type of language, the title of the music, the lyrics, the information of the performer, the author, the producer, or the production company.

The operation of the information providing system 1 of the present embodiment configured as described above will be described below.
FIG. 12 is a flowchart showing an example of the operation of the information providing system 1 of the present embodiment.
The operation of the information providing system 1 according to the present embodiment includes Step S101, Step S301, Step S303, and Step S307 similar to the flowchart of FIG. 10 showing the operation of the above embodiment, and further includes Steps S401 to S405. .

In this embodiment, the timeline information acquisition unit 402 acquires timeline information related to the content specified in step S303 and object information from the content information holding unit 410 (step S401).

Then, the scene specifying unit 404 specifies a scene corresponding to the time point designated by the viewing user based on the timeline information of the content acquired in step S401 (step S403). In step S307, the above-described specified or acquired information is further taken into consideration, and an object that appears in the still image is specified based on the image feature information.
Then, the preference information generating unit 406 generates the viewing user's preference information 108 based on the scene information specified by the scene specifying unit 404 in step S403 (step S405). At this time, the object information specified by the object specifying unit 306 in step S307 may also be included in the preference information 108.
For example, it is assumed that the user likes the music played in the program and touches the screen. In the present embodiment, the scene specifying unit 404 specifies a scene corresponding to the touched time point based on the timeline information, and acquires scene information. Here, for example, the title of the music flowing in the specified scene is generated as the preference information 108.

As described above, according to the information providing system 1 according to the embodiment of the present invention, the user can obtain the same effect as the above-described embodiment and use the timeline information related to the content specified by the user. The specified scene can be accurately identified.

(Fifth embodiment)
FIG. 13 is a functional block diagram showing a configuration of the information providing system 1 according to the embodiment of the present invention.
The information providing system 1 according to the present embodiment is different from the above-described embodiment in that the recommended action suitable for the user is generated based on the preference information 108 generated based on the still image obtained by the user designation. It is different in point.
As illustrated in FIG. 13, the server device 500 of the present embodiment includes a feature information acquisition unit 102, an object identification unit 104, and a preference information generation unit 106 that are the same as the server device 100 of the above-described embodiment of FIG. In addition, a recommended action generation unit 504 is provided.
The server device 500 can further include a profile information storage unit 502 that holds the preference information 108 generated by the preference information generation unit 106 as user profile information. The profile information storage unit 502 does not have to be provided in the server device 500, and it is only necessary that the server device 500 can refer to and use information held in the profile information storage unit 502.

In addition, the server apparatus 500 of this embodiment is the structure of the server apparatus 200 of the said embodiment of FIG. 7, the structure of the server apparatus 300 of the said embodiment of FIG. 9, and the structure of the server apparatus 400 of the said embodiment of FIG. Alternatively, combinations thereof can also be included.

FIG. 14 is a functional block diagram illustrating a main configuration of the recommended action generation unit 504 of the server device 500 according to the present embodiment.
In the information providing system 1 according to the embodiment of the present invention, the server device 500 stores user information that accumulates preference information 108 generated by the preference information generation unit 106 based on a still image obtained from user designation for a moving image. Data mining processing of the user information (profile information storage unit 502) and the stored viewing user preference information 108, and viewing in at least one field (for example, apparel, food, travel, etc.) in which the viewing user is interested Correspondence between user classification unit 512 that classifies users and each condition pattern that indicates each of a plurality of different recommended actions for each field and a combination of conditions for each action of the viewing user indicated by viewing user's preference information 108 The recommended condition holding unit (recommended condition storage unit 518) that holds the recommended condition indicating the relationship and the viewing user are classified. By obtaining recommended conditions corresponding to the field and applying the viewing user's action pattern obtained from the viewing user's preference information 108 accumulated in the profile information storage unit 502 to each condition pattern of the recommended conditions, a plurality of different conditions are obtained. A recommended action generation unit 504 (a recommended condition generation unit 514 and a recommended action output unit 516) that generates recommended action information 506 indicating at least one recommended action among recommended actions.

As illustrated in FIG. 14, the recommended behavior generation unit 504 of the server device 500 of the present embodiment includes a user classification unit 512, a recommended condition generation unit 514, a recommended behavior output unit 516, and a recommended condition storage unit 518. Including.

The user classifying unit 512 performs data mining processing on the stored viewing user preference information 108 and classifies the viewing user into at least one field in which the viewing user is interested. Note that the action history information 530 of the viewing user as shown in FIG. 15 may be subjected to data mining processing together with the preference information 108.

The action history information 530 can include the following information, for example. These pieces of information may be acquired from the moving image display device 10, or may be acquired from a site or the like used by the user after obtaining prior consent from the user. In the video display device 10, for example, the viewing application 42 is executed in cooperation with an application that executes other functions such as a browser function and a Twitter function, and the viewing application 42 obtains the user's consent in advance from the other functions. Various types of information may be collected and transmitted to the server device 500.
(1) Profile: A user profile acquired when user registration (or downloading of the viewing application 42) is used for the use of this system. For example, personal data such as age group, occupation, sex, hobby, etc. (2) SNS ( Social Networking Service) Input history: Words entered by the user using SNS and their contents, access frequency, accessed SNS site field, etc. (3) Browsing history: accessed website field, access frequency, stay time (4) Visit history: When the video display device 10 is a smartphone or the like, the visit area of the user, the name of the store visited, etc. acquired based on GPS data obtained using the GPS (Global Positioning System) function of the smartphone (5 ) Viewing history: programs viewed using the viewing application 42, viewing sites, Listening time (time zone), etc. (6) Purchase history: Purchased products, shopping time, shop store name, etc. that were purchased online on the moving image display device 10 (7) Touch history: program or viewing site touched using viewing application 42, user Preference information 108 including the specified object or scene
(8) Recommendation history: content recommended in the past, user's response to recommendation (whether “Like” was pressed, whether recommended action was executed, etc.), etc. (9) Schedule information: user calendar Schedule information such as (10) Mail history: words used in mail Note that the recommendation and response to the user will be described in detail in another embodiment described later.

Returning to FIG. 14, the recommended condition storage unit 518 includes, for each field, each condition pattern indicating a combination of each of a plurality of different recommended actions and a condition for each action of the viewing user indicated by the viewing user's preference information 108. The recommended condition 520 indicating the correspondence relationship is held.

The recommended condition generation unit 514 acquires the recommended condition 520 corresponding to the field in which the viewing user is classified. The recommended behavior output unit 516 applies the viewing user's behavior pattern obtained from the viewing user's preference information 108 stored in the profile information storage unit 502 to each condition pattern of the recommended conditions, thereby providing a plurality of different recommended behaviors. Recommended action information 506 indicating at least one of the recommended actions is generated.

First, the recommended condition generation unit 514 selects, from the recommended condition storage unit 518, a reference condition 522 in a field in which the user classified by the user classification unit 512 is estimated to be interested.
Note that the recommended condition generation unit 514 indicates a recommended condition 520 (reference condition 522) that indicates a correspondence relationship between a plurality of different recommended actions proposed for each field and a condition pattern that indicates a combination of conditions for each action of the viewing user. And a combination of personal conditions 524) are generated or updated for each viewing user based on the viewing user's preference information 108 and stored in the recommendation condition storage unit 518.
As described above, in the present embodiment, the recommended condition 520 is provided for each field (reference condition 522) and for each viewing user (personal condition 524), and the recommended condition generating unit 514 learns the viewing user's preference information 108. Can be updated while.

For example, as shown in FIG. 16, a recommended pattern is associated with a condition pattern in which a plurality of conditions are combined. At this time, as the plurality of conditions, those suitable for the user are selected based on the user preference information, and a condition pattern as shown in FIG. 16 is generated.

FIG. 16 shows an example in which the reference condition 522 is “whether or not the field of the touched object is apparel”. The personal condition 524 includes a plurality of different recommended conditions for each combined condition pattern, such as a condition for each action of a plurality of viewing users, for example, whether or not a tweet (action) related to clothes is a certain number or more (condition). Actions are associated. In FIG. 16, a condition pattern in which a tweet (behavior) related to clothes is not more than a certain number (condition) and a tweet (behavior) related to performers is more than a certain number (condition) is “a cat pattern matching with performers”. "Recommended one piece sale EC (Electronic Commerce) site" is associated with the recommended action.

In FIG. 16, a plurality of recommended actions, such as “Recommend an EC site for a cat pattern dress”, “Recommend an EC site for a cat pattern stall”, “ “Recommended”, “Recommend EC site selling cat-pattern dresses that match with performers”, etc. are included in the recommended conditional expression.

The recommended action output unit 516 first quantifies the viewing user's behavior pattern generated based on the viewing user's preference information 108 and applies it to the recommended condition. For example, the behavior patterns such as the number of tweets related to clothes, the frequency of purchases on the terminal, the presence of purchase history after touching, the number of tweets related to performers, etc. are quantified and applied to the recommended conditions, which is optimal for the user Can be selected. The recommended action output unit 516 outputs recommended action information indicating the recommended action selected in this way.

The recommended behavior information output from the recommended behavior output unit 516 can include, for example, text data or information such as a URL of a site that recommends a visit.
Note that recommended actions include not only product purchase information, but also browsing web pages containing related information, viewing related programs, restaurant reservations, menu proposals, tour applications, etc. Actions can be included.

In the moving image display device 10 of the present embodiment, the CPU 12 executes various processing operations corresponding to the computer program (viewing application 42), thereby realizing various components of the moving image display device 10 as various functions.
The computer program according to the embodiment of the present invention is specified by a viewing user in a procedure for displaying a movie and a procedure for displaying a movie on a computer for realizing the movie display device 10. A procedure for acquiring a still image corresponding to a certain point in time, a procedure for transmitting a still image to the server device 500, a procedure for receiving a recommended action generated based on the still image from the server device 500, and a procedure for sequentially accumulating it, and outputting a video The indicator indicating that the recommended action is provided from the server device 500 is output outside the area in which the user has received the notification by the procedure for notifying the viewing user and the notification procedure. In response to an instruction from the viewing user, a procedure for presenting recommended behavior information to the viewing user is executed.

The operation of the information providing system 1 of the present embodiment configured as described above will be described below. FIG. 17 is a flowchart illustrating an example of a processing procedure of the information providing system 1 according to the present embodiment.
This process is a process after the preference information 108 is generated based on the still image obtained from the user designation generated in step S107 of FIG. 4 or 10 of the above embodiment.
First, the preference information generation unit 106 of the server device 500 accumulates the preference information 108 generated based on the still image obtained from the user designation for the moving image in the profile information storage unit 502 (step S601). Then, the user classifying unit 512 of the server device 500 performs data mining processing on the viewing user preference information 108 accumulated in the profile information storage unit 502 (step S603). Then, the user classifying unit 512 classifies the viewing user into at least one field in which the viewing user is interested (step S605).

Then, the recommended condition generation unit 514 displays a plurality of different recommended actions suggested for each field, and recommended conditions indicating the relationship between the condition patterns when proposing a plurality of recommended actions, and the preference information of the viewing user Originally generated for each viewing user (step S607).
Then, the recommended behavior output unit 516 of the recommended behavior generation unit 504 digitizes the viewing user's behavior pattern generated based on the viewing user's preference information 108, and applies it to the recommended conditions, so that the Is selected for the viewing user, and recommended action information 506 indicating recommended actions is output (step S609).

As described above, according to the information providing system 1 of the embodiment of the present invention, the same effect as that of the above embodiment can be obtained, and the object or scene obtained from the still image corresponding to the time designated by the user can be handled. Can suggest recommended actions.

(Sixth embodiment)
FIG. 18 is a functional block diagram showing a configuration of the information providing system 1 according to the embodiment of the present invention.
The information providing system 1 of the present embodiment is different from the above-described embodiment in that it has a configuration that indicates that a recommended action has been proposed.
As shown in FIG. 18, the server device 600 of the present embodiment is similar to the server device 500 of the above-described embodiment of FIG. 13, the object specifying unit 104, the preference information generating unit 106, the profile information storage unit 502, and the recommendation An action generation unit 504 is provided, and a feature information reception unit 602 and a recommended action transmission unit 604 are further provided. In addition, the moving image display device 610 of the present embodiment has the same configuration and function as the moving image display device 10 of the above embodiment. The moving image display device 610 of the present embodiment further includes a feature extraction unit 611, a feature information transmission unit 612, a recommended behavior reception unit 614, a recommended behavior storage unit 616, a recommended behavior notification unit 618, and a recommended behavior presentation unit. 620.
Note that the server device 600 of this embodiment includes at least one of the server device 200 of the above embodiment of FIG. 7, the server device 300 of the above embodiment of FIG. 9, and the server device 400 of the above embodiment of FIG. Can also be included.

In the information providing system 1 according to the embodiment of the present invention, the server device 600 is generated by the recommended action generation unit 504 from the viewing user preference information 108 obtained based on the still image obtained from the user designation for the moving image. The recommended action transmission unit 604 for transmitting the recommended action information 506 to the video display device 610 of the viewing user.
The video display device 610 receives the recommended behavior information 506 suitable for the viewing user from the server device 600, and outputs the recommended behavior receiving unit 614 that sequentially accumulates in the recommended behavior accumulation unit 616 and the video of the video display device 610. Out of the screen area, an indicator indicating that the recommended behavior information 506 has been provided is output to notify the viewing user, and the recommended behavior information accumulated in the recommended behavior accumulation unit 616 in response to the notification. And a recommended action presentation unit 620 that accepts an instruction from the viewing user and presents the recommended action information 506 to the viewing user.

In the information providing system 1 according to the present embodiment, the moving image display device 610 extracts image feature information from a still image corresponding to a point in time specified by the viewing user in the moving image output to the moving image display device 610. An extraction unit 611 and a feature information transmission unit 612 that transmits image feature information extracted by the feature extraction unit 611 are provided.
The server device 600 includes a feature information receiving unit 602 that receives image feature information from the moving image display device 610.

FIG. 19 is a diagram for explaining a recommended action notification method for the moving image display device 610 of the information providing system 1 according to the embodiment of the present invention.
As illustrated in FIG. 19A, the recommended action notification unit 618 of the present embodiment displays an indicator 634 at a position that does not hinder the user's video viewing, such as near the outside of the screen area 632 of the display unit 630 of the video display device 610. indicate. The indicator 634 may be, for example, an icon or an animation. The recommended action notification unit 618 notifies the user that the recommended action has been suggested so as not to prevent the user from viewing the video. In addition to displaying on the moving image display device 610 on the display unit 630, an LED (Light Emitting Diode) display provided on the moving image display device 610 may be turned on or blinked for notification. In addition to the display output, notification can also be made by voice output, vibration output, or the like.

Further, in the information providing system 1 according to the embodiment of the present invention, the video display device 610 includes a user access notification unit (recommended behavior presentation unit) that notifies the server device 600 of user access to the recommended behavior provided from the server device 600. 620).

For example, as shown in FIG. 19B, there is a like button 642 for displaying that the user likes the recommended content in the recommended list 640 presented by the recommended action presentation unit 620, and the recommended content by the user. A delete button 644 or the like for deleting from the list can be included while notifying the user that the user does not like.

The user access notification unit of the recommended action presentation unit 620 notifies the server device 600 of the user access by the like button 642 and the delete button 644 and the user access to the URL included in the recommended content presented in the recommendation list 640.

In the server device 600, the user access notified from the video display device 610 is further accumulated in the profile information storage unit 502 as the preference information 108 of the viewing user. The recommended action generation unit 504 generates recommended action information 506 based on the updated viewing user preference information 108.

In the present embodiment, the like button 642 and the delete button 644 are configured to accept the user's will, but the present invention is not limited to this. For example, if a user has actually taken the recommended action, or did not purchase a recommended product, but purchased a similar product or purchased it at another store, It is good also as a structure which acquires the information of what kind of action was taken with respect to information.
Moreover, it is good also as a structure which acquires information, such as whether the recommendation list | wrist 640 was browsed, or it was browsed but which recommendation content was not performed more than fixed time.
As described above, the user access information for the recommended action information is transmitted from the moving image display apparatus 10 to the server apparatus 600, and the server apparatus 600 can learn the user access and update the recommended action.
As a result, even if the server device 600 proposes a recommended action that does not suit the user, it can be corrected while confirming the user's will. By repeating this, the appropriateness of the recommended action information Can be improved.

As described above, according to the information providing system 1 of the embodiment of the present invention, while having the same effect as the above embodiment, it is possible to notify the presence of recommended action information without disturbing the viewing of the user, Since it is possible to confirm whether or not the recommended action proposal is appropriate for the user, it is possible to improve the appropriateness of the recommended action proposal.
In addition, since the user can confirm recommended actions, determine whether or not necessary, and give instructions when it is convenient, viewing is not hindered like a television commercial (Commercial Message).

(Seventh embodiment)
FIG. 20 is a functional block diagram showing the configuration of the information providing system 701 according to the embodiment of the present invention.
The information providing system 701 of this embodiment is different from the above-described embodiment in that the server device includes a plurality of server devices.
As illustrated in FIG. 20, the information providing system 701 includes a user agent server 710, a video analysis server 720, and an information providing server 730.

The user agent server 710 is connected to a user profile database (DataBase: DB) 712, communicates with at least one moving image display device 10, receives image feature information of a still image obtained from user designation for the moving image, and performs video analysis. Transfer to server 720.

The video analysis server 720 is connected to the feature database (DB) 722 and the program database (DB) 724, and collates the image feature information of the still image transmitted from the moving image display device 10 with the feature of the feature database 722. The image analysis is performed, the object is specified, and the result is returned to the user agent server 710.

The user agent server 710 stores information on the identified object in the user profile database 712. The user agent server 710 further accumulates in the user profile database 712 other information related to the user, for example, action history information 530 shown in FIG. Note that the user profile database 712 may use, for example, big data stored in houseware or the like.

The recommended behavior database 732 holds a plurality of recommended behaviors provided in advance for each field.
The information providing server 730 receives preference information generated based on the still image obtained from the user designation for the moving image from the user agent server 710, generates recommended behavior suitable for the user based on the user's behavior history, and the user Return to the agent server 710.

The user agent server 710 transmits the recommended action information received from the information providing server 730 to the video display device 10. The moving image display apparatus 10 accumulates the recommended action information received from the user agent server 710 and displays an indicator 634 as shown in FIG. 19A to notify the user that the recommended action information has been provided. .

The operation of the information providing system 701 according to the embodiment of the present invention configured as described above will be described below.
First, the user views a program using the dedicated viewing application 42 on the moving image display device 10 such as a smartphone or a tablet. The broadcast is distributed to the moving image display apparatus 10 via Wi-Fi (Wireless Fidelity) or the like through a dedicated STB 3.

The user touches the screen when there is an interesting program, scene, or performer appearance. In addition, when interested in a performer or an appearing object, the user touches the object. The viewing application 42 stores the moment when the user touches as a still image and performs image analysis (primary processing).

The primary processing here refers to processing that extracts and digitizes screen feature points. As a result, the data can be made much smaller than the image data sent as it is.

The viewing application 42 detects the touch and transmits data to that effect to the user agent server 710 on the cloud 5. The data to be transmitted at that time includes, for example, “channel”, “broadcast time”, “touch position”, and the like in addition to the image feature information. The user agent server 710 on the cloud 5 transmits the received information to the video analysis server 720.

The video analysis server 720 analyzes the scene from the received data, and notifies the user agent server 710 of a plurality of words (names, etc.) of objects (people, things, music, scenes, etc.) that the user may be interested in. To do. When analyzing the scene, the tag data allocated in the program is also referred to. The tag data is a word assigned to a specific area (a part or all of a scene) of a specific scene.

In addition, the video analysis server 720 reanalyzes the image analysis data subjected to the primary processing on the moving image display apparatus 10 side using the analysis engine, and finds the object being shown. Upon receiving the word, the user agent server 710 accepts the word and the user's past behavior data (behavior feature, behavior history (input word such as SNS), browse history, program viewing history, purchase history, mail history, etc. And the data collected by the viewing application and the behavioral characteristics of the user analyzed from the data) and personal data (data that the user has permitted use, such as age, occupation, or hobby) are transmitted to the information providing server 730. To do.

The information providing server 730 is based on mining processing of big data (recommendation engine, etc.) and recommended analysis formulas created by humans (conditional formulas for deriving recommendations from the user's feature points). Is derived and notified to the user agent server 710.

The user agent server 710 transmits the derived “recommended behavior” to the viewing application 42 of the user's moving image display device 10. The viewing application 42 accumulates the received “recommended behavior” in the “recommended behavior queue”. Note that the viewing screen and the recommended action queue screen are separate, and even if the viewing application 42 receives “recommended behavior”, the contents are not immediately displayed directly on the screen.

The viewing application 42 informs the user that “recommended action” has been received from the user agent server 710 by blinking the indicator 634 of the moving image display device 10 or lighting / changing the color of a mark (button or the like) displayed on the edge of the screen. Notice.

The user sees the above notification, recognizes that the recommended action has come, and refers to the “recommended action queue” at an arbitrary timing. Note that the timing of referring may be after the end of the program.
The user first refers to the “title list” in the “recommended action queue”. “Recommended actions” that are not of interest at this point can also be discarded (click the discard button). The viewing application 42 also records that it has been discarded, and uses it as a material (parameter) when creating the next recommended action.

The user clicks on the title of “Recommended Actions” that interests them. The viewing application 42 records that the user clicked without discarding it, and uses it as a material for creating the next “recommended action”.
“Recommended action proposal content” and “related URL” are displayed for the recommended action clicked. A “I like it” button is also displayed. If the user likes the proposal, the user can notify the agent of his / her preference by clicking this “I like it” button. Also, when the related URL is clicked, it is determined that the user is “interested”. The viewing application 42 also records all the above operations and uses them as the material for creating the next “recommended action”.

As described above, according to the information providing system 701 of the present embodiment, the same effects as those of the above-described embodiment can be obtained, and the user's preference can be accurately estimated from the video viewing action.

(Eighth embodiment)
The information providing system of this embodiment is different from the above embodiment in that it has a configuration in which tag information is given to a moving image and preference information is generated according to the priority of the tag information.
The server device of the present embodiment can be combined with the configuration of any server device of the above embodiment.

In the information providing system according to the embodiment of the present invention, the server device (video analysis server 720) acquires information about a moving image, assigns at least one tag (word) to at least a part of the moving image, A tag information adding unit (not shown) that gives priority and generates tag information of the video, and a tag that acquires tag information of the video corresponding to the still image obtained from the user designation to the video according to the priority An information acquisition unit (not shown).
The preference information generation unit generates preference information of the viewing user including tag information acquired according to the priority by the tag information acquisition unit.

Here, the priority may not be included in the preference information, and a predetermined number of high priority tags may be acquired. Tag information including priority may be acquired.

In the information providing system of the present embodiment, the server device (information providing server 730) registers an object (product, service, actor, etc.) to which a tag is to be assigned, a tag to be assigned to the object, and a tag priority ( A registration reception unit (not shown) that accepts advertisement users (from dealers) and a tag information addition unit (not shown) search for videos related to the objects received by the registration reception unit, and tag the found related videos A charging unit (not shown) that charges (according to the advertising user (seller)) according to the priority of the tag included in the tag information generated by the assignment is further provided.

In the information providing system of the present embodiment, the server device (information providing server 730) analyzes personal profile information of a plurality of viewing users, generates a trend information, and advertises the trend information. An information providing unit (not shown) provided to a user (sales person).
The registration receiving unit receives a tag and an object determined by the advertising user (seller) based on the trend information.

The operation of the information providing system 701 of the present embodiment configured as described above will be described below.
First, we will explain the recommended sequence of actions by sellers.
The information providing server 730 periodically provides information on the trend of this week to apparel retailers. Trends are provided based on the number of user searches, purchase behavior, and tweets about program content.
The sales (advertisement request) dealer registers the product in the user agent server 710 based on the trend information. The user agent server 710 registers the registered product list in the information providing server 730.

The information providing server 730 searches for a program related to the registered product, and if it exists, notifies the user agent server 710 of the sales (advertisement request) dealer of the program.
A sales (advertisement request) dealer who knows the existence of a cat-patterned dress-related program requests that the “cat-patterned dress” tag be put on (purchase of tagging right).

The service provider registers the keyword of the tag and the product in the information providing server 730. In addition, the tag and position data are added to the timeline data of the program corresponding to the video analysis server 720. These instructions for registration and the like can be operated and executed according to an operation screen on a service provider's terminal device (not shown) or the like. As a result, tag data (related information such as name and type) is reliably transmitted when an image analysis request for touching the corresponding position is received.

As described above, according to the information providing system of the embodiment of the present invention, the same effects as those of the above embodiment can be obtained, and further, expansion into new business such as addition of tag information and provision of trend information. Can face.

For example, in the case of a general television CM, there are not a few viewers who take an action of watching only the program and not watching the CM because the program and the CM are clearly distinguished. This is due to the following feeling when viewed from the viewer's perspective.
(1) I want to watch the program, not the CM (2) I was crazy about the program so I don't want to be distracted by the CM (3) CMs of uninteresting products / services I don't want to watch. This makes channel changes or standing when watching broadcasts, and skips CMs when watching recordings, so that I can't watch the CMs. This trend is more pronounced with the recent increase in recorded viewing rates and increased time-saving viewing solutions.

On the other hand, Internet advertising is expanding the market for sluggish TV commercials. It is trying to provide accurate information by providing advertisements of contents that are likely to be of interest to each user, such as behavior matching advertisements based on purchase histories and purchase histories.
However, it is not perfect. For example, it makes sense to recommend “tance” to people who applied for an estimate of “moving”, but “tance” is often recommended for people who bought “tance”. Such a recommendation is not appropriate for the user because ordinary people do not buy large pieces of furniture such as chests.
That is, although this matching advertisement is more advanced than a dripping advertisement such as TV, it is far from being said that the contents are “feeling good” and “feeling good” for the user.

The information providing system according to the embodiment of the present invention solves the problems as described above and has the following effects.
(1) The user can recommend an action related to an event that he / she is interested in.
Since it is possible to selectively provide a good “recommended action” rather than a vague “recommended”, information useful to the user can be selectively acquired. By providing the user with carefully selected options, it is possible to provide a user-friendly situation.

(2) The recommendation engine learns your preferences.
A device (agent) that provides information to the user can be easily learned without intending his / her preference. In the initial operation of the device, the user's preference is not understood, and since the accumulated data is small, a recommendation error for suggesting recommended actions (“mistranslation” in terms of Japanese input) occurs. However, the user can notify the apparatus that he / she does not like the “misrecommendation” with a single touch operation. By repeating it, you can be made to suggest "recommended behavior" as intended.
The information (advertisement) provider can provide the information only to those who want the information.
Since information can be provided to a user who matches the information, effective information provision (advertisement) can be performed efficiently. In particular, it is possible to provide information in a timely manner when an “interesting” action occurs, so that it is possible to perform advertising behavior that “sounds” the user.

(3) Do not disturb program viewing.
Even if the user makes an intention to be “interested”, it is merely an intention to “be interested in the screen (program)”, and does not perform an action such as jumping to a direct link.
That is, even for a broadcaster, program viewing is not interrupted by jumping to a link or the like.
“Recommended behavior” based on the interest expressed by the user's action is displayed as an indicator 634 at the bottom of the screen. However, the recommended behavior is executed only by indicating that “recommended behavior” is queued. It can be carried out carefully after completion.

(4) It can be used without distinction between broadcasting / recording.
Since scenes can be discriminated by broadcasting or recording, “recommended actions” can be provided in the same way even during playback of recorded programs. Even in such a case, there is no CM skip and the like, so the advertising effect can be maintained.

(5) The user interface is simple.
The user can convey his / her will by simply touching an object of interest on the screen.

As mentioned above, although embodiment of this invention was described with reference to drawings, these are the illustrations of this invention, Various structures other than the above are also employable.
For example, in another embodiment, the screen can be specified by distinguishing whether the object specified by the user is an object displayed on the screen or music played in the background, depending on the method of specifying the screen by the user. May be. A designation method such as a single tap for an object or a swipe for music can be determined in advance, and when the screen is touched, information indicating which operation has been performed can be included in the designation information and transmitted to the server apparatus. The server device determines that an object in a still image is specified for a single tap, and music played in a back is specified for a swipe based on information on which operation method is specified.
In addition, music can be specified from objects in still images or timeline information.

Further, in another embodiment, it may be configured to detect that the profile information including the viewing user's preference information is updated and update the recommended action based on the updated information. There are various timings for generating recommended actions other than when the viewing user touches the screen. For example, generate recommended actions at a predetermined cycle, date and time, time of user's anniversary (birthday of the person or family, wedding anniversary, etc.), before and after the event (Christmas, Valentine's Day, etc.) Also good. Alternatively, a recommended action may be provided to the user by an approach from a distributor (advertiser), a program producer, or the like.

(Example 1)
As an embodiment of the present invention, a matching example (purchasing recommendation) will be described.
(Assumption)
Advertiser A A cat-patterned stole and dress are exhibited.
Viewer A We often tweet about clothes on Twitter (female). I bought a cat-patterned dress last week.
Viewer B I often watch programs that star S, a fan of S, appears.
Viewer C: I like fashion items for clothes. However, you can check it on the web and buy it at an actual store.

For example, the information of the advertiser A is acquired from an exhibition history on the Internet auction site.
Information about the viewer A is acquired from an SNS history, a net shopping use history, and the like.
Information about the viewer B is acquired from profile information, viewing history, and the like.
Information about the viewer C is acquired from profile information, visit history, and the like.

I am watching fashion-related programs. The guest is S, a talent.
Mr. S appeared wearing a cat-patterned dress, which is a trendy item this winter.
She wears a cat-patterned dress and a stall.
Cat pattern is popular this winter, but it is not mentioned in the program.
In the guest interview, when the full body photo of the guest was obtained, all three viewers A, B, and C touched the screen (clothes).

At this time, the image feature information of the still image when touched is transmitted from the moving image display device 10 of each viewer to the user agent server 710. Then, the image feature information is transferred from the user agent server 710 to the video analysis server 720.
In the video analysis server 720, the object specifying unit 104 specifies an object from the image feature information. Here, “Mr. Actress S” and “Cat Pattern Dress” are specified and transmitted from the video analysis server 720 to the user agent server 710.
In the user agent server 710, the preference information generation unit 106 accumulates “actress S” and “cat pattern dress” in the user profile database 712 as the preference information 108 of each viewer.
In the information providing server 730, the recommended action generation unit 504 generates recommended actions based on the preference information 108 of each viewer with reference to the user profile database 712.

As a result, the following “behavior” was the best recommendation for the three viewers.
Viewer A Product introduction on EC site of cat-patterned stall Viewer B Product introduction on cat-pattern one-piece EC site Viewer C The closest place to a store selling cat-patterned dress The recommended action information for the site includes the URL of the product introduction page for the cat-patterned stall on the EC site.
The recommended action information for the viewer B includes the URL of the product introduction page of the cat pattern dress on the EC site.
Recommended action information for the viewer C includes the URL of the store's homepage, the URL where the map of the store is posted, the URL of the site that presents navigation information to the store, and the like.

In this way, the viewer can identify the object that the viewer is interested in from the screen information just by touching the screen, and consider it as the user's preference information. Were provided respectively.

(Example 2)
As an embodiment of the present invention, a matching example (planning recommendation) will be described.
(Assumption)
Advertiser A Restaurant A Manager Advertiser B Hanaya B Manager Advertiser C Shoe Store C Manager Audience Viewer X Next week is a wedding anniversary man

Viewer X forgot that next week is a wedding anniversary.
Viewer X was watching a nostalgic trendy drama. A wife of the same age was also watching the trendy drama. In the drama, there was an impressive scene where flowers were handed over at Restaurant A, and his wife added “Like” to this scene.
In addition, his wife recently searched the boot site often. In particular, there were many searches for “half-length brown with fur”.

The video analysis server 720 analyzes the image feature information and designation information of the still image received from the moving image display device 10 via the user agent server 710, and specifies “scene to pass flowers at restaurant A” in the drama. To do. This information is transmitted to the user agent server 710 and stored as preference information of Mr. X's wife.
Based on the viewing history of Mr. X and Mrs. X's wife, the information provision server 730 presents Mr. X with a bouquet of restaurant A and florist B, and a brown half-length boot with fur. Proposed as a recommended action.
Here, the recommended action information for Mr. X includes restaurant A and florist B, the URL of the homepage of the shop selling half-length boots with fur in brown, the URL of the reservation page of the recommended menu of restaurant A, etc. It was.

(Example 3)
As an embodiment of the present invention, a matching example (participation type) will be described.
(Assumption)
Advertiser A Amusement (Sengoku period village) facility operator based on the Sengoku / Edo period motif Advertiser B Planning of armor and imitation swords during the Sengoku period, sales viewers Y Sengoku period likes, armor likes

Viewer Y likes items from the Sengoku period and often watches historical drama. Then, while watching the historical drama, a quiz (created by advertiser B) was displayed at the bottom of the screen.
Advertiser B then presented an armor and sword figure.

Then, from the information providing server 730, participation information for the “Armor Community SNS” was delivered to those who participated in the quiz. Then, when talking about the contents and hobbies of the quiz, an agent made a proposal for “Armored Cosplay OFF Party at Sengoku Era Village” sponsored by Advertiser B.
Since a certain number of people who agreed with the proposal were confirmed, a discounted electronic ticket was sent to them. Advertiser B also provided armor for cosplay and sold it on site.

Example 4
As an embodiment of the present invention, a matching example (sales recommendation: report) will be described.
(Assumption)
Advertiser A Cake shop Cake shop A wanted to increase the variety of cakes to acquire new customers. When a trend was inquired to the information providing server 730, a trend message saying “Christmas cakes move well during Christmas” was received.

So when I registered a chocolate cake, the reaction to it was slow. They clicked, but often did not lead to purchase. Then, it was reported as a report that the cakes that they eventually bought sold more simple ones with no decoration than Mr. A's cake.
Therefore, Mr. A was able to improve sales by adding a simple chocolate cake to the lineup.

Although the present invention has been described with reference to the embodiments and examples, the present invention is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

In addition, when acquiring and using information about a user in the present invention, this shall be done legally.

Hereinafter, examples of the reference form will be added.
1. Server device
Acquire extracted image feature information from the still image corresponding to the time point specified by the viewing user in the video output to the video display device,
Based on the acquired image feature information, identify an object that appears in the still image,
An information providing method for generating preference information of the viewing user including object information to be specified.
2. 1. In the information providing method described in
The server device further includes:
Obtaining designation information including any of a designated position, a designated range, and a designation method of the viewing user in the still image of the moving image output to the moving image display device at a time point designated by the viewing user;
An information providing method for specifying an object that appears in the still image in consideration of any of the designated position, designated range, and designated method.
3. 1. Or 2. In the information providing method described in
The server device further includes:
The content information of the moving image output to the moving image display device at the time specified by the viewing user, the specified time by the viewing user, and the elapsed time from the start of the moving image when specified by the viewing user Get the specified information including any one,
The content that matches the specified information acquired with reference to the content information holding unit that holds the content information of the video and the information of the object related to the content is the content that is output to the video display device And identify
An information providing method of referring to the content information holding unit, acquiring information on an object related to the specified content, and further specifying the acquired information to specify an object appearing in the still image.
4). 3. In the information providing method described in
The server device further includes:
With reference to a content information holding unit that holds timeline information of a plurality of scenes included in the video, obtain timeline information of the identified content,
Based on the acquired timeline information of the content, the scene corresponding to the time point specified by the viewing user is specified,
An information providing method for further generating preference information of the viewing user based on specified scene information.
5. 1. To 4. In the information providing method described in any of the above,
The server device further includes a user information storage unit that accumulates preference information generated based on the still image obtained from user designation to the moving image,
The server device further includes:
Data mining processing is performed on the stored preference information of the viewing user, and the viewing user is classified into at least one field in which the viewing user is interested,
For each of the fields, a plurality of different recommended actions and a recommended condition indicating a correspondence relationship between each condition pattern indicating a combination of conditions for each action of the viewing user indicated by the viewing user's preference information are held. ,
The recommended condition corresponding to the field in which the viewing user is classified is acquired, and the behavior pattern of the viewing user obtained from the viewing user's preference information stored in the user information storage unit is represented by each condition pattern of the recommended condition. An information providing method for generating recommended behavior information indicating at least one recommended behavior among the plurality of different recommended behaviors by applying to.
6). 5. In the information providing method described in
The server device further includes:
From the viewing user's preference information obtained based on the still image obtained from the user designation to the moving image, the recommended action information generated is transmitted to the moving image display device of the viewing user,
The moving image display device
Receiving the recommended behavior information suitable for the viewing user from the server device, sequentially storing,
Output an indicator indicating that the recommended behavior information has been provided outside the video output area of the video display device, and notify the viewing user,
An information providing method for receiving the notification and presenting the recommended behavior information to the viewing user in response to an instruction by the viewing user to browse the accumulated recommended behavior information.
7). 6). In the information providing method described in
The video display device further includes:
Notifying the server device of user access to the recommended behavior provided from the server device,
The server device further includes:
In the user information storage unit, the user access notified from the video display device is further accumulated as preference information of the viewing user,
An information providing method for generating recommended action information based on the updated preference information of the viewing user.
8). 1. In the information providing method according to any one of 7 to 7,
The server device further includes:
Obtaining information about a video, assigning at least one tag to at least a portion of the video, assigning a priority to the tag, and generating tag information for the video,
Obtaining the tag information of the video corresponding to the still image obtained from the user designation to the video according to the priority,
An information providing method for generating the preference information of the viewing user including the tag information acquired according to the priority.
9. 8). In the information providing method described in
The server device further includes:
Accept registration of the object to be given a tag, the tag to be given to the object, and the priority of the tag,
An information providing method for searching for a moving image related to the received object and charging according to the priority of the tag included in the tag information generated by adding the tag to the found related moving image.
10. 9. In the information providing method described in
The server device further includes:
Analyzing user information including the preference information of a plurality of viewing users, generating trend information,
Providing the trend information to advertising users;
An information providing method for receiving a tag and an object determined by the advertising user based on the trend information.
11. 1. To 10. In the information providing method described in any of the above,
The video display device further includes:
Extracting image feature information from a still image corresponding to a point in time specified by the viewing user in the video output to the video display device,
Transmitting the extracted image feature information;
The server device further includes:
An information providing method for receiving the image feature information from the video display device.
12 1. To 10. In the information providing method described in any of the above,
The server device further includes:
Extracting image feature information from a still image corresponding to a time point specified by a viewing user in the moving image output to the moving image display device;
An information providing method for acquiring the extracted image feature information.
13. 11. Or 12. In the information providing method described in
The server device further includes:
An information providing method for acquiring the image feature information extracted by at least one of the moving image display device and the server device.
14 On the computer,
A procedure for acquiring extracted image feature information from a still image corresponding to a time point specified by a viewing user in a moving image output to a moving image display device,
A procedure for identifying an object shown in the still image based on the acquired image feature information;
A program for executing a procedure for generating preference information of the viewing user including object information specified by a procedure for specifying the object.
15. 14 In the program described in
A procedure for acquiring designation information including any one of a designation position, a designation range, and a designation method of the viewing user in the still image of the moving image output to the moving image display device at a time point designated by the viewing user. ,
A program for causing a computer to further execute a procedure for specifying an object shown in the still image in consideration of any one of the designated position, designated range, and designation method.
16. 14 Or 15. In the program described in
The content information of the moving image output to the moving image display device at the time specified by the viewing user, the specified time by the viewing user, and the elapsed time from the start of the moving image when specified by the viewing user Procedure to get specified information including either
By referring to a content information holding unit that holds content information of a moving image and information of an object related to the content, content that matches the specified information acquired by the procedure of acquiring the specified information is displayed on the moving image display device. To identify content that has been output to
Referring to the content information holding unit, acquire information on an object related to the content specified by the procedure for specifying the content, and further specify the acquired information to specify an object that appears in the still image A program for causing a computer to further execute the procedure.
17. 16. In the program described in
A procedure for acquiring timeline information of the content specified by the procedure for specifying the content with reference to a content information holding unit that holds timeline information of a plurality of scenes included in the video,
A procedure for identifying a scene corresponding to a time point designated by the viewing user based on the timeline information of the content acquired by the procedure of acquiring the timeline information;
A program for causing a computer to further execute a procedure of further generating preference information of the viewing user based on scene information specified by the procedure of specifying the scene.
18. In a computer equipped with a storage device,
A procedure for storing the preference information generated by the procedure for generating the preference information based on the still image obtained from the user designation for the moving image in the storage device;
A procedure for performing data mining on the stored preference information of the viewing user and classifying the viewing user into at least one field in which the viewing user is interested;
For each field, a recommended condition indicating a correspondence relationship between each of a plurality of different recommended actions and each condition pattern indicating a combination of conditions for each action of the viewing user indicated by the viewing user's preference information is stored in the memory The procedure to hold in the device,
The recommendation condition corresponding to the field into which the viewing user is classified is acquired, and the behavior pattern of the viewing user obtained from the viewing user's preference information stored in the storage device is applied to each condition pattern of the recommendation condition By doing, the program for performing the procedure which produces | generates the recommended action information which shows at least 1 recommended action in the said some different recommended action.
19. 18. In the program described in
The recommended behavior information generated by the procedure for generating the recommended behavior from the preference information of the viewing user obtained based on the still image obtained from the user designation for the moving image is used as the moving image display device of the viewing user. A program for causing a computer to further execute the procedure of transmitting to the computer.
20. 19. In the program described in
In the storage device, user access notified from the video display device for the recommended action provided to the video display device is further accumulated as preference information of the viewing user,
A program for causing a computer to further execute a procedure for generating recommended behavior information based on the updated preference information of the viewing user.
21. 14 To 20. In any program,
A procedure for acquiring information about a video, assigning at least one tag to at least a part of the video, assigning a priority to the tag, and generating tag information of the video,
A procedure for acquiring tag information of a moving image corresponding to the still image obtained from user designation to the moving image according to the priority,
The program for making a computer perform further the procedure which produces | generates the said preference information of the said viewing user containing the said tag information acquired according to the said priority by the procedure which acquires the said tag information.
22. 21. In the program described in
A procedure for accepting registration of an object to be given a tag, a tag to be given to the object, and a priority of the tag;
In the procedure of assigning the tag information, the video associated with the object received in the procedure of accepting the registration is searched, and the tag included in the tag information generated by adding the tag to the found related video is generated. A program for causing a computer to further execute a charging procedure according to the priority.
23. 22. In the program described in
Analyzing user information including the preference information of a plurality of viewing users, and generating trend information;
Providing the trend information to the advertising user;
A program for causing a computer to further execute a procedure of receiving a tag and an object determined by the advertising user based on the trend information.
24. 14 Thru 23. In any program,
Causing the computer to further execute a procedure of receiving, from the moving image display device, image feature information extracted from a still image corresponding to a time point specified by a viewing user in the moving image output to the moving image display device. Program for.
25. 14 Thru 23. In any program,
A procedure for extracting image feature information from a still image corresponding to a time point specified by a viewing user in the moving image output to the moving image display device;
A program for causing a computer to further execute a procedure for acquiring the image feature information extracted in the procedure for extracting the image feature information.
26. To a computer that can be connected to the server device,
Steps to view the video,
A procedure for acquiring a still image corresponding to a point in time specified by the viewing user in the video output in the procedure for displaying the video,
A procedure for transmitting the still image to the server device;
A procedure for receiving recommended actions generated based on the still image from the server device and sequentially storing them.
A procedure for outputting an indicator indicating that the recommended action is provided from the server device outside the area where the video is output, and notifying the viewing user,
A program for receiving a notification by the notification procedure and executing a procedure for presenting recommended behavior information to the viewing user in response to an instruction from the viewing user to browse the accumulated recommended behavior.
27. 26. In the program described in
The recommended behavior information suitable for the viewing user, which is generated from the viewing user's preference information obtained from the still image obtained from the user designation for the moving image, is received from the server device and sequentially accumulated. procedure,
A procedure for outputting an indicator indicating that the recommended action information has been provided outside the region where the video is being output, and notifying the viewing user,
A program for causing a computer to further execute a procedure of presenting the recommended behavior information to the viewing user in response to an instruction by the viewing user to browse the accumulated recommended behavior information in response to the notification. .
28. 27. In the program described in
A program for causing a computer to further execute a procedure for notifying the server device of a user access to the recommended behavior provided from the server device.
29. 27. Or 28. In the program described in
A procedure for extracting image feature information from a still image corresponding to a time point designated by a viewing user in the output video;
A program for causing a computer to further execute a procedure of transmitting the image feature information extracted by the procedure of extracting the image feature information to a server device.

This application claims priority based on Japanese Patent Application No. 2013-054010 filed on Mar. 15, 2013, the entire disclosure of which is incorporated herein.

Claims

An information providing system comprising a video display device and a server device,
The server device
Feature information acquisition means for acquiring image feature information extracted from a still image corresponding to a time point designated by the viewing user in the video output to the video display device;
Based on the image feature information acquired by the feature information acquisition means, an object specifying means for specifying an object that appears in the still image;
An information providing system comprising: preference information generating means for generating preference information of the viewing user including object information specified by the object specifying means.
In the information provision system of Claim 1,
The server device
Designation for obtaining designation information including any one of the designation position, the designation range, and the designation method of the viewing user in the still image of the moving image output to the moving image display device at the time designated by the viewing user An information acquisition means;
The information providing system in which the object specifying unit specifies an object appearing in the still image in consideration of any one of the specified position, a specified range, and a specifying method.
In the information provision system according to claim 1 or 2,
The server device
The content information of the moving image output to the moving image display device at the time specified by the viewing user, the specified time by the viewing user, and the elapsed time from the start of the moving image when specified by the viewing user Specified information acquisition means for acquiring specified information including any of them,
Referring to a content information holding unit that holds video content information and object information related to the content, the content that matches the designation information acquired by the designation information acquisition unit is output to the video display device Content identifying means for identifying that the content has been processed,
The object specifying unit refers to the content information holding unit, acquires information on an object related to the content specified by the content specifying unit, and further captures the acquired information, and captures it in the still image. An information provision system that identifies objects.
In the information provision system according to claim 3,
The server device
A timeline information acquisition unit that acquires the timeline information of the content specified by the content specifying unit with reference to a content information holding unit that holds timeline information of a plurality of scenes included in the video;
Scene specifying means for specifying a scene corresponding to the time point designated by the viewing user based on the timeline information of the content acquired by the timeline information acquiring means;
The preference information generating means is an information providing system for further generating preference information of the viewing user based on the scene information specified by the scene specifying means.
In the information provision system in any one of Claims 1 thru | or 4,
The server device
A user information storage unit that accumulates preference information generated by the preference information generation unit based on the still image obtained from the user designation to the moving image;
User classifying means for classifying the viewing user into at least one field in which the viewing user is interested by data mining the stored preference information of the viewing user;
For each of the fields, a plurality of different recommended actions and a recommended condition indicating a correspondence relationship between each condition pattern indicating a combination of conditions for each action of the viewing user indicated by the viewing user's preference information is held. Recommended condition holding means;
The recommended condition corresponding to the field in which the viewing user is classified is acquired, and the behavior pattern of the viewing user obtained from the viewing user's preference information stored in the user information storage unit is represented by each condition pattern of the recommended condition. An information providing system comprising: recommended action generating means for generating recommended action information indicating at least one recommended action among the plurality of different recommended actions.
In the information provision system of Claim 5,
The server device
The recommended behavior information generated by the recommended behavior generation means is transmitted to the video display device of the viewing user from the preference information of the viewing user obtained based on the still image obtained from the user designation for the video. With a recommended action transmission means,
The video display device
Receiving the recommended action information suitable for the viewing user from the server device, and sequentially storing recommended action receiving means;
A recommended action notification means for outputting an indicator indicating that the recommended action information has been provided outside the area of the moving picture display device that outputs the video, and notifying the viewing user;
In response to the notification, an information providing system further comprising recommended action presenting means for presenting the recommended action information to the viewing user in response to an instruction by the viewing user to browse the accumulated recommended action information. .
In the information provision system according to claim 6,
The video display device
User access notifying means for notifying the server device of user access to the recommended behavior provided from the server device;
In the server device,
In the user information storage unit, the user access notified from the video display device is further accumulated as preference information of the viewing user,
An information providing system in which the recommended behavior generating means generates recommended behavior information based on the updated preference information of the viewing user.
In the information provision system in any one of Claims 1 thru | or 7,
The server device
Tag information giving means for acquiring information about a video, giving at least one tag to at least a part of the video, giving a priority to the tag, and generating tag information of the video;
Tag information acquisition means for acquiring tag information of a moving image corresponding to the still image obtained from user designation to the moving image according to the priority,
The preference information generating means
The information provision system which produces | generates the said preference information of the said viewing user containing the said tag information acquired according to the said priority by the said tag information acquisition means.
The information providing system according to claim 8,
The server device
An object to be given a tag, a tag to be given to the object, a registration accepting means for accepting registration of the priority of the tag,
The tag information adding unit searches the moving image related to the object received by the registration receiving unit, and generates the priority of the tag included in the tag information generated by adding the tag to the found related moving image. And an information providing system further comprising a charging means for charging.
In the information provision system of Claim 9,
The server device
Trend information generating means for analyzing user information including the preference information of a plurality of viewing users and generating trend information;
An information providing means for providing the trend information to the advertising user,
The registration receiving means is an information providing system for receiving a tag and an object determined by the advertising user based on the trend information.
In the information provision system in any one of Claims 1 thru | or 10,
The video display device
A feature extraction unit that extracts image feature information from a still image corresponding to a point in time specified by the viewing user in the video output to the video display device;
A transmission unit that transmits the image feature information extracted by the feature extraction unit,
The server device
An information providing system comprising a receiving unit that receives the image feature information from the moving image display device.
In the information provision system in any one of Claims 1 thru | or 10,
The server device
A feature extraction unit that extracts image feature information from a still image corresponding to a point in time specified by a viewing user in the video output to the video display device;
The information providing system for acquiring the image feature information extracted by the feature extraction unit.
The information providing system according to claim 11 or 12,
The information providing system for acquiring the image feature information extracted by at least one of the feature extraction unit of the moving image display device and the feature extraction unit of the server device.
A server device connected to the video display device,
Feature information acquisition means for acquiring image feature information extracted from a still image corresponding to a time point designated by the viewing user in the video output to the video display device;
Based on the image feature information acquired by the feature information acquisition means, an object specifying means for specifying an object that appears in the still image;
A preference information generating unit configured to generate the preference information of the viewing user including the object information specified by the object specifying unit.
A video display device connected to the server device,
A video display means for displaying a video;
Obtaining means for obtaining a still image corresponding to a time point designated by a viewing user in the moving picture output to the moving picture display means;
A still image transmitting means for transmitting the still image to the server device;
The recommended action generated based on the still image is received from the server device, and the recommended action is sequentially accumulated, and an indicator indicating that the recommended action is provided from the server device outside the area where the moving image is output And a recommended action notification means for notifying the viewing user,
In response to the notification by the recommended behavior notification means, a moving image display device comprising: presentation means for presenting recommended behavior information to the viewing user in response to an instruction from the viewing user to browse the accumulated recommended behavior .
The moving image display apparatus according to claim 15,
A moving image display device further comprising feature information transmission means for extracting image feature information from the still image and transmitting it to the server device.
The moving image display device according to claim 15 or 16,
A moving image display apparatus comprising: update means for transmitting user access to the recommended action information presented by the presenting means to the server apparatus and updating the recommended action.
The moving image display device according to any one of claims 15 to 17,
A designated position, a designated range, a designation method, content information, a designated time by the viewing user, and a designated time of the viewing user in the still image of the moving image output by the moving image display means when the viewing user designates; A moving image display device further comprising moving image information transmitting means for transmitting, to the server device, specified information including any of the elapsed time from the start of the moving image when specified by the viewing user.
The moving image display device according to any one of claims 15 to 18,
A moving image display apparatus further comprising user information providing means for transmitting the viewing history information of the viewing user to the server and providing the information as user information.
Server device
Acquire extracted image feature information from the still image corresponding to the time point specified by the viewing user in the video output to the video display device,
Based on the acquired image feature information, identify an object that appears in the still image,
An information providing method for generating preference information of the viewing user including object information to be specified.
On the computer,
A procedure for acquiring extracted image feature information from a still image corresponding to a time point specified by a viewing user in a moving image output to a moving image display device,
A procedure for identifying an object shown in the still image based on the acquired image feature information;
A program for executing a procedure for generating preference information of the viewing user including object information specified by a procedure for specifying the object.
In a computer equipped with a storage device,
A procedure for storing the preference information generated by the procedure for generating the preference information based on the still image obtained from the user designation for the moving image in the storage device;
A procedure for performing data mining on the stored preference information of the viewing user and classifying the viewing user into at least one field in which the viewing user is interested;
For each field, a recommended condition indicating a correspondence relationship between each of a plurality of different recommended actions and each condition pattern indicating a combination of conditions for each action of the viewing user indicated by the viewing user's preference information is stored in the memory The procedure to hold in the device,
The recommendation condition corresponding to the field into which the viewing user is classified is acquired, and the behavior pattern of the viewing user obtained from the viewing user's preference information stored in the storage device is applied to each condition pattern of the recommendation condition By doing, the program for performing the procedure which produces | generates the recommended action information which shows at least 1 recommended action in the said some different recommended action.
To a computer that can be connected to the server device,
Steps to view the video,
A procedure for acquiring a still image corresponding to a point in time specified by the viewing user in the video output in the procedure for displaying the video,
A procedure for transmitting the still image to the server device;
A procedure for receiving recommended actions generated based on the still image from the server device and sequentially storing them.
A procedure for outputting an indicator indicating that the recommended action is provided from the server device outside the area where the video is output, and notifying the viewing user,
A program for receiving a notification by the notification procedure and executing a procedure for presenting recommended behavior information to the viewing user in response to an instruction from the viewing user to browse the accumulated recommended behavior.