JPWO2020090790A1 - Information processing equipment - Google Patents

Abstract

The user device generates a first keyword generation unit that generates a first keyword based on the user's voice, and a plurality of second keywords that have a one-to-one correspondence with a plurality of object images extracted from an image indicated by an image signal. Based on the degree of relevance between the second keyword generation unit, each of the plurality of second keywords, and the first keyword, a comment is made from among the plurality of second keywords generated by the second keyword generation unit. It includes a specific unit that specifies the target keyword and a comment generation unit that generates a comment related to the target keyword.

Description

The present invention relates to an information processing device.

Patent Document 1 discloses a technique for displaying a recommendation regarding an object image when a user specifies an object image in an image displayed on a display device using a pointing device. Further, Patent Document 2 discloses a technique for recommending information about an object image specified by the user by voice in response to the voice of the user.

Japanese Unexamined Patent Publication No. 2017-228177 Japanese Unexamined Patent Publication No. 2013-88806

However, in the conventional technique, the user needs to specify the object image. That is, when the user does not specify the object image, it is not possible to generate a comment such as a recommendation in response to the user's ambiguous remark.

In order to solve the above problems, the information processing apparatus according to the preferred embodiment of the present invention includes a first keyword generation unit that generates a first keyword based on a user's voice, and a plurality of information processing devices extracted from an image indicated by an image signal. A second keyword generation unit that generates a plurality of second keywords having a one-to-one correspondence with the object image of the above, and the plurality of said based on the degree of association between each of the plurality of second keywords and the first keyword. A specific unit for specifying a target keyword to be commented from the second keyword of the above, and a comment generation unit for generating a comment related to the target keyword are provided.

According to the information processing apparatus according to the present invention, a comment can be generated in response to an ambiguous remark by the user without the user specifying an object image to be commented.

It is a block diagram which shows the whole structure of the service system which concerns on 1st Embodiment of this invention. It is a block diagram which illustrates the hardware composition of the user apparatus used in this embodiment. It is explanatory drawing which shows the data structure of the keyword table used in this embodiment. It is a functional block diagram which shows the function of the user apparatus used in the same embodiment. It is explanatory drawing which shows an example of the object image of the same embodiment. It is a flowchart which shows the operation of the user apparatus used in this embodiment. It is a functional block diagram which shows the function of the user apparatus used in 2nd Embodiment. It is a flowchart which shows the operation of the user apparatus used in this embodiment. It is explanatory drawing for demonstrating the evaluation result of the object image in the same embodiment. It is a functional block diagram which shows the function of the user apparatus used in 3rd Embodiment. It is a flowchart which shows the operation of the user apparatus used in this embodiment. It is a functional block diagram which shows the function of the user apparatus used in 4th Embodiment. It is explanatory drawing which shows the memory content of the evaluation table used in the same embodiment. It is a flowchart which shows the operation of the user apparatus used in this embodiment.

[1. First Embodiment]
[1.1. Service system configuration]
FIG. 1 is a block diagram showing an overall configuration of a service system according to the first embodiment of the present invention. The service system 1 shown in FIG. 1 provides a moving image distribution service. The moving image distribution service provides, for example, contents of a movie or terrestrial digital broadcasting.

As illustrated in FIG. 1, the service system 1 includes user devices 20_1 to 20_m (m is an integer of 1 or more) managed by users U_1 to U_m, a network NW, and a video distribution server 10. In the following description, when the same type of elements are not distinguished, only the common number among the reference codes is used as in the user device 20 or the user U.

The user device 20 is an information processing device that processes various types of information. The user device 20 is, for example, a portable information processing device such as a smartphone or a tablet terminal. However, any information processing device can be adopted as the user device 20. The user device 20 may be, for example, a terminal-type information device such as a personal computer.

The user device 20 receives the image signal Sg transmitted from the video distribution server 10 and displays an image, or transmits the image signal Sg to the television receiver 30 and displays the image on the television receiver 30. Can be made to.
User U may speak while watching the video. For example, user U may state his or her impressions of the video or tweet. In this case, although the user U's remark is related to the video, it is not possible to uniquely identify which object included in the image of the video the remark is related to because the remark is ambiguous. There are many. The user device 20 has a function of generating a comment such as a recommendation in response to an ambiguous statement of the user U.

[1.2. User device configuration]
FIG. 2 is a block diagram illustrating a hardware configuration of the user device 20. The user device 20 is a computer system including a processing device 21, a storage device 22, a communication device 23, an output device 24, an input device 25, a short-range wireless communication device 26, a GPS (Global Positioning System) device 27, and a bus 28. It will be realized. The processing device 21, the storage device 22, the communication device 23, the output device 24, the input device 25, the short-range wireless communication device 26, and the GPS device 27 are connected by a bus 28 for communicating information. The bus 28 may be composed of a single bus or may be composed of different buses between the devices. Each element of the user device 20 may be composed of a single device or a plurality of devices, and some elements of the user device 20 may be omitted.

The processing device 21 is a processor that controls the entire user device 20, and is composed of, for example, a single or a plurality of chips. The processing device 21 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, registers, and the like. In addition, a part or all of the functions of the processing device 21 are realized by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array). You may. The processing device 21 executes various processes in parallel or sequentially.

The storage device 22 is a recording medium that can be read by the processing device 21. The storage device 22 stores a plurality of programs including the control program PRa executed by the processing device 21, a keyword table TBLa, a comment table TBLb, and various data used by the processing device 21. The storage device 22 is composed of, for example, one or more types of storage circuits such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory).

A plurality of words are stored in the keyword table TBLa. Multiple words are broadly divided into nouns and adjectives. Noun words correspond to keywords. The first keyword KW1 and the second keyword KW2, which will be described later, are included in the noun words stored in the keyword table TBLa. Further, the adjective word is stored in association with the noun word. Adjectives have the function of modifying nouns. The correspondence between adjective words and noun words is determined according to the modification relationship of the words. For example, the adjective word "delicious" is associated with the noun word "food and drink."

FIG. 3 is an explanatory diagram showing a data structure of a noun word stored in the keyword table TBLa. As shown in the figure, the data structure of a noun word is a tree structure in which a plurality of words are layered according to their meanings. In this example, a plurality of words are classified into the first layer to the fourth layer. The number of layers may be 4 or more.

Further, in the keyword table TBLa, the degree of relevance indicating the degree of relevance between the noun word and the noun word is stored. The degree of relevance is determined in consideration of the relationship between the superordinate concept and the subordinate concept, as well as the use and function of the object indicated by the word. For example, "sake" and "wine" are both subordinate concepts of "sake". On the other hand, "Inoguchi" is not a subordinate concept of "Sake", but "Inoguchi" is used to drink "Sake". For this reason, the degree of relevance between "sake" and "inoguchi" is higher than the degree of relevance between "sake" and "wine".

The explanation is returned to FIG. The communication device 23 is a device that communicates with another device via a mobile communication network or a network NW such as the Internet. The communication device 23 is also referred to as, for example, a network device, a network controller, a network card, or a communication module. The communication device 23 can communicate with the video distribution server 10 via the network NW.

The output device 24 informs the user U of various information under the control of the processing device 21. The output device 24 includes a display device 241 and a speaker 242. The display device 241 displays an image. For example, various display panels such as a liquid crystal display panel or an organic EL (Electro Luminescence) display panel are suitably used as the display device 241.
Sound data is supplied to the speaker 242 from the processing device 21. The speaker 242 includes a DA converter. The sound data is converted into an analog signal by the DA converter, and the speaker 242 is driven by the analog signal.

The input device 25 is a device for the user U to input information for using the user device 20. The input device 25 accepts an input operation by the user U. The input device 25 of this example includes a microphone 251 and a touch panel 252. The touch panel 252 detects contact by the user U with respect to the display surface of the display device 241. The touch panel 252 accepts an operation of inputting a code such as a number and a character based on a contact position and an operation of selecting an icon displayed by the display device 241. The microphone 251 converts the voice of the user U into an analog electric signal, and outputs the electric signal as the voice signal Sa. The audio signal Sa is converted into a digital signal by an AD conversion unit (not shown) and supplied to the processing device 21 via the bus 28.

The short-range wireless communication device 26 is a device that communicates with another device by short-range wireless communication. Examples of short-range wireless communication include Bluetooth (registered trademark), ZigBee (registered trademark), WiFi (registered trademark), and the like. As another device, a television receiver 30 or the like is applicable.
The GPS device 27 receives radio waves from a plurality of satellites and generates position information from the received radio waves. The position information indicates the position of the user device 20. The position information may be in any format as long as the position can be specified. The position information indicates, for example, the latitude and longitude of the user device 20. In the present embodiment, it is exemplified that the position information is obtained from the GPS device 27, but the user device 20 may acquire the position information by any other method. For example, the user apparatus 20 may acquire location information using the cell ID assigned to the base station to which the user apparatus 20 communicates. Alternatively, when the user device 20 communicates with an access point of a wireless LAN (Local Area Network) using the short-range wireless communication device 26, the user device 20 is an identification address (MAC) on the network assigned to the access point. (Media Access Control) address) and the actual address (location) may be referred to each other in a database to be associated with each other to acquire location information. Alternatively, the user device 20 receives the ID information included in the advertisement packet compliant with the BLE (Bluetooth Low Energy) standard by using the short-range wireless communication device 26, and acquires the location information based on the ID information. You may.

[1.3. Functions of user device 20]
FIG. 4 is a functional block diagram showing the functions of the user device 20. By reading and executing the control program PRa from the storage device 22, the processing device 21 functions as a first keyword generation unit 210, a second keyword generation unit 220A, a specific unit 230A, and a comment generation unit 240.

The first keyword generation unit 210 generates the first keyword KW1 based on the voice of the user U indicated by the voice signal Sa.
Specifically, the first keyword generation unit 210 analyzes the voice of the user U and extracts nouns and adjectives from the analysis results. The first keyword generation unit 210 identifies the noun as a word of interest when the voice of the user U contains both a noun and an adjective. For example, when the voice of the user U is "the red car is suspicious", the noun "car" is specified as a word of interest. Further, when the voice of the user U does not include a noun but contains an adjective, the first keyword generation unit 210 specifies the adjective as a word of interest. For example, when the voice of the user U is "looks delicious", the adjective "delicious" is specified as a word of interest.

Further, the first keyword generation unit 210 determines whether or not the word of interest is included in the keyword table TBLa. The first keyword generation unit 210 does not generate the first keyword KW1 when the determination result is negative. Therefore, the first keyword KW1 is limited to the keywords included in the keyword table TBLa. On the other hand, when the determination result is affirmative and the attention word is a noun, the first keyword generation unit 210 generates the attention word as the first keyword KW1. When the determination result is affirmative and the attention word is an adjective, the first keyword generation unit 210 refers to the keyword table TBLa and generates a noun word associated with the attention word as the first keyword KW1. do. For example, when the word of interest is "delicious", the first keyword generation unit 210 generates "food and drink" as the first keyword KW1.

In this way, the first keyword generation unit 210 generates the first keyword KW1 related to the user U's remark even when the user U's remark is ambiguous.

Next, the second keyword generation unit 220A generates the second keyword KW2 for each of the object images extracted from the image indicated by the image signal Sg. The second keyword generation unit 220A has an extraction unit 221 and a conversion unit 222.

The extraction unit 221 extracts a plurality of object images from the image indicated by the image signal Sg. There are many object images in the image on one screen.

When the image indicated by the image signal Sg is the image shown in FIG. 5, the images extracted by the extraction unit 221 are, for example, object images OB1 to OB5.

The conversion unit 222 converts each of the plurality of object images OB1 to OB5 extracted by the extraction unit 221 into the second keyword KW2. The conversion unit 222 converts each object image OB into the second keyword KW2 by using, for example, an image recognition model learned by machine learning. However, the second keyword KW2 is included in the keywords stored in the keyword table TBLa. For example, the object image OB1 shown in FIG. 5 is converted into "wine", the object image OB2 is converted into "wine glass", the object image OB3 is converted into "clock", the object image OB4 is converted into "candle", and the object image OB5 is converted into "Western food". The object.

The specific unit 230A selects the target keyword Wx from a plurality of second keyword KW2 generated by the second keyword generation unit 220A based on the degree of association indicating the degree of association between the second keyword KW2 and the first keyword KW1. Identify. More specifically, the specific unit 230A refers to the keyword table TBLa and sets the degree of association between the second keyword KW2 and the first keyword KW1 indicating the degree of association between the second keyword KW2 and the first keyword KW1. Obtain for each group. The specifying unit 230A specifies the second keyword KW2 included in the combination of the second keyword KW2 and the first keyword KW1 having the highest degree of relevance as the target keyword Wx. As will be described later, the comment generation unit 240 generates a comment related to the target keyword Wx specified by the specific unit 230A.

For example, suppose that the user U looks at the image shown in FIG. 5 and says, "It looks delicious." Further, it is assumed that "food and drink" is generated as the first keyword KW1 and "wine", "wine glass", "clock", "candle", and "Western food" are generated as the second keyword KW2. .. In this case, the specific unit 230A has a degree of relation between "food and drink" and "wine", a degree of relation between "food and drink" and "wine glass", a degree of relation between "food and drink" and "clock", and "food and drink". The degree of relevance between "things" and "candles" and the degree of relevance between "food and drink" and "Western food" are acquired with reference to the keyword table TBLa. The specifying unit 230A compares the acquired plurality of relevance degrees and identifies the second keyword KW2 having the highest relevance degree as the target keyword Wx.

The comment generation unit 240 generates a comment related to the target keyword Wx. The comment means an explanation or an explanation of the target keyword Wx. Comments are also a concept that includes recommendations. Therefore, the comment includes information about the product recommended to the user U to purchase in relation to the target keyword Wx and the store handling the product. The comment generation unit 240 generates a comment by reading the comment stored in association with the target keyword Wx from the comment table TBLb. Further, the comment generation unit 240 may access a search site connected to the network NW, acquire information related to the target keyword Wx from the search site, and generate the acquired information as a comment. For example, when the target keyword Wx is "ramen", the comment generation unit 240 may search for a ramen shop near the position information generated by the GPS device 27 or the like and output the search result as a comment.

[1.4. Operation of user device 20]
Next, the operation of the user device 20 will be described. FIG. 6 is a flowchart showing the operation of the user device 20.

First, the processing device 21 identifies the word of interest based on the voice of the user U (step S1). The processing device 21 extracts the word of interest by performing a voice recognition process for converting the voice of the user U into a text and a specific process for specifying a noun and an adjective from the converted text. The extracted word is a noun or an adjective if the noun is not specified.

Next, the processing device 21 determines whether or not the word of interest is included in the keyword table TBLa (step S2). If the word of interest is not included in the keyword table TBLa, the processing device 21 returns the process to step S1 until the word of interest included in the keyword table TBLa is specified (that is, the determination result in step S2 is affirmative). Up to), the processing of steps S1 and S2 is repeated.

If the determination result in step S2 is affirmative, the processing device 21 determines whether or not the word of interest is a noun (step S3). When the word of interest is a noun, the processing device 21 generates the word of interest as the first keyword KW1. In the process of step S1, the processing device 21 extracts the noun or the adjective as the word of interest. Therefore, if the determination result of step S3 is negative, the word of interest is an adjective. In this case, the processing device 21 refers to the keyword table TBLa and generates a noun word associated with the word of interest as the first keyword KW1 (step S5).

Next, the processing device 21 extracts an object image from the image indicated by the image signal Sg (step S6). A single frame image usually has a plurality of object images. Therefore, the processing device 21 extracts a plurality of object images in the processing of step S6. After that, the processing device 21 converts each of the extracted object images into the second keyword KW2 (step S7).

Next, the processing device 21 identifies the target keyword Wx from the plurality of second keywords KW2 generated in step S7 based on the degree of association between the second keyword KW2 and the first keyword KW1.

Next, the processing device 21 generates a comment related to the target keyword Wx (step S9). In the process of step S9, the processing device 21 generates a comment by reading the comment stored in association with the target keyword Wx from the comment table TBLb. The processing device 21 outputs the generated comment by one of the following methods. (a) The processing device 21 causes the display device 241 to display a moving image represented by moving image data in which the generated comment image is superimposed. (b) The processing device 21 uses the short-range wireless communication device 26 to transmit the moving image data in which the generated comment image is superimposed to the television receiver 30. (c) The processing device 21 converts the generated comment into sound data, synthesizes the sound data representing the comment with the sound data of the moving image, and emits the combined result from the speaker 242. (d) The processing device 21 uses the short-range wireless communication device 26 to transmit the synthesis result of the sound data representing the comment and the sound data of the moving image to the television receiver 30. Any combination of the methods (a) to (d) may be used.

Further, the processing device 21 functions as a first keyword generation unit 210 in the processing from step S1 to step S5, functions as an extraction unit 221 in the processing of step S6, and functions as a conversion unit 222 in the processing of step S7. Further, the processing device 21 functions as the specific unit 230A in the processing of step S8, and functions as the comment generation unit 240 in the processing of step S9.

As described above, the information processing device which is an example of the user device 20 includes a first keyword generation unit 210 which generates the first keyword KW1 based on the voice of the user U, and a plurality of information processing devices extracted from the image indicated by the image signal Sg. Generated by the second keyword generation unit 220A that generates the second keyword KW2 for each of the object images of the above, and the second keyword generation unit 220A based on the degree of association between each second keyword KW2 and the first keyword KW1. A specific unit 230A for specifying a target keyword Wx to be a comment from a plurality of second keyword KW2, and a comment generation unit 240 for generating a comment related to the target keyword Wx are provided.

According to this aspect, the user U can generate a comment in response to the ambiguous remark of the user U without designating the object image to be the target of the comment.

Further, the other second keyword KW2 matches the first keyword KW1 in comparison with the degree of relevance between the second keyword KW2 and the first keyword KW1 when the first second keyword KW2 does not match the first keyword KW1. The degree of relevance is high. Therefore, when any one of the plurality of second keyword KW2 matches the first keyword KW1, the specifying unit 230A specifies the second keyword KW2 matching the first keyword KW1 as the target keyword Wx. In this case, the specific unit 230A determines whether each of the plurality of second keyword KW2 matches the first keyword KW1, and if the determination result regarding any of the plurality of second keyword KW2 is affirmative, the first keyword. The second keyword KW2 that matches KW1 can be specified as the target keyword Wx. Therefore, it is not necessary to refer to the keyword table TBLa to acquire the relevance degree, and the processing load can be reduced.

[2. Second Embodiment]
The service system 1 of the second embodiment is the same as the service system 1 of the first embodiment except for the function of the processing device 21 in the user device 20. FIG. 7 is a functional block diagram showing the functions of the processing device 21 of the second embodiment. The processing device 21 of the second embodiment is different from the processing device 21 of the first embodiment in that the second keyword generation unit 220B is provided in place of the second keyword generation unit 220A.

As shown in FIG. 7, the second keyword generation unit 220B includes an extraction unit 221, a conversion unit 222, and an analysis unit 223. The image signal Sg is supplied to the analysis unit 223. The analysis unit 223 analyzes the image signal Sg of the moving image and outputs the analysis result to the extraction unit 221.

For example, the analysis unit 223 evaluates each of a plurality of object images included in an arbitrary frame of the image signal Sg using the first evaluation item to the fourth evaluation item, and the total of the evaluation values is used as the analysis result. It is output to the extraction unit 221. In this case, the extraction unit 221 extracts an object image in which the total evaluation value exceeds a predetermined value.

The first evaluation item is the ratio of the area of the object image to the area of one screen, and the larger the ratio of the object image, the higher the evaluation value of the object image. The second evaluation item is the perspective of the object image as seen by the user U, and the closer the object image is to the front, the higher the evaluation value of the object image. The third evaluation item is the brightness of the object image, and the brighter the brightness of the object image, the higher the evaluation value of the object image. The fourth evaluation item is the position of the object image, and the closer the position of the object image is to the center of the screen, the higher the evaluation value of the object image. The first to fourth evaluation items are all elements that attract the attention of the user U in the image on one screen. By evaluating the object image using a plurality of evaluation items, it is possible to extract the object image that the user U is highly interested in.

Next, the operation of the user device 20 in the second embodiment will be described. FIG. 8 is a flowchart showing the operation of the user device 20 according to the second embodiment. The flowchart shown in FIG. 6 is the same as the flowchart of the first embodiment shown in FIG. 6, except that steps S6_1 and S6_1 are executed instead of step S6. The differences will be described below.

In step S6_1, the processing device 21 functions as the analysis unit 223, acquires a plurality of evaluation values obtained by evaluating each of the plurality of object images included in a certain frame for each evaluation item, and calculates the total of these evaluation values. For example, the analysis results of the object images OB1 to OB5 shown in FIG. 5 are as shown in FIG. In this example, the total evaluation value for each of the object images OB1 to OB5 is in the range of "11" to "16".

In step S6_2, the processing device 21 functions as an extraction unit 221 to extract an object image whose total evaluation value exceeds a predetermined value. For example, it is assumed that the predetermined value is "13" and the total evaluation value shown in FIG. 9 is obtained for each object image. In this case, the processing device 21 extracts the object images OB2 and OB5. Since the processes after step S7 are the same as the processes described in the first embodiment with reference to FIG. 6, the description thereof will be omitted.

As described above, according to the second embodiment, the second keyword generation unit 220B is based on the analysis result of the analysis unit 223 that analyzes the image signal Sg and the analysis result of the analysis unit 223, from the image indicated by the image signal Sg. It includes an extraction unit 221 that extracts a plurality of object images, and a conversion unit 222 that converts each of the plurality of object images into the second keyword KW2.

According to this aspect, since a plurality of object images are extracted based on the analysis result of the image signal Sg, the number of object images to be extracted is reduced as compared with the case where the object image is extracted without using the analysis result. Can be done. Therefore, the processing load of the conversion unit 222 can be reduced.

The analysis unit 223 may analyze the image signal Sg over a plurality of frames to generate an analysis result. In this case, the analysis unit 223 may adopt a fifth evaluation item related to the movement of the object image in addition to the first evaluation item to the fourth evaluation item. An example of the fifth evaluation item is the number of frames corresponding to the length of time that the moving object image exists in the screen. According to this evaluation item, the larger the number of frames (the object image exists in the screen). The longer the time is, the higher the evaluation value of the object image is. For example, when the main character of a movie moves, the movie is often shot to follow the movement of the main character. Therefore, when the moving image represented by the image signal Sg is a movie, the evaluation value of the object image representing the hero and the evaluation value of the object image representing the belongings possessed by the hero are increased. As a result, it is possible to increase the possibility that the extraction unit 221 extracts the object image of interest to the user U. On the contrary, the possibility that the extraction unit 221 extracts the object image that the user U does not pay attention to can be reduced.

[3. Third Embodiment]
The service system 1 of the third embodiment is the same as the service system 1 of the first embodiment except for the function of the processing device 21 in the user device 20 and the storage contents of the storage device 22. FIG. 10 is a functional block diagram showing the functions of the processing device 21 of the third embodiment. The processing device 21 of the third embodiment is different from the processing device 21 of the first embodiment in that the specific unit 230B is provided in place of the specific unit 230A.

The storage device 22 of the user device 20 of the third embodiment stores the action history table TBLc. The action history of the user U is stored in the action history table TBLC. The action history includes the Internet search history of the user U, the purchase history of products and services, the activity of the SNS (Social Networking Service), the bookmark of the Web browser, and the like.

The specific unit 230B has a plurality of second keywords generated by the second keyword generation unit 220A based on the degree of association indicating the degree of association between each second keyword KW2 and the first keyword KW1 and the action history of the user U. Specify the target keyword Wx from KW2.

First, the specific unit 230B selects the second keyword KW2 having a degree of association with the first keyword KW1 of a predetermined value or more from the plurality of second keyword KW2 generated by the second keyword generation unit 220A. The selected second keyword KW2 is a candidate for the target keyword Wx. Next, the specifying unit 230B identifies the target keyword Wx from the selected second keyword KW2 with reference to the action history stored in the action history table TBLC. For example, it is assumed that the second keyword KW2 selected based on the degree of relevance is "wine" and "Western food". Further, it is assumed that the purchase history of wine is recorded in the action history table TBLC. In this case, when the specific unit 230B detects that the user U has a purchase history of wine by referring to the action history table TBLC, the specific unit 230B specifies "wine" as the target keyword Wx among "wine" and "Western food". do. As a result, the comment generation unit 240 can generate a comment regarding "wine".

Next, the operation of the user device 20 in the third embodiment will be described. FIG. 11 is a flowchart showing the operation of the user device 20 according to the third embodiment. The flowchart shown in FIG. 6 is the same as the flowchart of the first embodiment shown in FIG. 6, except that steps S8_1 and S8_1 are executed instead of step S8. The differences will be described below.

In step S8_1, the processing device 21 functions as the specific unit 230B, and among the plurality of second keyword KW2 generated in step S7, the second keyword KW2 having a degree of association with the first keyword KW1 is selected by a predetermined value or more. do.

In step S8_2, the processing device 21 functions as the specific unit 230B, and among the second keyword KW2 selected in the process of step S8_1, the second keyword KW2 related to the action history is set as the target keyword Wx based on the action history. Identify.

According to the third embodiment, the specific unit 230B specifies the target keyword Wx from the plurality of second keywords KW2 based on the degree of relevance indicating the degree of relevance and the action history of the user U. According to this aspect, since the target keyword Wx is specified in consideration of the action history of the user U, it is possible to provide a comment that is of high interest to the user U as compared with the case where the action history of the user U is not considered. ..

In the operation of the user apparatus 20 described with reference to FIG. 11, the specific unit 230B is a candidate for the target keyword Wx among the plurality of second keyword KW2 generated by the second keyword generation unit 220A using the degree of relevance. The second keyword KW2 is selected (step S8_1), and then the target keyword Wx is specified based on the action history (step S8_1), but the order may be reversed. That is, the specific unit 230B selects the second keyword KW2 that is a candidate for the target keyword Wx from the plurality of second keyword KW2 generated by the second keyword generation unit 220A based on the action history, and then uses the degree of relevance. The target keyword Wx may be specified. In addition, the specific unit 230B may specify the target keyword Wx from the plurality of second keywords KW2 by simultaneously using the action history and the degree of relevance. For example, the specific unit 230B adds a predetermined value to the relevance degree for the second keyword KW2 related to the action history, compares the relevance degree to which the predetermined value is added between the plurality of second keyword KW2s, and compares the target keyword Wx. May be specified.

[4. Fourth Embodiment]
The service system 1 of the fourth embodiment is the same as the service system 1 of the first embodiment except for the function of the processing device 21 in the user device 20 and the storage contents of the storage device 22. FIG. 12 is a functional block diagram showing the functions of the processing device 21 of the fourth embodiment. The processing device 21 of the fourth embodiment is different from the processing device 21 of the first embodiment in that the specific unit 230C is provided in place of the specific unit 230A.

The storage device 22 of the user device 20 of the fourth embodiment stores the profile data DP and the evaluation table TBLd. The profile data DP indicates the profile of the user U. The profile means the attribute of the user U, and includes items such as age and gender.

In the evaluation table TBLd, the evaluation value for each item of the profile is stored in association with the keyword. The evaluation value is a value indicating the degree of interest of the user U in the keyword. FIG. 13 shows an example of the stored contents of the evaluation table TBLd. For example, for the keyword "car", the evaluation value of the gender "male" is "7", while the evaluation value of the gender "female" is "4". This shows that men are more interested in cars than women.

The specific unit 230C has a plurality of second keyword KW2 generated by the second keyword generation unit 220A based on the degree of association indicating the degree of association between each second keyword KW2 and the first keyword KW1 and the profile of the user U. Specify the target keyword Wx from among them.

First, the specific unit 230C selects the second keyword KW2 having a degree of association with the first keyword KW1 of a predetermined value or more from the plurality of second keyword KW2 generated by the second keyword generation unit 220A. Next, the specifying unit 230C uses the profile data DP and the evaluation table TBLd to specify the target keyword Wx from the selected second keyword KW2. Specifically, for each of the selected second keyword KW2, a total evaluation value is calculated by summing the evaluation values corresponding to each of a plurality of items in the profile of the user U, and the second keyword KW2 having the highest total evaluation value is calculated. Is specified as the target keyword Wx.

Next, the operation of the user device 20 in the fourth embodiment will be described. FIG. 14 is a flowchart showing the operation of the user device 20 according to the fourth embodiment. The flowchart shown in FIG. 12 is the same as the flowchart of the third embodiment shown in FIG. 12, except that step S8_3 is executed instead of step S8_2. The differences will be described below.

In step S8_3, the processing device 21 functions as the specific unit 230C, and among the second keyword KW2 selected in the process of step S8_1 based on the profile, the second keyword KW2 having the highest total evaluation value of the profile of the user U Is specified as the target keyword Wx.

According to the fourth embodiment, the specifying unit 230C specifies the target keyword Wx from the plurality of second keywords KW2 based on the degree of relevance indicating the degree of relevance and the profile of the user U. According to this aspect, since the target keyword Wx is specified in consideration of the profile of the user U, it is possible to provide a comment that is of high interest to the user U as compared with the case where the profile of the user U is not considered.

In the operation of the user apparatus 20 described with reference to FIG. 14, the specific unit 230B is a candidate for the target keyword Wx among the plurality of second keyword KW2 generated by the second keyword generation unit 220A using the degree of relevance. The second keyword KW2 is selected (step S8_1), and then the target keyword Wx is specified based on the profile (step S8_3), but the order may be reversed. That is, the specific unit 230C selects the second keyword KW2 that is a candidate for the target keyword Wx from the plurality of second keyword KW2 generated by the second keyword generation unit 220A based on the profile, and then uses the degree of relevance. The target keyword Wx may be specified. In addition, the specifying unit 230C may specify the target keyword Wx from a plurality of second keywords KW2 by using the profile and the degree of relevance at the same time. For example, the specific unit 230C may add the total evaluation value based on the profile to the degree of relevance and compare the addition results of the plurality of second keywords KW2 to specify the target keyword Wx.

[5. Modification example]
The present invention is not limited to the embodiments exemplified above. Specific modes of modification are illustrated below. Two or more embodiments arbitrarily selected from the following examples may be merged.

(1) In each of the above-described embodiments, the frame from which the extraction unit 221 extracts the object image from the image of the image signal Sg may be the following frame.
First, the extraction unit 221 may extract an object image in a frame having a high audience rating. In this case, the extraction unit 221 may acquire the audience rating from an external device in real time. Specifically, the extraction unit 221 extracts the object image at a frame in which the acquired audience rating exceeds a predetermined audience rating. It is presumed that the frame with a high audience rating is of higher interest to the user U than the other frames. Therefore, since the object image is extracted from the image of the frame in which the user U is highly interested, a comment useful to the user U can be generated.
Secondly, the extraction unit 221 may extract the object image in the frame in which the user U cheers based on the voice signal Sa of the user U.
Third, the extraction unit 221 may extract an object image in a frame that is the subject of the program based on the program information. For example, the extraction unit 221 may analyze the image signal Sg by using the analysis unit 223 described in the second embodiment to specify a frame that is the subject of the program. In this case, the analysis unit 223 may acquire program information from an external device via the network NW.
Further, in each of the above-described embodiments, the image signal Sg has been described as a signal indicating a moving image, but the image signal Sg may be a signal indicating a still image.

(2) In each of the above-described embodiments, the degree of association indicating the degree of association between the second keyword KW2 and the first keyword KW1 is stored in the keyword table TBLa, but is not limited thereto.
For example, the specific units 230A, 230B, and 230C acquire the degree of relevance according to the number of nodes specified from the keyword table TBLa (an example of keyword data) having a tree structure in which a plurality of words are layered according to meaning. May be good. Specifically, the first keyword generation unit 210 generates a word included in the keyword table TBLa as the first keyword KW1. Further, the second keyword generation units 220A and 220B generate the words included in the keyword table TBLa as the second keyword KW2. The specific units 230A, 230B, and 230C acquire the number of nodes in the route from the first keyword KW1 to the second keyword KW2 as the degree of relevance in the tree structure of the keyword table TBLa.
More specifically, it is assumed that the data structure of the keyword table TBLa is the tree structure shown in FIG. For example, if the first keyword KW1 is "liquor" and the second keyword KW2 is "french fries", the route from "liquor" to "french fries" is node "liquor"-> node "drink"-> node. "Food and drink"-> node "food"-> node "Western food"-> node "french fries". Therefore, the number of nodes in the route from the first keyword KW1 "liquor" to the second keyword KW2 "french fries" is "5". When the first keyword KW1 is "food and drink" and the second keyword KW2 is "french fries", the route from "food and drink" to "french fries" is node "food and drink" → node "food". → Node "Western food" → Node "French fries". Therefore, the number of nodes in the route from the first keyword KW1 “food and drink” to the second keyword KW2 “fried potato” is “3”. The smaller the number of nodes in the route connecting the first keyword KW1 and the second keyword KW2, the higher the degree of relevance. Therefore, in the above example, the first keyword KW1 "food and drink" and the second keyword KW2 "french fries" are used. The degree of relevance is higher than the degree of relevance between the first keyword KW1 "liquor" and the second keyword KW2 "french fries".
By specifying the degree of relevance according to the number of nodes, the storage capacity of the keyword table TBLa required in the user apparatus 20 can be reduced.

(3) In each of the above-described embodiments, the extraction unit 221 extracts the object image without considering the action history of the user U, but extracts the object image from the image indicated by the image signal Sg based on the action history. May be good. In this case, the extraction unit 221 refers to the action history table TBLC described in the third embodiment, specifies, for example, the user U's favorite color from the purchase history of the product, and extracts the object image of the specified color. You may. According to this modification, the object image can be narrowed down, so that the processing load of the conversion unit 222 can be reduced.

(4) The above-described embodiments can be combined as appropriate. For example, the second keyword generation unit 220B of the second embodiment may be used in place of the second keyword generation unit 220A of the third embodiment and the fourth embodiment.

(5) The block diagram used in the description of each of the above-described embodiments shows a block of functional units. These functional blocks (components) are realized by any combination of hardware and / or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically and / or logically coupled device, or directly and / or indirectly by two or more physically and / or logically separated devices. (For example, wired and / or wireless) may be connected and realized by these plurality of devices. For example, the function of the conversion unit 222 may be provided from a server device connected via the network NW. Similarly, the keyword table TBLa may be provided in the server device.
Further, the word "device" used in the description of each of the above-described embodiments may be read as another term such as a circuit, a device, or a unit.

(6) The order of the processing procedures, sequences, flowcharts, etc. in each of the above-described embodiments may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order and are not limited to the particular order presented.

(7) In each of the above-described embodiments, the input / output information and the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input / output may be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

(8) In each of the above-described embodiments, the determination may be made by a value represented by one bit (0 or 1) or by a truth value (Boolean: true or false). However, it may be performed by comparison of numerical values (for example, comparison with a predetermined value).

(9) In each of the above-described embodiments, the storage device 22 is a recording medium that can be read by the processing device 21, and examples thereof include a ROM and a RAM. Applications Discs, Blu-ray® discs), smart cards, flash memory devices (eg cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable discs, hard disks, floppy (registration). Trademarks) Disks, magnetic strips, databases, servers and other suitable storage media. Further, the program may be transmitted from the network NW. Further, the program may be transmitted from the communication network via a telecommunication line.

(10) Each of the above-described embodiments is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark). ), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered) It may be applied to systems that utilize other suitable systems and / or next-generation systems that are extended based on them.

(11) In each of the above embodiments, the information, signals, and the like described may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.
The terms described herein and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

(12) Each of the functions exemplified in FIGS. 4, 7, 10, and 12 is realized by any combination of hardware and software. Further, each function may be realized by a single device, or may be realized by two or more devices configured as separate bodies from each other.

(13) The program exemplified in each of the above-described embodiments is called an instruction, an instruction set, a code, or a code segment regardless of whether it is called a software, firmware, middleware, microcode or hardware description language, or by another name. , Program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures or functions, etc. should be broadly interpreted.
Further, software, instructions, and the like may be transmitted and received via a transmission medium. For example, the software may use wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to website, server, or other. When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

(14) In each of the above embodiments, the terms "system" and "network" are used interchangeably.

(15) In each of the above-described embodiments, the information, parameters, etc. may be represented by absolute values, relative values from predetermined values, or other corresponding information. May be good.

(16) In each of the above-described embodiments, the user device 20 may be a mobile station. Mobile stations can be used by those skilled in the art as subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

(17) In each of the embodiments described above, the term "connected", or any variation thereof, means any direct or indirect connection or connection between two or more elements. It can include the presence of one or more intermediate elements between two elements that are "connected" to each other. The connection between the elements may be physical, logical, or a combination thereof. As used herein, the two elements are by using one or more wires, cables and / or printed electrical connections, and, as some non-limiting and non-comprehensive examples, radio frequencies. It can be considered to be "connected" to each other by using electromagnetic energies such as electromagnetic energies having wavelengths in the region, microwave region and light (both visible and invisible) regions.

(18) In each of the embodiments described above, the statement "based on" does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

(19) Any reference to elements using designations such as "first", "second" as used herein does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

(20) As long as "inclusion," "comprising," and variations thereof in each of the embodiments described above are used herein or within the scope of the claims, these terms are used. As with the term "prepared", it is intended to be inclusive. Moreover, the term "or" as used herein or in the claims is intended to be non-exclusive.

(21) In the whole of the present application, if articles are added by translation, for example a, an and the in English, unless the context clearly indicates that these articles are not. Includes multiple.

(22) It will be apparent to those skilled in the art that the invention is not limited to the embodiments described herein. The present invention can be implemented as modifications and modifications without departing from the spirit and scope of the present invention, which is determined based on the description of the scope of claims. Therefore, the description herein is for purposes of illustration and has no limiting implications for the present invention. In addition, a plurality of embodiments selected from the embodiments exemplified herein may be combined.

1 ... Service system, 10 ... Video distribution server, 11 ... Processing device, 20 ... User device, 21 ... Processing device, 22 ... Storage device, 210 ... First keyword generation unit, 220A, 220B ... Second keyword generation unit, 220B ... second keyword generation unit, 221 ... extraction unit, 222 ... conversion unit, 223 ... analysis unit, 230A, 230B, 230C ... specific unit, 240 ... comment generation unit, KW1 ... first keyword, KW2 ... second keyword, TBLa ... Keyword table, TBLb ... Comment table, TBLc ... Action history table, Wx ... Target keywords.

Claims (5)

A first keyword generator that generates the first keyword based on the user's voice,
A second keyword generation unit that generates a plurality of second keywords corresponding to a plurality of object images extracted from an image indicated by an image signal on a one-to-one basis.
A specific unit that specifies a target keyword to be commented from the plurality of second keywords based on the degree of relevance between each of the plurality of second keywords and the first keyword.
A comment generator that generates comments related to the target keyword,
Information processing device equipped with. Of the plurality of second keywords, the relationship when the other second keyword matches the first keyword as compared with the degree of relevance when one second keyword does not match the first keyword. The degree of sex is high,
The information processing apparatus according to claim 1, wherein the specific unit specifies a second keyword that matches the first keyword among the plurality of second keywords as the target keyword. The second keyword generation unit is
An analysis unit that analyzes the image signal and
An extraction unit that extracts the plurality of object images from the image indicated by the image signal based on the analysis result of the analysis unit, and an extraction unit.
A conversion unit for converting each of the plurality of object images into the corresponding second keyword among the plurality of second keywords is provided.
The information processing apparatus according to claim 1 or 2. From claim 1, the specific unit specifies the target keyword from the plurality of second keywords based on the degree of the relationship between each second keyword and the first keyword and the behavior history of the user. The information processing apparatus according to any one of up to 3. Claims 1 to 3 specify the target keyword from the plurality of second keywords based on the degree of the relationship between each second keyword and the first keyword and the profile of the user. The information processing apparatus according to any one of the above items.

Images