JP7486257B1 - Information processing device, information processing method, and information processing program - Google Patents

Abstract

The information processing device includes a media information analysis unit that extracts concepts of multiple media information, a concept relationship calculation unit that calculates a conceptual relationship between the concepts of the multiple media information, and a search recommendation unit that selects multiple media information each having a concept that has a predetermined conceptual relationship to the concept of media information selected by a user.

Description

The present disclosure relates to an information processing device, an information processing method, and an information processing program capable of searching for media information.

Technology is known that displays multiple objects representing multiple different media information (still images, moving images, audio, text, etc.) reflecting a user's interests and preferences.

JP 2002-175317 A JP 2006-018843 A JP 2007-517317 A JP 2009-003637 A JP 2010-186214 A

When a search query is given, the multimedia search system of each patent document outputs, as search results, multimedia that is directly related to the search query. For example, if the query is a keyword, it outputs multimedia that includes the keyword, and if the query is an image, it outputs multimedia that includes images with image characteristics similar to the image or information about the subject. Furthermore, these conventional multimedia search systems require the user to input a search query first, and aim to produce search results that match the explicitly described search needs.

In such a multimedia search system, it is impossible to provide a user with information other than the information the user expects in response to a search query. In other words, the system can only provide information within the scope of the user's expectations. This situation is thought to involve the following problems:

- The system assumes that the user is aware of their search needs and can spontaneously describe them in language. (In image and multimedia searches, it only shows things that match the subject of the search. In other words, they are the same in that they aim to match the subject.)- As a result, the system cannot provide a breadth of information that goes beyond the explicit search needs, and therefore cannot satisfy the sensory search needs that users are unconscious of and cannot put into words.

In light of the above circumstances, the purpose of this disclosure is to allow users to discover new connections and expand their exploration by including in search results multimedia that the user may not have anticipated but that is guaranteed to be relevant to the search target.

An information processing device according to an embodiment of the present disclosure includes:
a media information analysis unit for extracting concepts of a plurality of pieces of media information;
a concept relationship calculation unit for calculating a concept relationship between concepts of the plurality of pieces of media information;
and a search time recommendation unit that selects a plurality of media information each having a concept that has a predetermined conceptual relationship with the concept of the media information selected by the user.

According to the present disclosure, by including in the search results multimedia that the user may not have anticipated but that is guaranteed to be relevant to the search target, it is possible to allow the user to discover new connections and expand their exploration.

Note that the effects described here are not necessarily limited to those described herein and may be any of the effects described in this disclosure.

1 illustrates an information processing system according to an embodiment of the present disclosure. 1 is an example of a top screen of a GUI of an electronic device. 13 is an example of a screen after a transition of the GUI of the electronic device (operation toward a first functional object). 13 is another example of a screen after a GUI transition of an electronic device (operation from bottom to top). 13 is another example of a screen after a GUI transition of an electronic device (operation from top to bottom). 13 is another example of a screen after a transition of the GUI of the electronic device (detailed display). 13 is another example of a screen after a transition of the GUI of the electronic device (second functional object). This shows the functional configuration of the information system. 3 shows an operation flow of the information processing device. A media information database is shown. 2 shows the configuration of a media information analysis unit. A conceptual database is shown. 1 shows a concept relationship database. 1 shows an analysis information database. 3 shows a configuration of a concept relationship calculation unit. 2 shows the configuration of an account creation unit. Indicates the account database. 2 shows the configuration of an account usage history analysis unit. The configuration of the startup recommendation unit is shown below. The configuration of the search recommendation unit is shown below. 2 shows an example of a top screen. 13 is an example of a screen after a transition of the GUI of the electronic device (autoplay). 13 is a modified example of the top screen. 13 is a modified example of the top screen. 13 is a modified example of the top screen. 13 is another example of a top screen of a GUI of an electronic device. 13 shows a modified example of the configuration of the startup recommendation unit. 11 is a diagram for explaining an example of the operation of a search recommendation unit; FIG. 11 is a diagram for explaining an example of the operation of a search recommendation unit; FIG. 10A and 10B are schematic diagrams illustrating a comparison between a conventional search and a search according to the present embodiment. 13 shows a modified example of the configuration of the search time recommendation unit.

Below, an embodiment of the present disclosure is described with reference to the drawings.

1. Information Processing System

Figure 1 shows an information processing system according to one embodiment of the present disclosure.

The information processing system 1 has an information processing device 10, a storage device 20, and a web server 30 that are communicatively connected via a network or the like. The information processing device 10 functions as a back-end computing server and executes arithmetic processing. The storage device 20 is a large-capacity non-volatile storage device that functions as a server-side database server. The information processing device 10 and the storage device 20 may be independently connected via a network or the like (as shown), or may be constructed as an integrated information processing device (not shown). The web server 30 is a front-end web server that provides data generated by the information processing device 10 through arithmetic processing and stored in the storage device 20 in response to a request from the electronic device 40, which is a client. The information processing device 10 is configured to select multiple media information for the electronic device 40, and to supply display data for displaying the selected multiple media information to the electronic device 40 via the web server 30.

The electronic device 40 is a device having a display device 41, such as a smartphone, a tablet computer, a personal computer, a head-mounted display, or VR glasses, and is a terminal device used by an end user. The control circuit 42 of the electronic device 40 changes the GUI displayed on the display device 41 by executing an application program in cooperation with the web server 30. The application program may be executed via a web browser, or may be installed and executed in the electronic device 40. By executing the application program, the electronic device 40 recommends and displays a plurality of media information to the user on the display device 41 at the time of startup, and further recommends and displays a plurality of different media information related to the media information selected by the user on the display device 41 from among the plurality of media information at the time of search. This aims to provide the user with a user experience such as the discovery of new connections and the expansion of exploration. The display device 41 may be a touch panel, a display not including a touch panel, a head-mounted display, a display unit built into the VR glasses, or the like.

In this embodiment, the media information is composed of at least one of a number of different modalities (formats), such as still images, moving images, music, text, location information, and tag information. The type of modality is not limited to this example and may be any format. The media information is linked to web pages on the network, such as videos, magazines, auctions, EC, art exhibitions, encyclopedias, SNS, and advertisements. The media information is not limited to data on the network and may be data stored in the storage device 20.

2. Electronic device GUI

Below, we will explain the GUI and changes in the GUI displayed on the display device 41 when the control circuit 42 of the electronic device 40 executes an application program.

Figure 2 is an example of the top screen of a GUI of an electronic device.

In this embodiment, the electronic device 40 is a smartphone, and the display device 41 is a vertically long rectangle. In this embodiment, the terms rectangle, square, quadrilateral, polygon, etc., include approximate rectangles, approximate squares, approximate quadrilaterals, approximate polygons, etc. that have rounded corners (R) but visually give the impression of the figure in question.

The control circuit 42 of the electronic device 40 sets a first display area 401 and a second display area 402 on the display device 41.

The first display area 401 is, for example, a square with each side having a length of 3L or more. When the display device 41 is, for example, a touch panel of a smartphone, the length of one side of the first display area 401 may be equal to or approximately equal to the short side of the display device 41.

The second display area 402 is a rectangle having a first side 403 and a second side 404. The first side 403 is adjacent to one side of the first display area 401 and has a length of, for example, 3L or more. The second side 404 is perpendicular to the first side 403 and has a length of, for example, L or more. The length of the second side 404 of the second display area 402 may be, for example, an integer multiple of L.

The control circuit 42 of the electronic device 40 displays on the display device 41 a top screen 400 including a plurality of first media objects 410, a plurality of second media objects 420, a first functional object 430, and an account object 440.

The first functional object 430 is, for example, a square with each side having a length of L. The first functional object 430 is positioned in the center of the first display area 401 (i.e., a square with each side having a length of 3L or more).

Each of the multiple (four in this example) first media objects 410 is, for example, a rectangle with a short side length of L and a long side length of 2L or more (for example, 2.02L). The four first media objects 410 are displayed around the first functional object 430 within the first display area 401. More specifically, the first functional object 430 and the four first media objects 410 form a square with each side length of 3L or more. That is, the first functional object 430 and the four first media objects 410 are arranged without gaps in the square first display area 401 with each side length of 3L or more.

A plurality of second media objects 420 are displayed in the second display area 402. The second media objects 420 are, for example, squares with each side having a length of L. The control circuit 42 of the electronic device 40 displays three second media objects 420 in a first direction, which is the length direction of the first side 403, and one or more second media objects 420 in a second direction, which is the length direction of the second side 404, in a matrix shape adjacent to the first display area 401 and such that all the second media objects 420 are adjacent to each other.

Each of the first media objects 410 indicates different media information. Each of the first media objects 410 includes, for example, a square image 411 with each side having a length of L or less, and non-image information 412 that is adjacent to the image 411 and is included in a square with each side having a length of L. For example, the first media object 410 includes an image 411 and text information 412 that is included in a square with each side having a length of L. The non-image information 412 (for example, text information) is information associated with the image 411, such as the name, description, and link information of the image 411.

Each of the second media objects 420 represents media information that is different from the media information of the first media object 410. For example, each of the second media objects 420 includes a square image 421, each of which has a side length less than or equal to L.

"Different media information related to each other" refers to, for example, media information related to a chair designed by a designer, a table designed by that designer or a designer with whom he or she has a connection, a building designed, a museum in which the designer's work is exhibited, places the designer has visited, still images or videos that the designer has photographed, works that have influenced the designer, or works that have influenced the designer, and is media information related to each other directly or indirectly. The relationship does not have to be a direct, unique relationship that everyone can understand. How to select "different media information related to each other" to be displayed as the first media object 410 and the second media object 420 will be explained later.

The first functional object 430 functions as a search window. The first functional object 430 has a functional button 431. The specific functions of the first functional object 430 and the functional button 431 will be described in detail later.

The account object 440 may be displayed outside the first display area 401 and the second display area 402.

Figure 26 is another example of a top screen of a GUI of an electronic device.

Alternatively, the first media object 410 may include one or more active action buttons 470. The active action buttons 470 are buttons that allow the user to actively save (clip), favorite, share, etc.

Next, we will explain how the GUI changes based on user operations.

Figure 3 shows an example of a screen after a transition in the GUI of an electronic device (operation toward the first functional object).

As shown in (A), the control circuit 42 of the electronic device 40 detects that the user performs an operation (a first operation for the user to display new information) on the display device 41 in which one first media object 410 or one second media object 420 is directed toward the first functional object 430. Then, as shown in (B), the control circuit 42 of the electronic device 40 switches the display so that, instead of the multiple first media objects 410 and the second media objects 420 that are being displayed, multiple new first media objects 410 and second media objects 420 each indicating different media information related to the media information indicated by the one first media object 410 or the second media object 420 that has been directed toward are displayed. In addition, the "operation of the first media object 410 or the second media object 420 moving toward the first functional object 430" may be a drag operation using an operator (a finger, a stylus pen, a mouse, etc.), or if the electronic device 40 is a head-mounted display or VR glasses, an operation using a hand-held controller, a part of the human body, or movement of the gaze, etc.

The arrangement of the multiple new first media objects 410 and second media objects 420 is the same as that of the top screen 400. That is, the multiple new first media objects 410 are displayed around the first functional object 430 in the first display area 401. The multiple new second media objects 420 are displayed in the second display area 402. The method of selecting the new first media objects 410 and second media objects 420 will be described in detail later.

In the new display screen, a back button 432 is displayed within the first functional object 430. When the user operates the back button 432 within the first functional object 430, the first media object 410 and the second media object 420 that were originally displayed are displayed in their original arrangement, as shown in (A). In this manner, the media information displayed in the multiple first media objects 410 and the multiple second media objects 420 is prioritized according to the user's preferences and is used when selecting the media information to be posted.

In this way, the GUI of the top screen 400 is composed of multiple figures. The first functional object 430 used for searching is arranged in the center of the screen, and the first media object 410 that displays media information is arranged so as to surround the first functional object 430. In the first media object 410, a combination of multiple types of media information, such as a still image and text, can be displayed, and one or more pieces of media information are displayed. A search is performed by operating the first media object 410 or the second media object 420 toward the first functional object 430. When transitioning from the top screen 400, the first functional object 430 does not change from the specified position, but the information in the first media object 410 and the second media object 420 is changed.

As a result, according to this embodiment, a search can be performed by moving the surrounding first media object 410 or second media object 420 toward the central first functional object 430. Since a search is performed simply by moving the first media object 410 or second media object 420 toward the first functional object 430 (e.g., drag and drop), no operation process is required after moving the first media object 410 or second media object 420 (e.g., drag and drop), and the user has high intuitive operability. In addition, since the first functional object 430 to be operated is equidistant from all first media objects 410 and in the center, there is no variation in the moving distance depending on the first media object 410 to be moved, and long-distance movement is not required, resulting in high intuitive operability.

As a modified example, the first media object 410, the second media object 420, and the first functional object 430 may be an ellipse, a circle, a polygon (including those with rounded corners), etc. The colors of the first media object 410, the second media object 420, and the first functional object 430 may be composed of one or more colors. The color of the first media object 410 may be changed according to the information (posted information) posted by the figure. For example, the color may be changed according to the category of media information. The boundaries of the first media object 410, the second media object 420, and the first functional object 430 do not need to be explicit. When the first media object 410 or the second media object 420 is held to move it, the user may be notified that the first media object 410 or the second media object 420 has been held by playing a sound effect, changing the shape of the figure, animation such as a color change or shaking of the figure, vibration of the electronic device 20, etc.

Figure 4 is another example of the screen after a GUI transition of an electronic device (operation from bottom to top).

As shown in (A), the control circuit 42 of the electronic device 40 detects that the user operates (the second action for the user to display new information) on the display device 41 in the direction from the second display area 402 to the first display area 401 (the direction of the arrow from bottom to top). Then, as shown in (B), the control circuit 42 of the electronic device 40 displays N new first media objects 410 in the first display area 401, each of which indicates media information indicated by the N second media objects 420 close to the first display area 401, in place of N first media objects 410 (N is an integer between 1 and 4) (three in this example) of the four first media objects 410 displayed in the first display area 401. The control circuit 42 of the electronic device 40 switches the display so that, instead of the N (three in this example) second media objects 420 displayed in the second display area 402, N new second media objects 420 each indicating different media information related to the N new first media objects 410 are displayed in the second display area 402. The control circuit 42 of the electronic device 40 caches the media information in advance, and when the cached media information runs out, it displays an update mark (e.g., a substantially circular arrow), reads in new media information, and displays it as the new second media object 420.

Figure 5 shows another example of the screen after a GUI transition of an electronic device (operation going from top to bottom).

As shown in (A), the control circuit 42 of the electronic device 40 detects that the user is operating the display device 41 in the direction from the first display area 401 to the second display area 402 (the direction of the arrow from top to bottom). Then, as shown in (G), the control circuit 42 of the electronic device 40 displays N new second media objects 420 in the second display area 402, each of which indicates media information indicated by the N first media objects 410 closest to the second display area 402, in place of N (N is an integer between 1 and 4) (four in this example) second media objects 420 out of the three or more second media objects 420 displayed in the second display area 402. The control circuit 42 of the electronic device 40 switches the display so that, instead of the N (four in this example) first media objects 410 displayed in the first display area 401, N new first media objects 410 each indicating different media information related to the N new second media objects 420 are displayed in the first display area 401. The control circuit 42 of the electronic device 40 caches the media information in advance, and when the cached media information runs out, displays an update mark, reads new media information, and displays it as the new first media object 410. The "operation from bottom to top" and the "operation from top to bottom" may be a drag-and-drop operation using an operator (a finger, a stylus pen, a mouse, etc.), or an operation using a handheld controller, a part of the human body, or a line of sight when the electronic device 40 is a head-mounted display or VR glasses.

Figure 6 is another example of a screen after the GUI of an electronic device has changed (detailed view).

The control circuit 42 of the electronic device 40 detects that one first media object 410 is operated on the display device 41 in a manner other than a movement operation (e.g., a long press, a tap, etc.) (a third operation for displaying a details page of the media information indicated by one first media object or one second media object). Then, the control circuit 42 of the electronic device 40 displays a details page 452 that shows in detail the media information indicated by the operated one first media object 410 or one second media object 420, and is larger than one first media object 410 and one second media object 420. As shown in the figure, the layout of the details page 452 may be changed depending on the modality of the media information (text, video, still image, etc.).

A new function may be added to the first functional object 430. The functions include searching for media information by moving the surrounding first media object 410 or second media object 420 toward the first functional object 430, searching for media information based on an image by capturing an image, and searching for media information based on text by inputting text. The display method of the searched media information can be changed. The media information may be displayed on the first media object 410 surrounding the first functional object 430 (same as the normal top screen), displayed in the same graphic configuration, displayed as a list of media information combining one or more media, or displayed as a list of media information limited to a specific media. The details of the media information can be displayed by performing a specified operation (click, double-click, tap, etc.) on the first media object 410 or the second media object 420. One or more pieces of media information are displayed on the details page 452.

In this way, the display size of the media information can be changed from the first media object 410 or the second media object 420 to the details page 452. The enlarged first media object 410 can display images and text using more pixels than the standard size first media object 410 or the second media object 420. Methods for changing the information display include increasing the text size, increasing the amount of information displayed, and increasing the types of media information. Active operations are possible for the first media object 410 or the second media object 420. Active operations include storing (clipping) favorite information, posting to other SNS, sharing media information, writing comments, etc. An interface for executing each of these active operation functions is an icon drawn on or near the first media object 410 or the second media object 420 in an individual or aggregated form. When aggregated, operating the aggregated icon opens a menu listing active operations.

From the details page 452, active operations similar to those of the first media object 410 are possible. The user can write a comment on each piece of media information. The user can view the comment together with the media information from which the comment was written. The user can add a reaction to the comment and evaluate the comment by clicking or tapping a designated icon. Information that the user can view regarding the written comment includes information about the comment writer, the content of the comment, and the evaluation of the comment. When the user selects the comment writer from the viewable information, the screen may transition to the comment writer's My Page. The comment writer's My Page may display the comment writer's free entry field (self-introduction, etc.), comment writing history, the number of users being followed (following number), and the number of users being followed (follower number). The user may be able to follow other users. Comments, article posts, advertisements, etc. written by users followed by the user may be preferentially displayed on the user's screen.

Figure 7 is another example of a screen after a GUI transition of an electronic device (second functional object).

When the first functional object 430 is operated (the fourth operation for the user to perform a text search for information) on the display device 41, the control circuit 42 of the electronic device 40 displays the second functional object 451. The second functional object 451 functions as a text search field. The control circuit 42 of the electronic device 40 displays one or more third media objects 450 each indicating media information related to the text entered in the second functional object 451. The "media information related to the text" may be selected by the method of selecting the "different media information related to each other" displayed as the first media object 410 and the second media object 420 described above, or may be selected by another method.

The second functional object 451 has a text search field and a function similar to the search window of the first functional object 430. When an operation is performed in which one third media object 450 is directed toward the second functional object 451, the control circuit 42 of the electronic device 40 displays, in the layout of the top screen 400, a plurality of first media objects 410 each indicating different media information related to the media information indicated by the third media object 450 to which the operation was directed.

3. Functional configuration of the information processing system

Figure 8 shows the functional configuration of the information system.

The information processing device 10 operates as a collection unit 110, a media information analysis unit 120, a conceptual relationship calculation unit 130, an account creation unit 140, an account usage history analysis unit 150, a startup recommendation unit 160, and a search recommendation unit 170 by the processor of the control circuit 100 loading various information processing programs recorded in the auxiliary storage device into the main storage device and executing them.

The storage device 20 stores a media information database 210, a concept database 220, a concept relationship database 230, an analysis information database 240, and an account database 250.

4. Operation of information processing device

Figure 9 shows the operation flow of the information processing device.

The collection unit 110 collects media information (step S1). The media information is composed of at least one of a number of different modalities (formats), such as still images, moving images, music, text, location information, and tag information. The media information is linked to any data that exists in the real world and the virtual world. For example, data stored on the web (also called the Internet or network), data stored in a personal device, and data measured using a specified sensor in the real world or virtual world are subject to the media information. The collection unit 110 stores the collected media information in the media information database 210.

Figure 10 shows the media information database.

A plurality of media data 210A are registered in the media information database 210. Each media data 210A is registered with a media ID 211, a title 212, text 213, an image identifier 214, a sender account ID 215, and an update date 216, which are associated with each other.

The media ID 211 may be a serial number that uniquely identifies one piece of media information within the media information database 210. The title 212 may be defined by meta information included in the collected media information. The text 213 is included in the collected media information. The image identifier 214 may be a URL (link address) uniquely assigned to the media information, or may be actual data. Each piece of media data 210A may include either the text 213 or the image identifier 214, or may include both. If the media information is music, it may not include both the text 213 and the image identifier 214, but may include a music identifier (not shown) indicating a URL or actual data.

The sender account ID 215 is the account ID of the end user who sent the media information. In addition to the sender account ID 215, a sender classification may be further registered, which is classified based on whether the sender is a member of the service, whether the sender is a Know Your Customer (KYC), etc. The update date 216 is the update date of the media information. The update date 216 may be classified based on whether the update date is within a specified time from the collection date.

The types and number of types of the multiple media data 210A may be classified according to the type and number of media by which the media data 210A is configured and provided. If an individual address is set for each media data 210A (e.g., SNS), this address information may also be collected.

Figure 11 shows the configuration of the media information analysis unit.

The media information analysis unit 120 calculates media features, which are the features of each piece of media information present on the network, using a model learned by machine learning, for example (step S2). Specifically, the media information includes multiple pieces of sub-media information M1, M2 (e.g., text M1 and image M2). The media information analysis unit 120 calculates media features 245, 246 for each of the multiple pieces of sub-media information M1, M2 (step S21). That is, the media information has multiple media features 245, 246 for each of the multiple pieces of sub-media information M1, M2.

The media information analysis unit 120 extracts the concept of each piece of media information (step S3). Specifically, the media information analysis unit 120 refers to the concept database 220, extracts the concept of each piece of media information, and breaks down the extracted concept into a dominant concept 243 and a sub-concept 244 (step S22). For example, the media information analysis unit 120 may classify a concept held by an object with a high occurrence rate in one piece of media information as the dominant concept 243, and a concept held by an object with a relatively low occurrence rate as the sub-concept 244. That is, the concept of the media information includes the dominant concept 243 and the sub-concept 244. The set of the dominant concept 243 and the sub-concept 244 held by a certain piece of media information is regarded as the concept of that media information.

Figure 12 shows a conceptual database.

A plurality of concept data 220A are registered in the concept database 220. Each concept data 220A is registered with a serial ID 221, a concept ID 222, and a concept name 223 in association with each other. The concept data 220A registered in the concept database 220 does not distinguish between dominant concepts 243 and sub-concepts 244. For example, the concept data 220A does not have a flag or the like that distinguishes between the dominant concept 243 and the sub-concept 244.

Figure 13 shows the concept relationship database.

A plurality of concept relationship data 230A are registered in the concept relationship database 230. A "concept" is a concept expressed in media information such as text or an image. Each concept relationship data 230A is registered in association with a serial ID 231, a concept relationship ID 232, a concept from 233, a concept to 234, and relationship strength 235. Relationship strength 235 refers to the strength of a directional connection (strength of a connection).

For example, the strength of relationship 235 from the concept "today" with concept ID "0" in FIG. 12 to the concept "event" with concept ID "1" is "0.61". In addition, just because the strength of relationship 235 from the concept "today" to the concept "event" is registered, the strength of relationship 235 in the reverse direction from the concept "event" to the concept "today" does not necessarily have to be registered. In addition, the strength of relationship 235 from the concept "today" to the concept "event" does not necessarily match the strength of relationship 235 in the reverse direction from the concept "event" to the concept "today".

The media information analysis unit 120 refers to the concept database 220 and the concept relationship database 230, and determines a concept having a predetermined concept relationship strength 235 to the extracted dominant concept 243 as an additional sub-concept 244 of the media information (step S23). For example, when the extracted dominant concept 243 is the concept from 233, the media information analysis unit 120 determines a concept to 234 whose relationship strength 235 from this concept from 233 is equal to or greater than a predetermined value as an additional sub-concept 244. The media information analysis unit 120 registers the media features 245, 246, the dominant concept 243, and the sub-concept 244 in the analysis information database 240.

Figure 14 shows the analysis information database.

A plurality of analysis information data 240A are registered in the analysis information database 240. Each analysis information data 240A is registered in association with a serial ID 241, a media ID 242, a dominant concept set 243, a sub-concept set 244, a text feature 245, an image feature 246, and a feature 247 of other media (e.g., music). The media ID 242 is the same as the media ID 211 in the media information database 210. The dominant concept set 243 includes the concept IDs 222 (FIG. 12) of all dominant concepts extracted for the media information of the media ID 242. The sub-concept set 244 includes the concept IDs 222 (FIG. 12) of all sub-concepts extracted for the media information of the media ID 242. The text feature 245, the image feature 246, and the feature 247 of other media include the feature of all sub-media included in the media information. The text feature 245, the image feature 246, and the feature of other media (for example, music) 247 are collectively referred to as “media feature.” One form of the feature format is a vector, but any format is acceptable.

Figure 15 shows the configuration of the conceptual relationship calculation unit.

The concept database 220 and the concept relationship database 230 are created in advance by the concept relationship calculation unit 130 and updated as necessary. That is, the concept relationship calculation unit 130 calculates and updates the conceptual relationships between concepts of multiple media information present on the network.

Specifically, the concept relationship calculation unit 130 reads out the media information database 210 and the analysis information database 240. The concept relationship calculation unit 130 extracts the sender account ID 215 and update date 216 of the media data 210A identified by a certain media ID 212 from the media information database 210. The concept relationship calculation unit 130 extracts the dominant concept set 243, the sub-concept set 244, the text feature 245, the image feature 246, and the feature 247 of other media of the analysis information data 240A identified by a certain media ID 242 from the analysis information database 240. The concept relationship calculation unit 130 calculates the concept relationship for each dominant concept set 243 and sub-concept set 244. The conceptual relationship calculation unit 130 may allow the sender account ID 215, the update date 216, the text feature amount 245, the image feature amount 246, and the feature amount of other media 247 to affect the calculation of the conceptual relationship. The conceptual relationship calculation unit 130 updates the conceptual database 220 and the conceptual relationship database 230 as needed based on the calculation result.

Figure 16 shows the configuration of the account creation section.

Meanwhile, the account creation unit 140 registers an account for a user who uses the electronic device 40 to log in to a service provided by the information processing system 1 (step S4). The user uses the electronic device 40 to input user information (step S41).

By logging in, the user is assigned a unique account and can use the unique services provided by the information processing system 1. The unique services include history management, tagging of media information, browsing of tagged media information, writing of comments, provision of content tailored to preferences, and distribution of advertisements. When registering for the first time, the user may be required to provide his/her name (user name), email address, mobile phone number, SNS account, personal certificate, payment information, etc. as necessary. When registering for the first time, the user may specify one or more keywords to be used as a criterion for determining the preference information. When determining the keywords, the preference information may be classified by simple questions. The preference information may be calculated using a model learned by machine learning from the network usage history of the media information viewed by the user, other users followed by the user, and other users who are followers of the user. When the user logs in, two-step authentication may be required based on the registered information. The device information on which the user logged in is registered in the server that operates the service, and two-step authentication may be required if the user logs in with a device that has no history. The account information is stored in the account database 250.

Figure 17 shows the account database.

A plurality of account data 250A are registered in the account database 250. Each account data 250A is a mutually associated and registered account ID 251, user name 252, preference feature 253, browsing history media ID list 254, following account list 255, and follower account list 256.

Based on the browsing history media ID list 254, following account list 255, follower account list 256, preference information at the time of initial registration, etc. registered in the account database 250, the account creation unit 140 calculates the preference features 253 of the user logging in to the service using the electronic device 40 using a model learned by machine learning, and registers the calculated preference features 253 in the account database 250 (step S42).

Figure 18 shows the configuration of the account usage history analysis unit.

When a user logs in to an account and uses a service provided by the information processing system 1, the account usage history analysis unit 150 recalculates the preference feature amount based on the user's usage history (step S5). Specifically, the account usage history analysis unit 150 reads the account ID 251, the preference feature amount 253, and the browsing history media ID list 254 from the account database 250, and recalculates the preference feature amount 253 based on the browsing history media ID list 254, which is the usage history (step S51). The account usage history analysis unit 150 may further recalculate the preference feature amount 253 based on the following account list 255 and the follower account list 256. The account usage history analysis unit 150 registers (updates) the recalculated preference feature amount 253 in the account database 250. In this way, the user's preferences are analyzed based on the keywords selected at the time of initial registration and the past usage history of the service.

Figure 19 shows the configuration of the startup recommendation section.

The start-up recommendation unit 160 selects a plurality of pieces of media information each having a media feature amount that satisfies a predetermined condition for the preference feature amount 253 as a plurality of first media objects 410 and a plurality of second media objects 420 to be initially displayed (step S6). Specifically, when the user starts an application program using the electronic device 40 (step S61), the start-up recommendation unit 160 reads the account database 250. The start-up recommendation unit 160 reads the preference feature amount 253 associated with the account ID 251 of the logged-in user from the account database 250. The start-up recommendation unit 160 extracts a plurality of analysis information data 240A having media features 245, 246, 247 that satisfy a predetermined condition for the read preference feature amount 253 from the analysis information database 240. This makes it possible to provide a method of recommending search need candidates to the user (displaying an initial screen based on preferences) and a display method (UI) to enable searches for search needs that the user is not aware of.

The startup recommendation unit 160 extracts from the media information database 210 a plurality of media data 210A having the same media ID 211 as the media ID 242 of the extracted plurality of analysis information data 240A. The data generation unit 180 generates a plurality of first media objects 410 and a plurality of second media objects 420 based on the title 212, text 213, and image identifier 214 of the extracted plurality of media data 210A (step S10). The plurality of first media objects 410 and the plurality of second media objects 420 are display data for displaying the selected plurality of media information, which are supplied to the electronic device 40 via the web server 30. The data generation unit 180 causes the display device 41 of the electronic device 40 to display a top screen 400 including the plurality of first media objects 410 and the plurality of second media objects 420 based on the plurality of media data 210A via the web server 30 (step S62). That is, the web server 30 supplies the electronic device 40 with display data for displaying the multiple pieces of media information selected by the information processing device 10 .

Figure 27 shows a modified example of the configuration of the startup recommendation section.

Furthermore, the startup recommendation unit 160 may not only extract the analysis information data 240A from the analysis information database 240, but also generate data of media information that matches the user's preference feature 253 (step S63). In this case, the data generation unit 180 may generate a first media object 410 or a second media object 420 for displaying the generated data of the media information.

The user browses the top screen 400 (step S7). The user operates (e.g., drags and drops) the first media object 410 so that it is directed above the first functional object 430, thereby selecting the first functional object 430 and performing a search (step S8). When transitioning from the top screen 400, the first functional object 430 does not change from the specified position, but the information in the first media object 410 is changed as follows (step S9).

Figure 20 shows the configuration of the search time recommendation unit. Figures 28 and 29 are diagrams for explaining an example of the operation of the search time recommendation unit.

When an operation is performed such that one first media object 410 is directed toward a first functional object 430, the search time recommendation unit 170 selects a plurality of pieces of media information each having a concept that has a predetermined conceptual relationship with the concept of the media information indicated by the one first media object 410. Specifically, the user operates the first media object 410 so as to direct it toward the first functional object 430 (step S81). In the example of FIG. 28, the media information indicated by the first media object 410 operated by the user is a "video of a cat jumping toward the moon."

The search time recommendation unit 170 extracts the media ID 211 of the media data 210A indicated by the first media object 410 selected by the user from the media information database 210. The search time recommendation unit 170 extracts the dominant concept 243 and sub-concept 244, and media features 245, 246, and 247 of the media ID 242 that is the same as the extracted media ID 211 from the analysis information database 240. The search time recommendation unit 170 extracts the concept ID 222 of the extracted dominant concept 243 and sub-concept 244 from the concept database 220.

When the dominant concept 243 and subconcept 244 identified by the extracted concept ID 222 are taken as concepts from 233, the search time recommendation unit 170 extracts concepts to 234 whose strength of relationship 235 from the concept from 233 satisfies a predetermined condition (e.g., a threshold value or more) from the concept relationship database 230. The search time recommendation unit 170 extracts media IDs 242 associated with the concept ID of the extracted concept to 234 (i.e., media IDs 242 of media information having the concept to 234 as a dominant concept 243 or subconcept 244) from the analysis information database 240. The search time recommendation unit 170 extracts analysis information data 240A of the extracted media ID 242 from the analysis information database 240 (step S91). This enables the search recommendation unit 170 to select media (media identified by the media ID 242) that has a predetermined conceptual relationship to the media data 210A indicated by the first media object 410 selected by the user.

As a process of broadening the search target for the selected media information, information (sub-concept) that the media information does not directly have can be added using the concept from 233 and the concept to 234. This process can be performed not only once but also recursively any number of times. For example, a second concept (sub-concept) can be derived from a first concept (dominant concept), and a third concept (sub-concept) can be derived from the second concept (sub-concept). That is, the search time recommendation unit 170 may add at least one dominant concept 243 and/or sub-concept 244 having a predetermined concept relationship to at least some of the dominant concepts 243 and/or sub-concepts 244 of the concept of the media information selected by the user to the concept of the media information. This allows the search time recommendation unit 170 to further select media having a concept that does not have a direct concept relationship with the concept of the media information selected by the user.

In the examples of Figures 28 and 29, the search-time recommendation unit 170 extracts the dominant concept 243 "cat" (Dominant in Figure 29) of "video of a cat jumping towards the moon" and the sub-concepts 244 "moonlight" and "jump" (Sub in Figure 29). The search-time recommendation unit 170 extracts "hunchback" and "rescue cat adoption event" (Sub1-1 in Figure 29) as sub-concepts 244 (concept to 234) that have a relationship with the dominant concept 243 "cat" (concept from 233). The search-time recommendation unit 170 extracts media information 210A (e.g., a video of exercises to prevent hunchback, an advertisement for a rescue cat adoption event) that has the sub-concepts 244 "hunchback" and "rescue cat adoption event".

As described above, even if the strength of relationship 235 from the concept "today" (first concept) to the concept "event" (second concept) is registered, the strength of relationship 235 from the concept "event" to the concept "today" in the reverse direction does not necessarily have to be registered. In addition, the strength of relationship 235 from the concept "today" to the concept "event" does not necessarily match the strength of relationship 235 from the concept "event" to the concept "today" in the reverse direction. For this reason, for example, even if media information having the concept "event" is extracted from media information having the concept "today" using a key, media information having the concept "today" is not extracted from media information having the concept "event" using a key. Here, in addition to media information having a direct relationship derived from the dominant concept 243, media information having an indirect relationship derived from the subconcept 244 is selected. As a result, media information that is unexpected and not patterned is extracted for the user.

Figure 30 shows a schematic comparison between a previous search and a search in this embodiment.

As shown in (A), conventional searches are based on the assumption that users are aware of their search needs and can spontaneously describe them in language. For this reason, first, the user's preferences are uniquely identified by search words (including image search and multimedia search), and as a result, information that is directly related to the user's preferences (search words) is searched for. In such a multimedia search system, it is impossible to provide the user with information other than what the user expects in response to the search query. In other words, the system can only provide the user with information within the scope of his or her expectations.

In contrast, as shown in (B), in this embodiment, one of the recommended media information is used as a search query, so that the user can enter a search query unconsciously, and a suggestive tendency preference is acquired, rather than the user's unique preference. In addition to media information that has a direct relationship to the search query (one of the recommended media information), media information that has an indirect relationship can also be included in the search results. This allows the user to perform a search even if he or she is unaware of the search needs, and by including multimedia that the user did not expect but that is guaranteed to be related to the search target in the search results, it is possible to provide the user with the discovery of new connections and the expansion of exploration.

Returning to the description of the operation, the search time recommendation unit 170 extracts preference features 253 associated with the account ID 251 of the logged-in user from the account database 250. The search time recommendation unit 170 filters the analysis information data 240A based on the extracted preference features 253 (step S92).

The search recommendation unit 170 extracts from the media information database 210 a plurality of media data 210A having the same media ID 211 as the media ID 242 of the plurality of post-filtering analysis information data 240A. The data generation unit 180 generates a plurality of first media objects 410 and a plurality of second media objects 420 based on the title 212, text 213, and image identifier 214 of the extracted plurality of media data 210A (step S10). The data generation unit 180 causes the display device 41 of the electronic device 40 to display a top screen 400 including a plurality of first media objects 410 and a plurality of second media objects 420 based on the plurality of media data 210A on the display device 41 via the web server 30.

Figure 31 shows a modified example of the configuration of the search recommendation section.

Furthermore, the search time recommendation unit 170 may not only extract the analysis information data 240A from the analysis information database 240, but also generate media information data that matches the user's preference feature 253 (step S93). In this case, the data generation unit 180 may generate a first media object 410 or a second media object 420 for displaying the generated media information data.

When the first search is performed, the search time recommendation unit 170 extracts the dominant concept 243 and the subconcept 244 from the media information 210A to be searched. They may be pre-calculated. At this time, the search time recommendation unit 170 may add a concept that has a relationship in the concept relationship database 230 to each dominant concept 243 as a subconcept 244. The search time recommendation unit 170 searches the media information 210A database, the analysis information database, the concept database, and the concept relationship database 230 to obtain one or more pieces of media information 210A and features related to the media information 210A to be searched. At this time, the search time recommendation unit 170 narrows the search range to only the media information 210A that has a dominant concept 243 or a subconcept 244 common to the media information 210A to be searched. Finally, the search time recommendation unit 170 performs display based on the display priority (including not displaying). The display priority and whether or not to display may be determined based on the user's preferences, the importance of the concept using TF-IDF (Term Frequency - Inverse Document Frequency), and the like.

When the nth search is performed, the search time recommendation unit 170 extracts dominant concepts 243 and subconcepts 244 from the media information 210A to be newly searched. These may be pre-calculated. At this time, the search time recommendation unit 170 may add concepts that have a relationship in the concept relationship database 230 to each dominant concept 243 as subconcepts 244. The search time recommendation unit 170 searches the media information 210A database, the analysis information database, the concept database, and the concept relationship database 230 to obtain one or more pieces of media information 210A and feature quantities related to the media information 210A to be searched. At this time, the search time recommendation unit 170 narrows the search range to only media information 210A that has a dominant concept 243 or subconcept 244 in common with the media information 210A to be searched. Here, the search range may be further narrowed down using the overlapping parts of all or any of the search ranges from the first to the n-1th search, or the search range may be widened using those search ranges. The search ranges from the first to the n-1th search may not be used. Finally, the search time recommendation unit 170 performs display based on the display priority (including not displaying). The display priority may be determined based on the user's preferences, the importance of the concept using TF-IDF (Term Frequency - Inverse Document Frequency), and the like.

According to this embodiment, media information recommended to a user is selected based on not only the relationship between words, but also the directional relationship between the dominant concept 243 and the sub-concept 244. Here, in addition to the media information with a direct relationship derived from the dominant concept 243, the media information with an indirect relationship derived from the sub-concept 244 is selected. Therefore, rather than presenting ordinary related information, media information that is rather surprising to the user is presented. For the user, a different topic that the user knew about but did not realize was related to a topic of interest may be presented. In this way, according to this embodiment, it is possible to stimulate the user's intellectual curiosity and provide the user with the discovery of new connections and the expansion of exploration. This creates an incentive for the user to want to use the service more.

According to this embodiment, as described above, the conceptual relationship calculation unit 130 calculates and updates the conceptual relationship between the concepts of multiple media information present on the network. The account usage history analysis unit 150 calculates and updates the preference feature 253. Therefore, even if the multiple media information present on the network does not change (increase or decrease), the media information recommended to a specific user changes dynamically both at startup and search due to changes in the relationship between the multiple media information and changes in the user's preference feature 253. Therefore, it is possible to always provide users with the opportunity to discover new connections and expand their exploration.

5. Other

(1) Account history service

A logged-in user can view past usage history. Usage history includes posting history, search history, history of media information for which detailed information was viewed, comments viewed, comments written, etc. A user can delete all or part of the history. A user can select the history to view. If a search history is selected, the search history at the time of selection is displayed. If a history of media information for which detailed information was viewed is selected, the detailed information can be confirmed. Each user can check the users who are following them and the users who are being followed by them. From these confirmation screens, users can move to their respective My Pages. Account usage history is stored in the account database 250.

(2) Influence functions (advertising, flyers)

A logged-in user who wishes to post an advertisement can use media objects 410, 420 to post their own advertisement. The advertisements are viewed by the user. When the user selects an advertisement, the existing web browser is launched and the user can move to an external site or check detailed information. The advertisements viewed by the user are selected based on keywords selected by the advertiser. The relevance and appropriateness of the selected keyword and the posted advertisement are evaluated by the analysis system, and the keyword, the display date and time, and the display location are set as the advertisement slot, and the display price of this advertisement slot can be set by the person wishing to display the advertisement. The price for the advertisement slot can be set by the person wishing to display the advertisement making a bid. The display priority of the advertisements is set based on the analysis of the relevance with the keywords and the unit price, etc. The display priority of advertisements that are analyzed to have low relevance to the keywords can be set low.

(3) Posting function

Users can post media information consisting of one or more media that they have created. Keywords can be set for posted media information. Posted media information is set using the keywords set when posting and highly relevant keywords extracted using an analysis function, and a keyword group is set. Media information that is judged to be highly relevant and important based on the keyword group is set to a group for that posted media information, and is displayed when that media information is searched using the system's functions. Posters can select and delete all or part of the media information that has been judged to be highly relevant and set to a group. Posters can add media information that is judged to be highly relevant to a group of media information that they have posted.

(4) Autoplay function

Figure 21 shows an example of a top screen. Figure 22 shows an example of a screen after the GUI of the electronic device has changed (autoplay).

21 , a first media object 410 and a second media object 420 indicate media information including video and audio. As shown in (A), when a user performs an operation (such as a touch operation) on the first media object 410 that does not involve movement, a play button may be displayed in the first media object 410. As shown in (B), when a user performs an operation (such as a touch operation) on the second media object 420 that does not involve movement, a play button may be displayed in the second media object 420.
Furthermore, when the user performs an operation that does not involve movement on the first media object 410 or the second media object 420 on which the play button is displayed, a transition to a details page (Figure 22) may be made and the start screen (A) may be displayed.

The user can automatically play the listed media information. The automatic playback method is a combination of one or more of the following methods: an automatic transition method in which the media information displayed on the screen is switched every time a certain time has passed, a method in which each piece of media information such as a video is played in sequence, and a method in which audio information is played in sequence by reading out text. On the start screen of FIG. 22 (A), the user can change the automatic transition time of the video, which is the media information, the automatic playback speed, and the playback speed of the audio information. When the user operates the screen during various playbacks of (B), the operation screen of (C) is displayed. On the operation screen, the user can adjust the playback speed, play, stop playback, rewind for a certain amount of time, forward for a certain amount of time, move to the previous media information, move to the next media information, and the like. On the other hand, as shown in (D), in the case of media mainly consisting of still images, the still images are displayed large. As described above, the media information is composed of at least one of a plurality of different modalities (formats), such as still images, video images, audio, and text. The type of modality is not limited to this example and may be any format.

(5) Variations in the top screen

Figures 23, 24, and 25 are modified examples of the top screen.

23, the layout of the top screen 400 is not limited to the layout of the above embodiment. FIGS. 24 and 25 are examples of the top screen 400 in which the electronic device 40 is displayed on a browser of a personal computer rather than a smartphone. FIG. 24 is an example of the top screen 400 in which the community function, the recommendation function, and other functions are listed together with the media display function. FIG. 25 is an example of the top screen 400 that displays only media information. In FIG. 25, only the display area 471 including the first media object 410 and the first functional object 430 may be displayed. For example, it is useful to display only the display area 471 when the displayable area is narrow, such as VR glasses. In any of FIGS. 23, 24, and 25, any layout may be used as long as multiple first media objects 410 are displayed around the first functional object 430.

Although each embodiment and each variant of the present technology has been described above, the present technology is not limited to the above-described embodiments, and various modifications can of course be made without departing from the spirit and scope of the present technology.

1 Information processing system 10 Information processing device 20 Storage device 30 Web server 40 Electronic device 400 Top screen 401 First display area 402 Second display area 41 Display device 410 First media object 42 Control circuit 420 Second media object 430 First functional object 440 Account object 450 Third media object 451 Second functional object 110 Collection unit 120 Media information analysis unit 130 Concept relationship calculation unit 140 Account creation unit 150 Account usage history analysis unit 160 Start-up recommendation unit 170 Search-time recommendation unit 210 Media information database 220 Concept database 230 Concept relationship database 240 Analysis information database 250 Account database

Claims (11)

Extracting a plurality of concepts contained in each of a plurality of pieces of media information, and classifying the extracted plurality of concepts into dominant concepts and sub-concepts;
A concept having a predetermined conceptual relationship with the dominant concept of each piece of media information is additionally determined as a sub-concept of the media information;
a media information analysis unit that sets a set including all of the dominant concepts and all of the sub-concepts of the media information as a concept set of the media information;
a concept relationship calculation unit that calculates a concept relationship between each concept included in a concept set of a certain media information and each concept included in a concept set of another media information ;
and a search recommendation unit that allows a user to select media information and selects a plurality of media information having individual concepts that have a predetermined conceptual relationship with each concept included in a concept set of the selected media information. 2. The information processing device according to claim 1,
The search recommendation unit selects a plurality of media information having a dominant concept and/or a sub-concept having a predetermined conceptual relationship with the dominant concept and the sub-concept included in the concept set of the media information selected by the user. 3. The information processing device according to claim 2,
the search time recommendation unit determines that dominant concepts and/or sub-concepts having a predetermined conceptual relationship with at least a portion of the dominant concepts and/or sub-concepts among the dominant concepts and the sub-concepts included in the concept set of the media information selected by the user are recursively included in the concept set of the media information selected by the user;
An information processing device that selects a plurality of media information items having a dominant concept and/or a sub-concept that has a predetermined conceptual relationship with the concept set . 4. The information processing device according to claim 2,
When selecting the media information, the search recommendation unit
When the dominant concept and the sub-concept included in the concept set of the media information selected by the user or the search time recommendation unit are taken as a concept from, a concept to is extracted whose strength of relationship from the concept from is equal to or greater than a threshold value;
An information processing device that selects a plurality of pieces of media information having the extracted concept "to" as a dominant concept or a sub-concept. 5. The information processing device according to claim 4,
An information processing device in which the presence or absence or strength of a relationship from a first concept, which is a concept from, to a second concept, which is a concept to, does not match the presence or strength of a relationship from the second concept, which is a concept from, to the first concept, which is a concept to. 2. The information processing device according to claim 1,
The media information analysis unit calculates media feature quantities which are feature quantities of a plurality of pieces of media information;
an account creation unit that calculates a preference feature amount of a user who uses an electronic device;
a start-up recommendation unit that selects a plurality of pieces of media information each having a media feature amount that satisfies a predetermined condition with respect to the preference feature amount as a plurality of pieces of media information to be initially displayed;
The information processing device further comprises: 7. The information processing device according to claim 6,
The media information includes a plurality of sub-media information,
the media information analysis unit calculates a media feature amount for each of the plurality of sub-media information;
The media information has a plurality of media feature amounts for each of the plurality of sub-media information. 2. The information processing device according to claim 1,
The information processing device further comprises: a data generating unit configured to generate display data for displaying the selected plurality of pieces of media information, the display data being supplied to the electronic device via a web server. 7. The information processing device according to claim 6,
The information processing device further comprises an account usage history analysis unit that recalculates the preference feature amount based on the usage history of the user. By computer,
Extracting a plurality of concepts contained in each of a plurality of pieces of media information, and classifying the extracted plurality of concepts into dominant concepts and sub-concepts;
A concept having a predetermined conceptual relationship with the dominant concept of each piece of media information is additionally determined as a sub-concept of the media information;
A set including all of the dominant concepts and all of the sub-concepts of the media information is defined as a concept set of the media information;
Calculating a conceptual relationship between each concept included in a concept set of a certain piece of media information and each concept included in a concept set of another piece of media information ;
An information processing method comprising the steps of: having a user select media information; and selecting a plurality of media information pieces each having an individual concept that has a predetermined conceptual relationship with an individual concept included in a concept set of the selected media information. A control circuit of the information processing device,
Extracting a plurality of concepts contained in each of a plurality of pieces of media information, and classifying the extracted plurality of concepts into dominant concepts and sub-concepts;
A concept having a predetermined conceptual relationship with the dominant concept of each piece of media information is additionally determined as a sub-concept of the media information;
a media information analysis unit that sets a set including all of the dominant concepts and all of the sub-concepts of the media information as a concept set of the media information;
a concept relationship calculation unit that calculates a concept relationship between each concept included in a concept set of a certain media information and each concept included in a concept set of another media information ;
An information processing program that operates as a search recommendation unit that allows a user to select media information and selects a plurality of media information having individual concepts that have a predetermined conceptual relationship with each concept included in a concept set of the selected media information.

Images