JP2024061039A - Information processor, information processing method and information processing program - Google Patents

Abstract

To conduct appropriate information provision.SOLUTION: An information processor according to the present application has an acquisition part and a provision part. The acquisition part acquires the interest object information that indicates an interest object of a user and the message information that indicates a message in a messenger application. Based on the interest object information and the message information, the provision part provides the user with information about the interest object that is displayed with the message in the messenger application.SELECTED DRAWING: Figure 6

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, there are known techniques for providing various information to users. For example, there is a technique for providing a product recommendation page in response to an input of a search query (for example, Patent Document 1).

Patent No. 6723182

However, there is room for improvement in the above conventional technology. The above conventional technology merely provides a product recommendation page that corresponds to a query, and it is difficult to provide information to a user if the user has not entered a query. Therefore, it is desirable to provide more appropriate information without relying on the user inputting a query, etc.

The present application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can provide appropriate information.

The information processing device according to the present application is characterized by comprising: an acquisition unit that acquires interest subject information indicating an interest subject that is an object of interest of a user, and message information indicating a message in a messenger application; and a provision unit that provides the user with information regarding the interest subject that is displayed together with the message in the messenger application, based on the interest subject information and the message information.

According to one aspect of the embodiment, it is possible to provide appropriate information.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram showing an example of a predetermined operation indicating a user's interest. FIG. 3 is a diagram illustrating an example of information processing executed by the information processing system. FIG. 4 is a diagram illustrating an example of information processing executed by the information processing system. FIG. 5 is a diagram illustrating an example of information processing executed by the information processing system. FIG. 6 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 7 is a diagram illustrating an example of a user information storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of a target information storage unit according to the embodiment. FIG. 9 is a diagram illustrating an example of a content storage unit according to the embodiment. FIG. 10 is a diagram illustrating an example of a model information storage unit according to the embodiment. FIG. 11 is a flowchart showing an example of the flow of information processing. FIG. 12 is a flowchart showing an example of the flow of information processing. FIG. 13 is a flowchart showing an example of the flow of information processing. FIG. 14 is a flowchart showing an example of the flow of information processing. FIG. 15 is a flowchart showing an example of the flow of information processing. FIG. 16 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device.

Below, the information processing device, information processing method, and information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. Furthermore, the embodiments can be appropriately combined as long as they do not cause inconsistencies in the processing content. Furthermore, the same parts in the following embodiments will be given the same reference numerals, and duplicated explanations will be omitted.

(Embodiment)
[1. An example of information processing]
First, an example of information processing executed by the information processing system 1 will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of information processing according to an embodiment. The information processing device 100 estimates a target in which a user is interested (also called a "target of interest"), and executes information processing including providing information on the estimated target of interest. For example, the target of interest includes various targets of trading such as facilities such as stores and products. The trading target is not limited to products, and may be the use of a service, etc. Note that the above is merely an example, and the target of interest may be any target as long as it is a target of interest of the user.

[1-1. Overview of processing in information processing system]
From here, an overview of the flow of information processing in the information processing system 1 will be described with reference to Fig. 1. In the following, a case where the information processing device 100 of the information processing system 1 performs processing such as estimating the user's interests will be described as an example, but each process shown below may be performed not only by the information processing device 100 but also by any device included in the information processing system 1, such as the terminal device 10, depending on the device configuration and function included in the information processing system 1. For example, the terminal device 10 may perform processing such as estimating the user's interests, but this point will be described later.

The information processing device 100 acquires information about the model (also called "model information") from the external device 20 (step S1). For example, the information processing device 100 acquires various models such as model M1, which is a domain estimation model used to estimate a domain. Note that the information processing device 100 may learn various models such as model M1 on its own device, but this will be described later.

The information processing device 100 also acquires information about the target (also referred to as "target information") from the external device 20 (step S2). The information processing device 100 acquires information about facilities such as stores (also referred to as "facility information") as target information. For example, the information processing device 100 acquires facility information about facilities such as physical stores that exist at the time of processing from the external device 20. The facility information includes various information about the facility. For example, the facility information includes various information used to provide information about the facility. For example, the facility information includes various information about the facility such as the location (position) of the facility, business hours of the facility, category, an image of the facility, trading objects provided by the facility, and images of the trading objects provided by the facility.

The information processing device 100 also acquires information on trading objects traded in the online shopping mall (also referred to as "trading object information") as object information. For example, the information processing device 100 acquires trading object information on trading objects being sold (listed) in the online shopping mall from the external device 20. The trading object information includes various information on the trading object. For example, the trading object information includes various information used to provide information on the trading object. For example, the trading object information includes various information on the trading object, such as the seller, selling price, an image of the trading object, and category.

The target information may also include information from various services. For example, the target information may include information posted about the target on a social networking service (SNS). For example, the target information may include posted information (content) posted by a user on the SNS. For example, the posted information may be text information posted by a user on Instagram (registered trademark), TikTok (registered trademark), YouTube (registered trademark), Twitter (registered trademark), Facebook (registered trademark), etc., or image information such as images and videos.

The information processing device 100 also acquires information about the user (also referred to as "user information") from the terminal device 10 used by the user (step S3). The terminal device 10 transmits the user information about the user to the information processing device 100. For example, the information processing device 100 acquires user information including user behavior information from the terminal device 10 used by the user. For example, the information processing device 100 acquires user information including an operation history of the user's operation of the terminal device 10.

For example, the user information may include a history of the user's actions, such as actions on the Internet. For example, the user information may include a user's actions history, such as a browsing history of information about trading objects in an electronic commerce (EC) service such as an online shopping mall, a purchasing history, etc. Note that trading objects traded in an electronic commerce service include trading objects traded in various ways, such as an online shopping mall (online mall), an online shopping site, an auction site, a flea market site, etc. The user information may also include a user's actions history in the real world, such as the user's location information.

The user information also includes sensor information detected about the user. For example, the user information includes sensor information detected by the terminal device 10. The user information may also include location information detected by the terminal device 10. The user information may also include attribute information of the user. For example, the attribute information includes various information such as information indicating demographic attributes such as age, sex, residential area, and occupation, and information indicating psychographic attributes such as interests and lifestyle. Note that the attribute information is merely an example, and the user information may include any information related to the user. The information processing device 100 may also acquire the user information of the user from the external device 20.

Then, the information processing device 100 performs processing to provide information to the user based on the various information. The information processing device 100 estimates the user's interests based on the user information (step S4). For example, the information processing device 100 may estimate the user's interests based on various information such as images generated by the user operating the terminal device 10, but this point will be described later.

Then, the information processing device 100 provides information to the user based on the estimated interests of the user (step S5). For example, the information processing device 100 provides the user with information indicating the interests of the user at a predetermined timing. For example, the information processing device 100 transmits content corresponding to the user's interests to the terminal device 10 used by the user. In this way, the information processing device 100 estimates the interests of the user and provides the user with information about the estimated interests, thereby allowing the information processing device 100 to provide appropriate information.

[1-2. Details of various processes]
Based on the above-mentioned information processing, various processes will be described in detail. Note that the same points as those described above will not be described as appropriate. In addition, various processes described below may be combined as appropriate. For example, a map content described later may be provided (displayed) in a messenger application described later. Note that the above is only an example, and various processes may be combined in any way as long as the combined processes are possible. In the following, the information processing system 1 will be described as the processing subject, but depending on the device configuration included in the information processing system 1, any device such as the information processing device 100 or the terminal device 10 may perform the process. In addition, the process described below as the information processing device 100 may also be the processing subject of the terminal device 10. In addition, in the following description, the user U1 will be described as an example of a user.

[1-2-1. Information provided by
First, an example of providing information according to a user's interests in the information processing system 1 will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of a predetermined operation indicating a user's interests. In FIG. 2, a process using an image (also called a "captured image") generated in response to a capture (screenshot) operation by a user is described. The information used to estimate the user's interest is not limited to a captured image, and any information may be used. For example, the information used to estimate the user's interest may be an image captured by the user using the terminal device 10, sensor information, etc. A variety of information may be used, as will be discussed below.

In FIG. 2, in response to an operation by user U1 on a capture image C1 displayed by app AP11, which is an application installed on terminal device 10, the information processing system 1 estimates whether or not an object contained in the capture image C1 is an object of interest to user U1. In the example shown in FIG. 2, the information processing system 1 generates a capture image C1, which is an image capturing content showing a store, steak house SH, by user U1 operating the terminal device 10.

For example, app AP11 may be an application for the user to manage capture images. For example, the user causes app AP11 to display on the terminal device 10 capture images (also called "unprocessed capture images") for which the need for stock (favorite) registration has not yet been processed, and performs an operation to determine whether or not the displayed capture images need to be registered as favorites. The information processing system 1 manages the capture images for which the user has performed an operation to register them as favorites, as capture images that indicate the user's interests.

In FIG. 2, user U1 performs an operation such as swiping or flicking to move captured image C1 displayed on terminal device 10 to the left or right, thereby determining whether or not to register captured image C1 as a favorite.

For example, when user U1 performs operation OP1 to move captured image C1 to the left, the terminal device 10 accepts the operation OP1 as an operation that does not require the capture image C1 to be registered as a favorite. In other words, when user U1 performs operation OP1 to move captured image C1 to the left, the information processing system 1 determines that user U1 has no interest in the subject of captured image C1, and does not register captured image C1 as a favorite for user U1.

For example, when user U1 performs operation OP2 to move captured image C1 to the right, the terminal device 10 accepts operation OP2 as an operation to register captured image C1 as a favorite. In other words, when user U1 performs operation OP2 to move captured image C1 to the right, the information processing system 1 registers captured image C1 as a favorite for user U1, assuming that user U1 is interested in the subject of captured image C1.

Note that the left-right directions of the above operations OP1 and OP2 are merely examples, and the left-right directions of operations OP1 and OP2 may be reversed. In this case, the terminal device 10 accepts an operation by the user to move the captured image to the left as an operation that requires registration of the captured image as a favorite. Also, the terminal device 10 accepts an operation by the user to move the captured image to the right as an operation that does not require registration of the captured image as a favorite.

The above operations OP1 and OP2 are merely examples of operations as to whether or not to register a favorite, and the terminal device 10 may accept various operations by the user as operations as to whether or not to register a favorite. In FIG. 2, when user U1 designates button BT1, the terminal device 10 may accept the designation as an operation as to not require the capture image C1 to be registered as a favorite. Also, when user U1 designates button BT2, the terminal device 10 may accept the designation as an operation as to require the capture image C1 to be registered as a favorite.

In addition, although only the capture image C1 is illustrated in FIG. 2, when there are multiple unprocessed capture images, the terminal device 10 may sequentially display the unprocessed capture images in response to an operation on whether or not to register the unprocessed capture images as favorites. In FIG. 2, after accepting an operation on whether or not to register the capture image C1 as favorites, the terminal device 10 may display another unprocessed capture image (e.g., capture image C2) and accept this as an operation on whether or not to register the capture image C2 as favorites. In addition, after accepting an operation on whether or not to register the capture image C2 as favorites, the terminal device 10 may display an unprocessed capture image (e.g., capture image C3) if there is an unprocessed capture image, and accept this as an operation on whether or not to register the capture image C3 as favorites. In this way, when there are multiple unprocessed capture images, the terminal device 10 sequentially displays the capture images C1, C2, C3, etc. in response to an operation on whether or not to register the capture images as favorites, and sequentially accepts an operation on whether or not to register each unprocessed capture image as favorites.

For example, when user U1 performs an operation to register captured image C1 as a favorite, information processing system 1 estimates the subject of interest of user U1 using captured image C1. For example, information processing device 100 estimates the subject of interest of user U1 based on the results of an analysis process that analyzes captured image C1. Information processing device 100 estimates the subject of interest of user U1 based on the subject included in captured image C1. In this case, information processing device 100 estimates steak house SH, which is a store included in captured image C1, as the subject of interest of user U1. Note that information processing system 1 may also estimate the subject of interest of the user using a model, but this will be described later.

The information processing system 1 provides information about the subject of interest to the user at a predetermined timing. In FIG. 2, the information processing system 1 provides information about the steak house SH to the user U1 at a predetermined timing. For example, the information processing system 1 provides information about the steak house SH to the terminal device 10 used by the user U1 when the user U1 approaches the steak house SH. For example, the information processing system 1 compares the location information of the user U1 with the location of the steak house SH, and displays information about the steak house SH on the terminal device 10 used by the user U1 when the user U1 is located within a predetermined range from the steak house SH. Note that, when the subject of interest of the user is a product, the information processing system 1 may provide information about the product at a timing when the user can purchase the product. For example, when the subject of interest of the user is a product, the information processing system 1 may provide information about the product when the product is in stock.

In the above example, the user's operation of registering a captured image as a favorite has been described as an example of a predetermined operation indicating the user's interest, but the predetermined operation indicating the user's interest may be any operation, not limited to the user's operation of registering a captured image as a favorite. For example, the predetermined operation indicating the user's interest may be the user's operation of generating a captured image. In this case, in FIG. 2, when user U1 performs an operation of generating captured image C1, the information processing system 1 may infer that the steak house SH, which is an object included in captured image C1, is an object of interest of user U1.

As described above, the information processing system 1 estimates the user's subject of interest in response to a specific operation by the user, and provides the user with information related to the estimated subject of interest, thereby enabling the provision of appropriate information. In this manner, when a user performs a favorite registration operation, the information processing system 1 estimates the subject of interest at the time the user performed the registration operation, based on information obtainable from the terminal device. Then, the information processing system 1 provides the user with information related to the estimated subject of interest at a specific timing.

The information processing system 1 analyzes the captured image to estimate the target of interest. For example, the information processing system 1 may use a domain estimation model to identify a domain and estimate the target according to the domain, which will be described later. The information processing system 1 may also analyze sensor information detected by a specified sensor, not limited to the captured image, to estimate the target of interest of the user. For example, the information processing system 1 may estimate a store or the like that is a facility of interest (also called an "interesting facility") based on the direction and position, such as sound, position, browsing history, etc. The information processing system 1 may also estimate the target of interest of the user using attribute information of the user. For example, the information processing system 1 may estimate the target of interest of another user that is similar to the attribute of the user as the target of interest of the user.

The information processing system 1 may provide information notifications such as reminders related to the subject of interest at any timing depending on the subject of interest. For example, if the subject of interest is a product, the information processing system 1 may provide a push notification related to the subject of interest when a condition is met. Also, if the subject of interest is an event, the information processing system 1 may provide a push notification when tickets are sold. For example, the information processing system 1 may provide a reminder when providing content of a domain corresponding to the subject of interest.

For example, when the subject of interest is a store, the information processing system 1 may issue a reminder while displaying gourmet-related content, map-related content, etc. Also, when the subject of interest is a product, the information processing system 1 may issue a reminder while displaying e-commerce-related content, etc. For example, the information processing system 1 may issue a reminder when a corresponding message is displayed in a messenger application. For example, the information processing system 1 may estimate the subject of interest of the user based on information provided on television, radio, etc.

1-2-2. Domain-based processing
From here, an example of the case where the information processing system 1 provides information based on a domain will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of information processing executed by the information processing system. Note that the same points as those described in Fig. 2 and the like will not be described as appropriate. Also, a domain is used to classify the source of information such as content. For example, a domain indicates a classification of various services that are the source of information, such as EC service #1, restaurant introduction service #1, SNS service #1, SNS service #2, etc.

3 shows a case where the information processing system 1 uses model M1, which is a domain estimation model, to estimate the domain of the content (also referred to as "content CO2") corresponding to the capture image C2, and performs processing based on the estimated domain. Note that the information indicated as the character string "XXXXXX" in FIG. 3 is a character string indicating the name of the product (e.g., the name of a hair dryer) that is the subject of content CO2. For example, capture image C2 is a capture image generated by user U1, and is a capture image that is required to be registered as a favorite for user U1. Also, model M1 outputs information for estimating the domain to which the content corresponding to the input capture image belongs. For example, model M1 is a model that outputs a score indicating the possibility that the content corresponding to the input capture image belongs to each domain. Note that model M1 may be a model that outputs any information as long as it is possible to estimate the domain of the input data.

The information processing device 100 may acquire the model M1 from a model providing server or the like that provides a learning model, or may learn the model M1 on its own device. For example, the information processing device 100 acquires the model M1 from a model providing server or the like that provides a learning model. An example of the information processing device 100 learning a domain estimation model such as the model M1 will be described later.

In FIG. 3, the information processing device 100 inputs a capture image into the model M1, thereby estimating the domain of the content corresponding to the capture image. The information processing device 100 inputs the capture image C2 of the content CO2 into the model M1 (step S11). The model M1 to which the capture image C2 has been input outputs information indicating the domain estimated for the content CO2 (step S12). In the example of FIG. 3, the model M1 to which the capture image C2 has been input outputs scores corresponding to each of multiple domains such as EC service #1, restaurant introduction service #1, SNS service #1, and SNS service #2.

The information processing device 100 estimates that the domain of the content CO2 is domain DM1, as shown in the estimated domain information OT1, using the scores corresponding to each of the multiple domains output by the model M1. For example, the information processing device 100 estimates that the domain with the highest score among the scores corresponding to each of the multiple domains output by the model M1 is the domain to which the content CO2 belongs. For example, if the domain with the highest score among the scores corresponding to each of the multiple domains is EC service #1, the information processing device 100 estimates that the content CO2 is content of EC service #1 (also referred to as "domain A"). Furthermore, if the domain with the highest score among the scores corresponding to each of the multiple domains is restaurant introduction service #1 (also referred to as "domain B"), the information processing device 100 estimates that the content CO2 belongs to EC restaurant introduction service #1.

In addition, the information processing system 1 estimates the user's subject of interest using a query estimation model for each domain. In this case, for example, the information processing system 1 estimates that the subject related to the query extracted using the query estimation model is the user's subject of interest. The information processing system 1 estimates the user's subject of interest using a query estimation model (model M11) corresponding to EC service #1 (domain A), a query estimation model (model M12) corresponding to restaurant introduction service #1 (domain B), a query estimation model (model M13) corresponding to SNS service #1 (domain C), etc.

The information processing system 1 estimates the subject of interest of a user using a query estimation model corresponding to a domain estimated for a captured image, among a plurality of query estimation models corresponding to each of a plurality of domains. For example, query estimation models such as models M11 to M13 take an image such as a captured image as input, extract and output information to be used as a query in the input image (e.g., the name of an object, etc.). Note that while details of query estimation models such as models M11 to M13 will be described later, the query estimation model may be a model that outputs any information as long as it is capable of estimating information to be used as a query for input data. The query estimation model may be, for example, a single model common to all domains.

The information processing device 100 may acquire a query estimation model corresponding to each domain such as models M11, M12, M13, etc. from a model providing server or the like that provides a learning model, or may learn a query estimation model corresponding to each domain such as models M11, M12, M13, etc. on its own device. For example, the information processing device 100 acquires a query estimation model corresponding to each domain such as models M11, M12, M13, etc. from a model providing server or the like that provides a learning model. An example of the information processing device 100 learning a query estimation model corresponding to each domain such as models M11, M12, M13, etc. will be described later.

In FIG. 3, when the domain corresponding to the captured image C2 is domain A, the information processing device 100 estimates the interests of the user U1 using the model M11 of domain A. For example, when the information processing device 100 estimates the domain of the content CO2 to be domain A based on the output of the model M1 to which the captured image C2 is input, it estimates the interests of the user U1 based on the captured image C2 using the model M11 of domain A, as shown in query estimation EX1.

For example, the information processing device 100 inputs the capture image C2 to the model M11, causes the model M11 to output information to be used as a query, and estimates the subject of interest of the user U1 based on the information output by the model M11. In FIG. 3, when the model M11 to which the capture image C2 has been input outputs the character string "XXXXXX", the information processing device 100 estimates that the subject of interest of the user U1 is the object indicated by the character string "XXXXXX" (also referred to as "product X").

Furthermore, when model M11 to which captured image C2 has been input outputs the character string "XXXXXX", information processing device 100 extracts the character string "XXXXXX" as a query to be used for providing information to user U1. Using the character string "XXXXXX" extracted using model M11, information processing device 100 performs a search with a domain corresponding to domain A as the target domain, as shown in information search PS1. In FIG. 3, information processing device 100 determines a domain similar to domain A as the target domain. For example, information processing device 100 determines a domain related to EC services, similar to domain A (EC service #1), as the target domain. Information processing device 100 determines EC service #2, which is a domain similar to domain A (EC service #1), as the target domain.

Then, the information processing device 100 performs a search using the character string "XXXXXX" as a query, targeting the target domain, EC service #2. From among the contents of the target domain, EC service #2, the information processing device 100 extracts the content (also referred to as "content CO5") of the target (product X) indicated by the character string "XXXXXX" as the second content. As a result, the information processing device 100 extracts, from the contents of EC service #2, which is a target domain (second domain) different from domain A (first domain) of content CO2 (first content), content CO5 introducing product X as the second content.

Then, the information processing device 100 provides the content CO5 to the user U1. The information processing device 100 transmits the content CO5 to the terminal device 10 used by the user U1, and the terminal device 10 that receives the content CO5 displays the content CO5. In this way, the information processing device 100 displays the content CO5 on the terminal device 10 used by the user U1. As a result, the information processing device 100 provides the content CO5 (second content) of the EC service #2 (second domain) that is different from the EC service #1 (first domain) of the content CO2 (first content). In this way, the information processing device 100 estimates the user's interest based on the first content, and provides the user with information on the second content related to the estimated interest, thereby allowing the information processing device 100 to provide appropriate information.

Note that the above is merely an example, and the information processing device 100 may determine the target domain using various information. In FIG. 3, when there are multiple domains similar to domain A, the information processing device 100 may determine the domain used by user U1 among the multiple domains similar to domain A as the target domain. For example, the information processing device 100 may determine EC service #2 used by user U1 as the target domain among EC service #2 and EC service #3, which are domains similar to domain A (EC service #1).

For example, the information processing device 100 may determine, as the target domain, EC service #2 for which user U1 has points, among EC service #2 and EC service #3, which are domains similar to domain A (EC service #1). For example, the information processing device 100 may determine, as the target domain, EC service #2 for which user U1 has the most points, among EC service #2 and EC service #3, which are domains similar to domain A (EC service #1). For example, the information processing device 100 may determine, as the target domain, the domain with the lowest price for the object of interest, among EC service #1, EC service #2, and EC service #3.

When the domain corresponding to the capture image (hereinafter referred to as "capture image C3") is domain B, the information processing device 100 estimates the interests of user U1 using model M12 of domain B. For example, when the information processing device 100 estimates that the domain of content CO3 is domain B based on the output of model M1 to which capture image C3 is input, it estimates the interests of user U1 based on capture image C3 using model M12 of domain B, as shown in query estimation EX2.

For example, the information processing device 100 inputs the capture image C3 to the model M12, causes the model M12 to output information to be used as a query, and estimates the subject of interest of the user U1 based on the information output by the model M12. In FIG. 3, when the model M12 to which the capture image C3 has been input outputs the character string "YYY", the information processing device 100 estimates that the subject of interest of the user U1 is the subject indicated by the character string "YYY" (also referred to as "Restaurant Y").

Furthermore, when model M12 to which captured image C3 has been input outputs the character string "YYY", information processing device 100 extracts the character string "YYY" as a query to be used for providing information to user U1. Using the character string "YYY" extracted using model M12, information processing device 100 performs a search with a domain corresponding to domain B as the target domain, as shown in information search PS2. In FIG. 3, information processing device 100 determines a domain similar to domain B as the target domain. For example, information processing device 100 determines a domain related to a restaurant introduction service, similar to domain B (restaurant introduction service #1), as the target domain. Information processing device 100 determines restaurant introduction service #2, which is a domain similar to domain B (restaurant introduction service #1), as the target domain.

Then, the information processing device 100 performs a search using the character string "YYY" as a query for the restaurant introduction service #2, which is the target domain. From among the contents of the restaurant introduction service #2, which is the target domain, the information processing device 100 extracts the content (also referred to as "content CO6") of the target (restaurant Y) indicated by the character string "YYY" as the second content. As a result, the information processing device 100 extracts content CO6 introducing restaurant Y as the second content from the contents of the restaurant introduction service #2, which is the target domain (second domain) different from domain B (first domain) of content CO3 (first content). The information processing device 100 then provides content CO6 to user U1. The information processing device 100 transmits content CO6 to the terminal device 10 used by user U1.

Note that the above-mentioned processing is merely an example, and the information processing system 1 may perform processing using various information. The information processing system 1 may estimate the domain of the content without using a domain estimation model. In this case, the information processing system 1 may estimate whether the domain of the content corresponding to the captured image is a domain related to products or a domain related to stores, depending on whether the content corresponding to the captured image is content related to e-commerce or content related to gourmet food. For example, if the content corresponding to the captured image is content related to e-commerce, the information processing system 1 may estimate that the domain of the content is a domain related to products. For example, if the content corresponding to the captured image is content related to gourmet food, the information processing system 1 may estimate that the domain of the content is a domain related to stores. Note that the above is merely an example, and the information processing system 1 may appropriately use various information to estimate the domain of the content.

As described above, when providing information on a subject corresponding to a captured image, the information processing system 1 replaces the subject with a domain to which the user is directed and suggests it to the user. In this way, the information processing system 1 provides the user with content from a specified domain related to the subject of the user's interest. For example, if the domain corresponding to the captured image is not the specified domain, the information processing system 1 provides content from the specified domain.

In addition, the information processing system 1 performs domain estimation using a domain estimation model. For example, the information processing system 1 generates a query using a different query estimation model for each domain, and provides content that is a search result in a specific domain. For example, the information processing system 1 may use a query extraction model that has learned where the query is located in an image. For example, the information processing system 1 provides information on a specific domain when a specific condition is satisfied. If there is a specific domain that provides the subject of interest at a lower price than the estimated domain, the information processing system 1 provides the user with content that indicates the subject of interest in the specific domain.

[1-2-3. Example of information provided through map display]
Next, an example of information provision by map display regarding a subject of interest of a user in the information processing system 1 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of information processing executed by the information processing system. In Fig. 4, the processing will be described using a store called Cafe XXXX (also referred to as "target store CX") as an example of a subject of interest, but information provision by map display may be similarly provided not only for facilities such as stores, but also for trading objects such as goods.

In FIG. 4, map content is displayed by app AP12, which is an application installed on the terminal device 10. In the example shown in FIG. 4, the information processing system 1 shows a case where, when the location of user U1 is close to the target store CX, information to be provided INF11 related to Cafe XX○○ store (target store CX) is superimposed on map content MP1 and displayed. For example, app AP12 is a map application installed on the terminal device 10 used by the user. The terminal device 10 displays various information related to app AP12.

For example, the information processing system 1 generates a capture image (also called a "capture image C11") that is an image capturing content showing a target store CX, which is a store, by the user U1 operating the terminal device 10. For example, the terminal device 10 generates a capture image C11 of the target store CX in the SNS service #1 in response to the user U1's capture operation on the content related to the target store CX posted by another user on the SNS service #1. In FIG. 4, when the user U1 performs an operation to register the capture image C11 as a favorite, the information processing system 1 uses the capture image C11 to estimate the target of interest of the user U1. For example, the information processing device 100 estimates the target of interest of the user U1 based on the result of an analysis process that analyzes the capture image C1. The information processing device 100 estimates the target of interest of the user U1 based on the object included in the capture image C11. In this case, the information processing device 100 estimates the target store CX, which is a store included in the capture image C1, as the target of interest of the user U1.

The information processing system 1 provides the user with map content that displays information indicating the location of interest, which is a location corresponding to the target of interest indicated by the target of interest information, and the target of interest superimposed on a map. In FIG. 4, the information processing system 1 shows a case in which, when the user U1 is located within a predetermined range from the target store CX, the information to be provided INF11 related to the target store CX is superimposed on the map content MP1 and displayed. For example, the information processing system 1 compares the location information of the user U1 with the address of the target store CX, and, when the user U1 is located within a predetermined range from the target store CX, the information to be provided INF11 related to the target store CX is superimposed on the map content MP1 and displayed on the terminal device 10.

The map content MP1 in FIG. 4 is a content in which the provision information INF11, which is located at a position corresponding to the target of interest of the user U1 included in the range to be displayed, is superimposed on the map. For example, the map content MP1 is a content in which the provision information INF11 in the form of a pin icon indicating the location of the target store CX that is the target of interest of the user U1 is superimposed on a map showing a predetermined range from the location where the user U1 is located. The provision information INF11 includes an image IM1 indicating the target store CX that is the target of interest. The provision information INF11 also includes text information TX1, "Cafe XX XX store," indicating the target store CX that is the target of interest. The provision information INF11 also includes an icon IC1 indicating that the domain corresponding to the capture image C11 used to estimate that the user U1 is interested in the target store CX is SNS service #1.

In FIG. 4, the information processing system 1 provides the user U1 with map content MP1 in which the information to be provided INF11 is superimposed on a map. The information processing system 1 causes the terminal device 10 used by the user U1 to display the map content MP1 in which the information to be provided INF11 is superimposed on a map. The information processing system 1 causes the terminal device 10 to display map content MP1 in which the information to be provided INF11 located at a position corresponding to the subject of interest of the user U1 included in the range to be displayed is superimposed on the map. For example, the information processing system 1 displays map content MP1 in which the information to be provided INF11 in the form of a pin icon indicating the location of the target store CX that is the subject of interest of the user U1 is superimposed on a map showing a predetermined range from the location where the user U1 is located.

For example, the information processing system 1 causes the terminal device 10 to display map content MP1 in which provision information INF11 including an image IM1 indicating a target store CX that is an object of interest is superimposed on a map. Also, for example, the information processing system 1 causes the terminal device 10 to display map content MP1 in which provision information INF11 including text information TX1 "Cafe XX XX store" indicating a target store CX that is an object of interest is superimposed on a map. Also, for example, the information processing system 1 causes the terminal device 10 to display map content MP1 in which provision information INF11 including an icon IC1 indicating that a capture image C11 used to estimate that the object of interest of user U1 is the target store CX is a capture image of SNS service #1 is superimposed on a map.

In this way, the information processing system 1 can provide the user with appropriate information by superimposing information on the user's estimated subjects of interest on a map. As described above, the information processing system 1 may display the user's subjects of interest on a map as reminder subjects. For example, the information processing system 1 acquires subject information indicating the user's subjects of interest, identifies a position related to the subject indicated by the subject information, and when the user displays a map, superimposes and displays content indicating the identified position and subject.

For example, when the object of interest is a store, the information processing system 1 displays the store location. When the object of interest is a product, the information processing system 1 displays the store location that sells the product. When the object of interest is a product, the information processing system 1 displays the location of the nearest store that sells the object of interest. When the object of interest is a product, the information processing system 1 displays an image of the product, rather than an image of the store, superimposed on the store location.

As described above, the information processing system 1 may provide a reminder of a store corresponding to information posted on an SNS service. For example, the information processing system 1 obtains a capture image of the posted information on an SNS, analyzes the capture image, identifies a store indicated by the posted image, and provides information of the identified store to a user. For example, the information processing system 1 may analyze a capture image of a post on an SNS service, and provide information indicating a store such as a cafe identified by the analysis to a user. For example, the information processing system 1 may provide information to a user using environmental information such as weather and congestion. For example, the information processing system 1 may provide information to a user when a predetermined condition is met, such as when it is sunny, when it is unoccupied, when the user is in the vicinity, etc. For example, the information processing system 1 may estimate that a facility near the user's location when the user performs a predetermined operation, such as specifying a button to stock (register), is an object of interest.

[1-2-4. Examples of information provided through messages]
Next, an example of information provision by messages related to the user's objects of interest in the information processing system 1 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of information processing executed by the information processing system. In Fig. 5, processing will be described using stores such as Izakaya BA Shibuya (also referred to as "target store CY") and Izakaya BB Shibuya (also referred to as "target store CZ") as examples of objects of interest, but information provision by messages may be similarly provided not only for facilities such as stores, but also for trading objects such as goods.

The terminal device 10 in FIG. 5 shows an example of a chronological display of message exchanges in group GP1, which includes three users U1, U2, and U3. The terminal device 10 in FIG. 5 shows an example of a display on the terminal device 10 when user U1 inputs a message MS1, "Shibuya pub...", and the information processing system 1 provides a message MS2 and information to be provided INF21, INF22, etc. Note that the message MS1 may be any information as long as it is possible to identify a request for information on pubs in Shibuya. For example, the message MS1 may be a directive text such as "Tell me about pubs in Shibuya", or a command to the information processing system 1 requesting information to be provided, such as "(command) Shibuya pub". The display of the message exchanges in group GP1 shown in FIG. 5 is similar to that of a general group chat, so a detailed description will be omitted.

In FIG. 5, app AP13, which is an application installed on terminal device 10, displays various information such as messages entered by each user and information on subjects of interest (information to be provided). In the example shown in FIG. 5, when a user such as user U1 enters a message on a subject of interest, the information processing system 1 displays the information to be provided together with the message. For example, app AP13 is a messenger application installed on terminal device 10 used by the user. The terminal device 10 displays various information related to app AP13.

For example, the information processing system 1 generates a capture image (also referred to as "capture image C21") that is an image capturing content showing a target store CY, which is a store, by user U1 operating the terminal device 10. In FIG. 5, when user U1 performs an operation to register capture image C21 as a favorite, the information processing system 1 uses the capture image C21 to estimate the target of interest of user U1. For example, the information processing device 100 estimates the target of interest of user U1 based on the results of an analysis process that analyzes the capture image C1. The information processing device 100 estimates the target of interest of user U1 based on the object included in the capture image C21. In this case, the information processing device 100 estimates the target store CY, which is a store included in the capture image C1, as the target of interest of user U1.

In addition, the information processing system 1 generates a capture image (also called "capture image C22") that is an image capturing content indicating the target store CZ, which is a store, by user U2 operating the terminal device 10. In FIG. 5, when user U2 performs an operation to register the capture image C22 as a favorite, the information processing system 1 estimates that the target store CZ, which is a store included in the capture image C22, is a target of interest for user U2.

The information processing system 1 provides information on the subject of interest displayed together with the message in the app AP13 based on the subject of interest information and the message information. For example, when the message in the app AP13 corresponds to a subject of interest of the user, the information processing system 1 provides information on the subject of interest. In FIG. 5, the information processing system 1 provides, as information to be provided, information on the subject of interest that corresponds to the message "Izakaya in Shibuya" among the subjects of interest of users U1, U2, and U3 included in group GP1 including user U1.

The information processing system 1 acquires information indicating the interests of user U1, information indicating the interests of user U2, and information indicating the interests of user U3, and determines the interests corresponding to "Shibuya" and "izakaya" among the interests of users U1, U2, and U3 as the targets for providing information (also referred to as "information targets"). In FIG. 5, the information processing system 1 determines the target store CY that is the interest of user U1 and the target store CZ that is the interest of user U2 as the targets for providing information to group GP1 including users U1, U2, and U3. The information processing system 1 provides information on the targets determined as the information targets to each of users U1, U2, and U3.

In FIG. 5, the information processing system 1 provides each of users U1, U2, and U3 with a message MS2 saying "It looks like there's a good store nearby!", information to be provided INF21 for the target store CY, and information to be provided INF22 for the target store CZ. Following the message MS1 input by user U1, the information processing system 1 causes the terminal device 10 to display the message MS2, information to be provided INF21 for the target store CY, and information to be provided INF22 for the target store CZ.

In this way, the information processing system 1 can provide appropriate information to the user by providing the user with information on the estimated subjects of interest for each user who uses the message app. As described above, the information processing system 1 may display the user's subjects of interest together with the message. For example, the information processing system 1 obtains subject information indicating the subjects of interest of the user, identifies subjects among the subjects of interest of the user that are related to the message posted in the message app, and displays information indicating the identified subjects in the message app following the message.

For example, when a user posts a message, the information processing system 1 may select from the user's interests or from the interests of other users. For example, when a message from another user is directed to a certain user, the information processing system 1 may display the interests of the user. For example, the information processing system 1 may display information indicating interests of each user belonging to the message group that are related to the message. For example, the information processing system 1 may determine whether the message is a question for everyone, and if so, may determine the information provision target from the targets of everyone. For example, when the interest is a store, the information processing system 1 may provide information to be provided in which a link to a store reservation page is set. For example, when the interest is a product, the information processing system 1 may provide information to be provided in which a link to a product purchase page is set.

[1-3. Summary]
As described above, when a user performs an operation to instruct generation of a capture image, the information processing system 1 captures information displayed on the screen of the terminal device 10. The information processing system 1 sequentially provides the captured information to the user and accepts an operation by the user as to whether or not registration is required. For example, the information processing system 1 accepts an operation of swiping an image in one of the left and right directions (e.g., right) as an operation requiring registration, and accepts an operation of swiping an image in the other of the left and right directions (e.g., left) as an operation not requiring registration.

The information processing system 1 performs image analysis and estimates the subject of interest (subject of interest) of the user at the time of capture. For example, the information processing system 1 estimates the subject of interest, which is the subject of interest about which the user wants to be reminded.

The information processing system 1 estimates the domain of the image contained in the captured image using a model that has learned the screen characteristics of each domain. For each estimated domain, the information processing system 1 estimates the portion of the image that contains important information. For example, the information processing system 1 estimates a title range, an image range, a description range, etc. For example, the information processing system 1 performs character analysis for each range using OCR (Optical Character Recognition) or the like, and generates a query. For example, the information processing system 1 generates any query, such as an image query, a text query, etc.

The information processing system 1 uses the query to estimate corresponding products from products offered in other domains (redirection domains) that correspond to the estimated domain. For example, the information processing system 1 identifies products that are of interest to the user and are sold in a domain different from the domain being viewed at the time of capture.

The information processing system 1 provides the identified subject of interest to the user in accordance with the category to which the subject belongs. For example, in the case of food or merchandise, the information processing system 1 provides information on a page selling the subject at the destination domain. For example, in the case of providing information at the destination domain during an event, the information processing system 1 may automatically post content including a link to the destination domain when the name or category of the product is entered in a message app.

When the object of interest is a store, the information processing system 1 may continue to display the location of the store on the map. For example, the information processing system 1 may display a speech bubble on the map and superimpose the captured image.

The information used by the information processing system 1 to estimate the target of interest is not limited to the captured image. The information processing system 1 may sense various kinds of music and information around the user, and use the sensed information to estimate the target of interest of the user. For example, the information processing system 1 may analyze the sensed audio information and estimate the target of interest of the user based on the sound contained in the audio information. In this case, if the sound contained in the audio information includes a product name, the information processing system 1 may estimate that the product is the target of interest of the user. If the sound contained in the audio information includes a store name, the information processing system 1 may estimate that the store is the target of interest of the user. In addition, if the sound contained in the audio information is a song, the information processing system 1 may estimate that the song or the artist corresponding to the song is the target of interest of the user. For example, the information processing system 1 may estimate a facility of interest such as a store that is the target of interest based on the user's position information and traveling direction. In this case, the information processing system 1 may estimate that a facility such as a store located in the traveling direction from the user's position is the target of interest of the user. The information processing system 1 may estimate that a transaction object sold in a store located in the traveling direction from the user's position is an object of interest to the user. For example, the information processing system 1 may estimate the object of interest to the user based on a captured image of the object of purchase at the time of purchase. By performing the various processes described above, the information processing device 100 can provide appropriate information.

2. Configuration of Information Processing System
As shown in Fig. 6, the information processing system 1 includes a terminal device 10, an external device 20, and an information processing device 100. The multiple terminal devices 10, the external devices 20, and the information processing device 100 are connected to each other so as to be able to communicate with each other via a network N in a wired or wireless manner. Note that the information processing system 1 shown in Fig. 6 may include multiple external devices 20 and multiple information processing devices 100.

The terminal device 10 according to the embodiment is a device (mobile terminal) such as a smartphone used by a user. The terminal device 10 is an information processing device used by a user to access content such as a web page displayed in a browser or content for an application. For example, the terminal device 10 may be a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a mobile phone, a PDA (Personal Digital Assistant), or the like. Note that the terminal device 10 is not limited to the above examples, and may be, for example, a smart watch or a wearable device.

The terminal device 10 accepts various operations from the user. For example, the terminal device 10 accepts various operations from the user via the display surface using a touch panel function. The terminal device 10 may also accept various operations from buttons provided on the terminal device 10, or a keyboard or mouse connected to the terminal device 10. The terminal device 10 collects various information by storing the various information in a storage unit, for example. For example, the terminal device 10 collects the user's behavior history and sensor information detected by a sensor. The terminal device 10 may transmit the collected various information to an external device such as the external device 20 or the information processing device 100. The terminal device 10 provides the user's behavior history, such as an image captured by the user's operation (captured image), to the information processing device 100.

The terminal device 10 displays various information. The terminal device 10 displays various information provided by the information processing device 100. The terminal device 10 displays various information using the application AP11. For example, the terminal device 10 accepts an operation indicating a user's evaluation of a captured image displayed by the application AP11. The terminal device 10 displays information related to a map using the application AP12. The terminal device 10 displays information related to a message using the application AP13.

The terminal device 10 displays information related to the user's subject of interest, estimated using information when the user performs a predetermined operation indicating the user's interest, at a predetermined timing. The terminal device 10 displays second content related to the user's subject of interest, estimated based on first content in which the user performed a predetermined operation indicating the user's interest. The terminal device 10 displays information related to the user's facility of interest, estimated based on content in which the user performed a predetermined operation indicating the user's interest.

The terminal device 10 displays map content that displays, on a map, information indicating the target of interest and a location of interest that is a position corresponding to the target of interest indicated by the target of interest information indicating the target of interest that is a target in which the user's interest has shown interest, superimposed on a map. The terminal device 10 displays information regarding the target of interest that is displayed together with the message in the messenger application, based on the target of interest information indicating the target of interest that is a target in which the user's interest has shown interest, and message information indicating the message in the messenger application.

The external device 20 according to the embodiment is an information processing device that provides various information, and is realized, for example, by a server device or a cloud system. For example, the external device 20 provides user information as the various information. As another example, the external device 20 provides information on the subject of a transaction in Internet shopping, an online shopping mall, a flea market site, an auction site, a travel or restaurant reservation site, a credit card contract site, a financial product provision site, etc.

The information processing device 100 according to the embodiment is an information processing device capable of communicating with various devices via a predetermined network N such as the Internet, and is realized, for example, by a server device or a cloud system. For example, the information processing device 100 is connected to various other devices via the network N so as to be able to communicate with them.

The information processing device 100 estimates a target of interest in which the user has shown interest, using information when the user performs a specific operation that indicates the user's interest, and provides the user with information related to the estimated target of interest at a specific timing. The information processing device 100 estimates a target of interest in which the user has shown interest, based on first content in which the user performed a specific operation that indicates the user's interest, and provides the user with second content related to the estimated target of interest. The information processing device 100 estimates a facility of interest in which the user has shown interest, based on content in which the user performed a specific operation that indicates the user's interest, and provides the user with information related to the estimated facility of interest.

The information processing device 100 acquires interest target information indicating an interest target, which is an object of interest shown by the user's interest, and provides the user with map content that displays, on a map, an interest location, which is a location corresponding to the interest target indicated by the acquired interest target information, and information indicating the interest target. The information processing device 100 provides the user with information regarding the interest target that is displayed together with a message in a messenger application, based on interest target information indicating an interest target, which is an object of interest shown by the user's interest, and message information indicating a message in a messenger application.

As described above, the information processing device 100 may function as a search device that searches for information such as content of a specific domain based on a query. For example, the information processing device 100 has a function as a search engine that executes a search process targeting trading objects traded in a specific domain. For example, the information processing device 100 has a database (also called a "trading object database") in which trading objects to be the subject of a search process using a query are indexed and stored, and executes a search process targeting information in the trading object database. For example, the information in the trading object database is stored in the memory unit 120 (see FIG. 6).

1 illustrates a case where the information processing device 100 also functions as a search device, that is, the information processing device 100 and the search device are integrated, but the information processing device 100 and the search device may be separate entities. In this case, the information processing system 1 (see FIG. 6) includes a search device that performs a search process targeting a query and provides a search service that provides search results. For example, the information processing device 100 transmits a query acquired using a query estimation model or the like to the search device, receives search results from the search device, and transmits content generated using the search results to the terminal device 10 used by the user. For example, when the external device 20 is a search device, the information processing device 100 transmits a query acquired using a query estimation model or the like to the external device 20, receives search results from the external device 20, and transmits content generated using the search results to the terminal device 10 used by the user.

The system configuration and the processing subject are not limited to the above-described information processing system 1, and may be in various forms. In the above-described example, the information processing device 100 is a server that estimates the user's interests and the terminal device 10 is a client. The information processing device 100 and the terminal device 10 are separate (separate devices). However, the information processing device 100 and the terminal device 10 may be integrated. For example, the terminal device 10 may function as an information processing device that estimates the user's interests. In this case, the information processing system 1 may have the terminal device 10 that also functions as an information processing device that estimates the user's interests and a server device that provides information used by the terminal device 10 for processing. For example, the server device transmits various information displayed in various applications such as the apps AP11, AP12, and AP13 to the terminal device 10. The terminal device 10 displays various information to provide the user with various information. The above-described system configuration is merely an example, and the information processing system 1 may have any device configuration as long as it is possible to provide the user with desired information.

3. Configuration of information processing device
Hereinafter, an example of the functional configuration of the above-mentioned information processing device 100 will be described. Fig. 6 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 6, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

(Communication unit 110)
The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from various other devices.

(Memory unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 also has a user information storage unit 121, a target information storage unit 122, a content storage unit 123, and a model information storage unit 124. The storage unit 120 stores various information used for providing information, including but not limited to the above.

For example, the storage unit 120 stores information (application programs, etc.) related to various applications such as app AP11, app AP12, and app AP13. For example, the storage unit 120 stores information (image information, etc.) about a map (e.g. map MP1, etc.) displayed in app AP12.

(User information storage unit 121)
The user information storage unit 121 stores various user information related to a user. Fig. 7 is a diagram showing an example of a user information storage unit according to an embodiment. In the example shown in Fig. 7, the user information storage unit 121 has items such as "user ID" and "user information". For example, the "user information" includes items such as "attribute information", "behavioral history", and "interesting subject".

"User ID" is an identifier that identifies a user. "Attribute information" is attribute information related to the attributes of a user associated with the "user ID." For example, attribute information includes the user's age, gender, telephone number, address, etc. Note that the above-mentioned attribute information is merely an example, and may include various information other than age and gender, such as information indicating demographic attributes such as occupation, and information indicating psychographic attributes such as interests and lifestyle.

"Behavior history" is the behavior history of a user associated with a "user ID." The behavior history includes an operation history including a capture image in which content displayed on the terminal device 10 is captured by the user's operation, and information in which the content corresponding to the capture image and the date and time when the capture image was generated are associated with the capture image. The behavior history may include various types of sensor information detected regarding the user. The behavior history may also include a history of location information detected by the terminal device 10.

The behavioral history also includes various information such as the purchase history of purchases made by the user and the browsing history of content viewed by the user. For example, the purchase history is information about the transaction object purchased by the user, the type of transaction object, the number of purchases of the transaction object, and information about the date and time when the transaction object was purchased. For example, the browsing history may be a history of content displayed on the terminal device 10 used by the user. For example, the browsing history is information about content viewed (displayed) by the user, the type of content, the number of times the content was displayed, and information about the date and time when the content was displayed.

"Target of interest" is a target (target of interest) in which a user associated with a "user ID" shows interest. For example, a list of targets (targets of interest) in which a user shows interest is stored in "target of interest". For example, the list of interests includes, for each target of interest of a user, information for identifying the target of interest, information regarding the date and time when the target is estimated to have become a target of interest for the user, etc.

For example, in FIG. 7, "U1" identified by the user ID has attribute information "CH1", behavioral history "PH1", and interest "WA1". Note that in the example shown in FIG. 7, the attribute information, etc. is expressed as an abstract code such as "CH1", but the attribute information, etc. may be a specific numerical value, a specific character string, a file format containing various information, etc.

The user information storage unit 121 may store various types of information according to the purpose, not limited to the above. The user information storage unit 121 may store, for each user, information indicating the number of times the information has been displayed for each object, for example, the number of times the information has been viewed by the user.

(Target information storage unit 122)
The target information storage unit 122 stores various types of target information related to various targets including trading objects such as goods and facilities such as stores. FIG. 8 is a diagram showing an example of a target information storage unit according to an embodiment. In the example shown in FIG. 8, the target information storage unit 122 includes items such as "target ID", "target", "target information", "category", and "corresponding location information". Note that in FIG. 8, in order to explain "category" and "corresponding location information", they are described as separate items from "target information", but category information and location information may be included in the target information.

"Target ID" is an identifier that identifies a target. "Target" indicates the target associated with the "Target ID." "Target information" is target information of the target identified by the "Target ID."

"Category" is information about the category to which the object identified by the "object ID" belongs. "Corresponding location information" is information about the product to which the object identified by the "object ID" belongs.

For example, in FIG. 8, "M1" identified by the object ID has an object "MA1". Note that in the example shown in FIG. 8, the object is expressed by an abstract code such as "MA1", but the object is information that can identify an object that may be an object of a user's interest. For example, the object is a specific character string (store, etc.) that indicates a facility such as a store. For example, the object is a specific character string (product name, etc.) that indicates a transaction object such as a product.

In addition, in FIG. 8, "M1" identified by the target ID has target information "MD1". Note that in the example shown in FIG. 8, the target information is expressed by an abstract code such as "MD1", but the target information may be in a file format that includes various information related to the target.

In addition, in Figure 8, "M1" identified by the target ID is in the category "CT1." Note that in the example shown in Figure 8, the category is expressed by an abstract code such as "CT1," but the category is information that indicates the category of the target.

In addition, in FIG. 8, "M1" identified by the object ID has corresponding location information "PD1". Note that in the example shown in FIG. 8, the corresponding location information is expressed as an abstract code such as "PD1", but the corresponding location information indicates location information corresponding to the object, such as information indicating latitude and longitude (latitude and longitude information) or an address such as "XX, Z town, Y city, X prefecture". For example, if the object is a facility such as a store, the corresponding location information indicates the location of the facility. Also, for example, if the object is a trading object such as a commodity, the corresponding location information indicates the location of the facility such as a store where the trading object is sold (provided).

The target information storage unit 122 may store various information according to the purpose, not limited to the above. For example, the target information storage unit 122 may store information such as the score or the number of times each target is displayed.

(Content storage unit 123)
The content storage unit 123 stores information related to content. Fig. 9 is a diagram showing an example of the content storage unit according to the embodiment. In the example shown in Fig. 9, the content storage unit 123 has items such as "content ID" and "content".

"Content ID" is an identifier that identifies the content. "Content" is information about the content associated with the "Content ID." Specifically, the content may indicate information about the contents of the content.

For example, the content is content related to a portal site. As another example, the content may be content related to a news site, an auction site, a weather forecast site, a shopping site, a finance (stock price) site, etc.

The content may also be related to a route search site, a map providing site, a travel site, a restaurant introduction site, a web blog site, a social networking site, etc.

For example, in FIG. 9, "CN1" identified by the content ID has content "CO1". Note that in the example shown in FIG. 9, the content is expressed by an abstract code such as "CO1", but the content may be a specific numerical value, a specific character string, or a file format containing various information.

The content storage unit 123 may store various types of information depending on the purpose, not limited to the above.

(Model information storage unit 124)
The model information storage unit 124 stores information about the model. For example, the model information storage unit 124 stores information (model data) about a trained model (model) trained (generated) by a learning process. The model information storage unit 124 stores data used in the learning (trained data) in association with the trained model (model). FIG. 10 is a diagram illustrating an example of a model information storage unit according to an embodiment. In the example illustrated in FIG. 10, the model information storage unit 124 includes items such as "model ID", "purpose", "model data", and "trained data". In the example of FIG. 10, the model information storage unit 124 stores data used in the learning (trained data) in association with the trained model (model).

"Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model data" indicates the data of the model. Figure 10 etc. shows an example in which conceptual information such as "MDT1" is stored in "model data", but in reality, various information that constitutes the model is included, such as information on the model configuration (network configuration) and information on parameters. For example, "model data" includes information including the nodes in each layer of the network, the functions employed by each node, the connection relationships between the nodes, and the connection coefficients set for the connections between the nodes.

"Learning data" refers to data used to train a trained model (model). In "Learning data", information indicating the data set used to train the corresponding model is stored. For example, "Learning data" associates data (input information) with the correct answer information (output information) corresponding to that data, and stores the data as learning data (also called "data for learning"). Figure 10 shows an example in which conceptual information such as "LDT1" is stored in "Learning data", but in reality, various information related to the data used to train the corresponding model, such as data (input information) and the correct answer information (output information) corresponding to that data, is included.

In FIG. 10, the model (model M1) identified by the model ID "M1" indicates that its use is "domain estimation." Model M1 is a model for estimating the domain of content. For example, model M1 indicates that it is a model that outputs information for classifying which domain content corresponding to input data belongs to (for example, a score corresponding to each domain). Also, it indicates that the model data of model M1 is model data MDT1. Also, it indicates that the learning data used to learn model M1 is learning data LDT1.

The model (model M11) identified by the model ID "M11" indicates that its application is "query estimation (domain A)". Model M11 is a model that accurately extracts information corresponding to domain A. For example, model M11 is a model that outputs information indicating the results of image analysis of an input image. Model M11 extracts and outputs information (e.g., the name of an object, etc.) used as a query in an image corresponding to content of domain A. For example, model M11 indicates that it is a model that outputs information used for a query extracted from an input image. Also, it indicates that the model data of model M11 is model data MDT11. Also, it indicates that the learning data used to learn model M11 is learning data LDT11.

The model (model M12) identified by the model ID "M12" indicates that its application is "query estimation (domain B)". Model M12 is a model that accurately extracts information corresponding to domain B. For example, model M12 is a model that outputs information indicating the results of image analysis of an input image. Model M12 extracts and outputs information (e.g., the name of an object, etc.) used as a query in an image corresponding to content of domain B. For example, model M12 indicates that it is a model that outputs information used for a query extracted from an input image. Also, it indicates that the model data of model M12 is model data MDT12. Also, it indicates that the learning data used to learn model M12 is learning data LDT12.

The model (model M13) identified by the model ID "M13" indicates that its application is "query estimation (domain C)". Model M13 is a model that accurately extracts information corresponding to domain C. For example, model M13 is a model that outputs information indicating the results of image analysis of an input image. Model M13 extracts and outputs information (e.g., the name of an object, etc.) used as a query in an image corresponding to content of domain C. For example, model M13 indicates that it is a model that outputs information used for a query extracted from an input image. Also, it indicates that the model data of model M13 is model data MDT13. Also, it indicates that the learning data used to learn model M13 is learning data LDT13.

Note that while Figure 10 illustrates models up to M13, the model information storage unit 124 may also store models that estimate (predict) information corresponding to various domains, such as model M14 targeting domain D.

The model information storage unit 124 may store various information according to the purpose, not limited to the above. For example, the model information storage unit 124 may describe a model (object recognition model) used to recognize objects, etc. included in an input image. For example, the object recognition model may output information indicating an object in the input image (object name, etc.) and information indicating the position of the object (coordinate information, etc.).

(Control unit 130)
The control unit 130 is a controller, and is realized, for example, by a central processing unit (CPU) or a micro processing unit (MPU) executing various programs (an example of an information processing program) stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

As shown in FIG. 6, the control unit 130 has an acquisition unit 131, a reception unit 132, a determination unit 133, an estimation unit 134, a generation unit 135, and a provision unit 136, and realizes or executes the functions and actions of the information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 6, and may be other configurations as long as they perform the information processing described below.

(Acquisition unit 131)
The acquisition unit 131 acquires various pieces of information. The acquisition unit 131 acquires various pieces of information from the storage unit 120. The acquisition unit 131 acquires various pieces of information from the user information storage unit 121, the target information storage unit 122, the content storage unit 123, the model information storage unit 124, etc.

The acquisition unit 131 receives various information from an external information processing device via the communication unit 110. The acquisition unit 131 receives various information from the terminal device 10 or the external device 20. For example, the acquisition unit 131 acquires user information related to the user from the external device 20.

The acquisition unit 131 acquires information when a user performs a predetermined operation that indicates the user's interest. The acquisition unit 131 acquires information indicating the content on which the predetermined operation that indicates the user's interest was performed. The acquisition unit 131 acquires a capture image of the content on which the predetermined operation that indicates the user's interest was performed. The acquisition unit 131 acquires a capture image generated by the user's operation.

The acquisition unit 131 acquires interest target information indicating an interest target that is an object in which the user's interest has shown interest. The acquisition unit 131 acquires message information indicating a message in a messenger application. The acquisition unit 131 acquires interest target information indicating an interest target that is a trading object or facility in which the user's interest has shown interest. The acquisition unit 131 acquires interest target information indicating an interest target that is a product or store in which the user's interest has shown interest.

The acquisition unit 131 acquires sensor information when the user performs a predetermined operation. The acquisition unit 131 acquires location information of the user when the user performs a predetermined operation.

The acquisition unit 131 may acquire, as user information, from the external device 20, user attribute information, purchase history on the Internet, online shopping, online shopping malls, auction sites, flea market sites, etc., and behavioral history such as content browsing history. The acquisition unit 131 then stores such user information in the user information storage unit 121. For example, the acquisition unit 131 acquires user information from the external device 20 at predetermined intervals and updates the user information stored in the user information storage unit 121.

The acquisition unit 131 may also acquire information about various objects such as trading objects and facilities from the external device 20. For example, the acquisition unit 131 acquires information about trading objects traded on an auction site or a flea market site from the external device 20. Then, the acquisition unit 131 stores the information about the trading objects in the object information storage unit 122. For example, the acquisition unit 131 acquires information about facilities such as stores from the external device 20. Then, the acquisition unit 131 stores the information about facilities such as stores in the object information storage unit 122.

(Reception unit 132)
The reception unit 132 receives various requests. The reception unit 132 receives various requests from an external information processing device. The reception unit 132 receives information indicating various requests from the external information processing device via the communication unit 110. For example, the reception unit 132 receives a request from the terminal device 10 or the external device 20. The reception unit 132 receives a request for information from the terminal device 10. For example, the reception unit 132 receives a request for information displayed in various applications such as apps AP11, AP12, and AP13 from the terminal device 10.

(Determination unit 133)
The determination unit 133 executes a determination process for determining various information. For example, the determination unit 133 executes the determination process based on various information acquired by the acquisition unit 131. The determination unit 133 executes the determination process based on various information stored in the storage unit 120. For example, the determination unit 133 executes the determination process based on various information received from an external information processing device. The determination unit 133 executes an extraction process for extracting various information. For example, the determination unit 133 executes the extraction process based on various information acquired by the acquisition unit 131. The determination unit 133 executes the extraction process based on various information stored in the storage unit 120. For example, the determination unit 133 executes the extraction process based on various information received from an external information processing device.

For example, the determination unit 133 executes the determination process based on the information estimated by the estimation unit 134. For example, the determination unit 133 executes the determination process based on the information generated by the generation unit 135.

The determination unit 133 determines the timing for providing information. The determination unit 133 determines the timing according to the subject of interest. When the subject of interest is a product, the determination unit 133 determines the timing for providing information to be the timing when the user becomes able to purchase the product. The determination unit 133 determines that it is the timing for providing information when the user becomes able to purchase the product. The determination unit 133 determines that it is the timing for providing information when the electronic money or points held by the user reach or exceed a predetermined value.

When the object of interest is a store, the determination unit 133 determines that the timing for providing information is the timing when the user approaches the store. When the user approaches the store, the determination unit 133 determines that it is the timing for providing information. When the user's location is within a predetermined range from the store, the determination unit 133 determines that it is the timing for providing information.

The determination unit 133 determines the domain for which information is to be provided. The determination unit 133 determines the second content for which information is to be provided. The determination unit 133 determines the second domain based on a predetermined criterion. The determination unit 133 determines a domain similar to the first domain of the first content as the second domain. The determination unit 133 determines the second content based on a predetermined criterion. The determination unit 133 determines content indicating a target corresponding to the target of the first content as the second content. The determination unit 133 determines content indicating the same product as the product of the first content as the second content. The determination unit 133 determines the second domain according to the user. The determination unit 133 determines a domain corresponding to a service currently being used by the user as the second domain.

The decision unit 133 decides whether to provide information on the subject of interest based on the message indicated by the message information and the subject of interest of the user. The decision unit 133 decides whether to provide information on the subject of interest based on a comparison between the message indicated by the message information and the subject of interest of the user. If the message indicated by the message information includes the subject of interest of the user, the decision unit 133 decides to provide information on the subject of interest. The decision unit 133 decides information on the subject of interest corresponding to the message by the user. The decision unit 133 decides information on the subject of interest corresponding to a message by a user other than the user among multiple users including the user.

(Estimation unit 134)
The estimation unit 134 executes an estimation process to estimate various pieces of information. For example, the estimation unit 134 executes the estimation process based on various pieces of information acquired by the acquisition unit 131. The estimation unit 134 executes the estimation process based on various pieces of information stored in the storage unit 120. The estimation unit 134 executes the estimation process based on various pieces of information received from an external information processing device.

For example, the estimation unit 134 executes the estimation process based on the information determined by the determination unit 133. For example, the estimation unit 134 executes the estimation process based on the information generated by the generation unit 135.

The estimation unit 134 estimates the target of interest of the user, which is a subject in which the user has shown an interest, by using information when the user performed a specific operation that indicates the user's interest. The estimation unit 134 estimates the target of interest of the user based on the content in which the specific operation that indicates the user's interest was performed. The estimation unit 134 estimates the target of interest of the user based on the subject included in the content.

The estimation unit 134 estimates the user's subject of interest using a capture image generated by the user's operation. The estimation unit 134 estimates the user's subject of interest based on the objects included in the capture image. The estimation unit 134 estimates the user's subject of interest based on the results of an analysis process that analyzes the capture image.

The estimation unit 134 estimates the user's interests based on the domain of the captured image. The estimation unit 134 estimates the user's interests based on the domain estimated using a domain estimation model that estimates the domain of the captured image. The estimation unit 134 estimates the domain of the captured image using a domain estimation model that takes the screen as input and outputs information indicating the domain of the input image.

The estimation unit 134 estimates the user's subject of interest using sensor information when the user performs a specified operation. The estimation unit 134 estimates the user's subject of interest using the user's location information when the user performs a specified operation. The estimation unit 134 estimates a transaction object or a facility as the user's subject of interest. The estimation unit 134 estimates a product or a store as the user's subject of interest.

The estimation unit 134 estimates facilities of interest that are facilities in which the user has shown an interest, based on content in which the user performed a specified operation that indicates the user's interest. The estimation unit 134 estimates facilities of interest that are stores in which the user has shown an interest. The estimation unit 134 estimates facilities of interest of the user, based on content in a specified service. The estimation unit 134 estimates facilities of interest of the user, based on content posted on SNS. The estimation unit 134 estimates facilities of interest of the user, based on content posted by users other than the user.

The estimation unit 134 estimates the facilities of interest of the user based on the objects included in the content. The estimation unit 134 estimates the facilities of interest of the user using a captured image generated by the user's operation. The estimation unit 134 estimates the facilities of interest of the user based on the objects included in the captured image. The estimation unit 134 estimates the facilities of interest of the user based on the results of an analysis process that analyzes the captured image.

The estimation unit 134 estimates the user's facilities of interest based on the domain of the captured image. The estimation unit 134 estimates the user's facilities of interest based on the domain estimated using a domain estimation model that estimates the domain of the captured image. The estimation unit 134 estimates the domain of the captured image using a domain estimation model that takes the screen as input and outputs information indicating the domain of the input image. The estimation unit 134 estimates the user's facilities of interest using sensor information when the user performs a specified operation. The estimation unit 134 estimates the user's facilities of interest using location information of the user when the user performs a specified operation.

The estimation unit 134 estimates a target of interest, which is a subject in which the user has shown an interest, based on the first content in which the user performed a predetermined operation indicating the user's interest. The estimation unit 134 estimates the target of interest of the user based on the subject included in the first content. The estimation unit 134 estimates the target of interest of the user using a capture image of the first content generated by the user's operation.

(Generation unit 135)
The generating unit 135 executes a generating process to estimate various pieces of information. For example, the generating unit 135 executes the generating process based on various pieces of information acquired by the acquiring unit 131. The generating unit 135 executes the generating process based on various pieces of information stored in the storage unit 120. The generating unit 135 executes the generating process based on various pieces of information received from an external information processing device.

For example, the generation unit 135 executes the generation process based on the information determined by the determination unit 133. For example, the generation unit 135 executes the generation process based on the information generated by the generation unit 135.

The generation unit 135 generates information to be provided to the user. The generation unit 135 generates map content in which information indicating the target of interest and a position of interest, which is a position corresponding to the target of interest indicated by the target of interest information, is superimposed on a map. The generation unit 135 generates map content including an icon indicating the position of interest. The generation unit 135 generates map content including an image indicating the target of interest. The generation unit 135 generates map content including text information indicating the target of interest.

The generation unit 135 generates map content including information indicating the subject of interest estimated by the estimation unit 134. The generation unit 135 generates map content in which information indicating a position of interest corresponding to the subject of interest, which is a trading object or a facility, is superimposed on a map. The generation unit 135 generates map content in which information indicating a position of interest corresponding to the subject of interest, which is a product or a store, is superimposed on a map.

The generation unit 135 generates information about a transaction object or facility that is displayed together with a message in the messenger application. The generation unit 135 generates information about a product or store that is displayed together with a message in the messenger application.

The generating unit 135 generates various information such as content (map content) to be provided to an external information processing device by appropriately using various techniques. The generating unit 135 generates information such as content to be provided to the terminal device 10. For example, the generating unit 135 generates information such as content to be provided to the terminal device 10 based on information stored in the storage unit 120. The generating unit 135 may generate information such as content by any process as long as it is possible to generate information such as content to be provided to an external information processing device. For example, the generating unit 135 generates information such as content to be provided to the terminal device 10 by appropriately using various techniques related to image generation and image processing. For example, the generating unit 135 generates information such as content to be provided to the terminal device 10 by appropriately using various techniques such as Java (registered trademark). The generating unit 135 may generate information such as content to be provided to the terminal device 10 based on the format of CSS, JavaScript (registered trademark), or HTML. Furthermore, for example, the generation unit 135 may generate information such as content in various formats, such as JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), and PNG (Portable Network Graphics).

The generation unit 135 generates various types of information to be provided to the user. The generation unit 135 generates content such as map content MP1. The generation unit 135 generates various types of information such as information to be provided INF11, INF21, INF22, etc.

The generation unit 135 may also function as a learning unit that executes a learning process to learn a learning model (model). In this case, the generation unit 135 executes the learning process based on various information acquired by the acquisition unit 131. The generation unit 135 executes the learning process based on information from an external information processing device and information stored in the storage unit 120. The generation unit 135 executes the learning process based on information stored in the model information storage unit 124. The generation unit 135 stores the model generated by learning in the model information storage unit 124.

The generation unit 135 learns a domain estimation model for estimating the domain of the content. The generation unit 135 learns a query estimation model for estimating (extracting) information to be used as a query in the content.

The generation unit 135 performs a learning process. The generation unit 135 performs various learning. The generation unit 135 learns various information based on the information acquired by the acquisition unit 131. The generation unit 135 learns (generates) a model. The generation unit 135 learns various information such as a model. The generation unit 135 generates a model through learning. The generation unit 135 learns the model using various machine learning techniques. For example, the generation unit 135 learns the parameters of the model (network). The generation unit 135 learns the model using various machine learning techniques.

The generation unit 135 generates various learning models such as models M1, M11, M12, and M13. The generation unit 135 learns network parameters. For example, the generation unit 135 learns network parameters of various learning models such as models M1, M11, M12, and M13. The generation unit 135 performs a learning process using the learning data stored in the model information storage unit 124 to generate various learning models such as models M1, M11, M12, and M13. For example, the generation unit 135 generates a model used for voice recognition. The generation unit 135 generates various learning models such as models M1, M11, M12, and M13 by learning network parameters of various learning models such as models M1, M11, M12, and M13.

The generation unit 135 performs a learning process based on the learning data (teacher data) stored in the model information storage unit 124. The generation unit 135 performs a learning process using the learning data stored in the model information storage unit 124, thereby generating various learning models such as models M1, M11, M12, and M13.

For example, the generation unit 135 generates a model used to estimate the probability (likelihood) of a store closing. For example, when store information of a store is input, the generation unit 135 generates a model that outputs a score indicating the probability (likelihood) that a store corresponding to the store information will close from a trading site. The generation unit 135 generates model M1 by learning the parameters of the network of model M1.

The method of learning by the generation unit 135 is not particularly limited, but for example, learning data that links data (input information) with its correct answer information (output information) may be prepared, and the learning data may be input to a computational model based on a multilayer neural network for learning.

For example, when learning a model M1, the generation unit 135 learns the model M1 using the learning data LDT1 stored in the model information storage unit 124. For example, the learning data LDT1 includes data that associates a captured image of a content with information (correct answer information) indicating the domain to which the content applies.

For example, when the generation unit 135 inputs a captured image to the model M1, the learning process is performed so that the model M1 outputs the domain to which the captured image is associated. When the generation unit 135 inputs a captured image C1 to the model M1, the learning process is performed so that, among the scores of each domain output by the model M1, the score of the domain to which the captured image C1 is associated (e.g., domain A) becomes large. When the generation unit 135 inputs a captured image C1 to the model M1, the learning process is performed so that, among the scores of each domain output by the model M1, the scores of domains other than domain A become small. For example, when the generation unit 135 inputs a captured image C1 to the model M1, the learning process is performed so that the score of domain A output by the model M1 approaches 1. For example, when the generation unit 135 inputs a captured image C1 to the model M1, the learning process is performed so that the scores of domains other than domain A output by the model M1 approach 0.

The generation unit 135 may also learn a query estimation model corresponding to each domain, such as models M11, M12, and M13. For example, when learning model M11, the generation unit 135 learns model M11 using learning data LDT11 stored in the model information storage unit 124. For example, the learning data LDT11 includes data that associates a capture image of content corresponding to domain A with information (correct answer information) indicating an object in the capture image. The correct answer information is information indicating an object used in a query from a capture image. The correct answer information is information regarding a character string indicating a transaction object such as a product name, or a character string indicating a facility such as a store name. For example, the correct answer information may be a character string to be used as a query in a capture image and an area in which the character string is located.

For example, when a capture image is input to the model M11, the generation unit 135 performs a learning process using an arbitrary learning method so that the information output by the model M11 becomes correct answer information (label) associated with the capture image. For example, the generation unit 135 learns a model M11 adapted to the domain A by learning using learning data LDT11 in which a capture image of content corresponding to the domain A and the correct answer information are associated with the capture image.

For example, when a captured image of content corresponding to domain A is input to model M11, generation unit 135 performs a learning process so that model M11 outputs information indicating the target indicated by the correct answer information associated with the captured image. Note that any learning process can be adopted for the learning process of model M11 as long as it is possible to learn a model that extracts desired information from an image, and it may be similar to conventional learning such as image recognition and optical character recognition (OCR), and a detailed description thereof will be omitted.

The generation unit 135 may also learn models M12, M13, etc. in the same manner as model M11. For example, when learning model M12, the generation unit 135 learns model M12 using learning data LDT12 stored in the model information storage unit 124. For example, the learning data LDT12 includes data that associates a capture image of content corresponding to domain B with information (correct answer information) that indicates an object in the capture image. For example, the generation unit 135 learns model M12 that is suitable for domain B by learning using learning data LDT12 in which a capture image of content corresponding to domain B and its correct answer information are associated.

For example, when learning a model M13, the generation unit 135 learns the model M13 using the learning data LDT13 stored in the model information storage unit 124. For example, the learning data LDT13 includes data that associates a capture image of content corresponding to domain C with information (correct answer information) that indicates an object in the capture image. For example, the generation unit 135 learns a model M13 that is suitable for domain C by learning using the learning data LDT13 that associates a capture image of content corresponding to domain C with its correct answer information.

The learning method of each model is not limited to the above-mentioned method, and any known technology can be applied. Each model may be generated using various conventional technologies related to machine learning as appropriate. For example, the model may be generated using a machine learning technology related to supervised learning such as SVM (Support Vector Machine). Also, for example, the model may be generated using a machine learning technology related to unsupervised learning. For example, the model may be generated using a deep learning technology. For example, the model may be generated using various deep learning technologies such as DNN (Deep Neural Network), RNN (Recurrent Neural Network), and CNN (Convolutional Neural Network). Note that the description regarding the generation of the above models is an example, and the model may be generated by a learning method selected appropriately according to the obtainable information, etc. For example, the generation unit 135 may use any method to generate models M1, M11, M12, M13, etc., as long as the generation unit 135 can learn models M1, M11, M12, M13, etc. so as to output information corresponding to correct answer information when an image included in the training data is input.

(Providing Unit 136)
The providing unit 136 provides various information. The providing unit 136 transmits various information to an external information processing device via the communication unit 110. The providing unit 136 transmits various information to the terminal device 10 or the external device 20. The providing unit 136 transmits content to the terminal device 10. The providing unit 136 transmits information related to the object of transaction to the terminal device 10.

The providing unit 136 provides the user with information on the subject of interest estimated by the estimation unit 134 at a predetermined timing. The providing unit 136 provides the information in accordance with the decision by the determination unit 133. The providing unit 136 provides the information when the decision unit 133 decides to provide the information. The providing unit 136 provides the information at the timing determined by the decision unit 133.

The providing unit 136 provides information about the subject of interest at a timing according to the subject of interest. If the subject of interest is a product, the providing unit 136 provides information about the subject of interest at a timing when the user can purchase the product. If the subject of interest is a store, the providing unit 136 provides information about the subject of interest at a timing when the user approaches the store. The providing unit 136 displays information about the subject of interest on the terminal device 10 used by the user. The providing unit 136 transmits information about the subject of interest to the terminal device 10 used by the user.

The providing unit 136 provides the user with information about the facility of interest estimated by the estimation unit 134. The providing unit 136 causes the terminal device 10 used by the user to display the information about the facility of interest. The providing unit 136 transmits the information about the facility of interest to the terminal device 10 used by the user.

The providing unit 136 provides the user with second content related to the subject of interest estimated by the estimation unit 134. The providing unit 136 provides the second content of a predetermined domain. The providing unit 136 provides the second content of a second domain, which is a predetermined domain different from the first domain of the first content. The providing unit 136 provides the second content of the second domain determined based on a predetermined criterion. The providing unit 136 provides the second content of the second domain determined according to the user. The providing unit 136 displays the second content on the terminal device 10 used by the user. The providing unit 136 transmits the second content to the terminal device 10 used by the user.

The providing unit 136 provides the map content generated by the generating unit 135. The providing unit 136 provides the user with map content that displays, on a map, information indicating the location of interest, which is a position corresponding to the target of interest indicated by the target of interest information, and the target of interest. The providing unit 136 provides the map content including an icon indicating the location of interest. The providing unit 136 provides the map content including an image indicating the target of interest. The providing unit 136 provides the map content including text information indicating the target of interest.

The providing unit 136 provides map content including information indicating the subject of interest estimated by the estimation unit 134. The providing unit 136 provides map content that displays information indicating a position of interest corresponding to the subject of interest, which is a transaction object or a facility, superimposed on a map. The providing unit 136 provides map content that displays information indicating a position of interest corresponding to the subject of interest, which is a product or a store, superimposed on a map. The providing unit 136 displays the map content on the terminal device 10 used by the user. The providing unit 136 transmits the map content to the terminal device 10 used by the user.

When the determination unit 133 determines to provide information on the subject of interest, the provision unit 136 provides the user with information on the subject of interest that is displayed together with the message in the messenger application. The provision unit 136 provides the user with information on the subject of interest that is displayed together with the message in the messenger application based on the subject of interest information and the message information.

The providing unit 136 provides information on the transaction object or facility generated by the generating unit 135. The providing unit 136 provides information on the product or store generated by the generating unit 135. The providing unit 136 provides information on the subject of interest corresponding to the message indicated by the message information. The providing unit 136 provides information on the subject of interest corresponding to the message by the user. The providing unit 136 provides information on the subject of interest corresponding to the message by the user other than the user among multiple users including the user.

The providing unit 136 provides information on a transaction object or facility that is displayed together with a message in a messenger application. The providing unit 136 provides information on a product or store that is displayed together with a message in a messenger application. The providing unit 136 causes information on the object of interest to be displayed together with the message on a terminal device used by the user. The providing unit 136 transmits information on the object of interest that is displayed together with the message to the terminal device used by the user.

The providing unit 136 provides the information generated by the generating unit 135. The providing unit 136 provides content including the information generated by the generating unit 135. The providing unit 136 transmits the content to the terminal device 10 used by the user. The providing unit 136 transmits information to be displayed in the app AP11 to the terminal device 10.

The providing unit 136 transmits information to be displayed in the app AP12 to the terminal device 10. The providing unit 136 transmits map content MP1 to the terminal device 10. The providing unit 136 transmits map content MP1, on which information about Cafe XX, which is an object of the user's interest, is superimposed, to the terminal device 10 used by the user. The providing unit 136 transmits map content MP1, on which content CO11 including an image and character string of Cafe XX is superimposed, to the terminal device 10 used by the user.

The providing unit 136 transmits information to be displayed in the app AP13 to the terminal device 10. The providing unit 136 transmits content CO21, CO22, etc. to be displayed together with the message MS1, etc. in the app AP13 to the terminal device 10. The providing unit 136 transmits content CO21, CO22, etc. including information about a pub in Shibuya that is the subject of the user's interest to the terminal device 10 used by the user.

4. Processing Procedure
Next, the procedure of information processing executed by the information processing device 100 according to the embodiment will be described with reference to Fig. 11 to Fig. 15. Fig. 11 to Fig. 15 are flowcharts showing an example of the flow of information processing. For example, Fig. 11 to Fig. 15 are flowcharts showing an example of the flow of information processing executed by the information processing device.

First, FIG. 11 will be described. For example, FIG. 11 shows an example of a processing flow for providing information based on an estimation of a user's interests performed by the information processing device 100.

In FIG. 11, the information processing device 100 estimates a subject of interest in which the user has shown an interest, using information when the user performs a predetermined operation that indicates the user's interest (step S101). Then, the information processing device 100 provides the user with information about the subject of interest at a predetermined timing (step S102).

Next, FIG. 12 will be described. For example, FIG. 12 shows an example of a processing flow for providing second content in response to an estimation result based on the first content performed by the information processing device 100.

In FIG. 12, the information processing device 100 estimates a subject of interest in which the user has shown an interest based on a first content in which the user has performed a predetermined operation indicating the user's interest (step S201). Then, the information processing device 100 provides the user with a second content related to the subject of interest (step S202).

Next, FIG. 13 will be described. For example, FIG. 13 shows an example of a processing flow for providing information based on an estimation of facilities of interest to a user, which is performed by the information processing device 100.

In FIG. 13, the information processing device 100 estimates facilities of interest that are facilities in which the user has shown an interest, based on content in which the user has performed a predetermined operation that indicates the user's interest (step S301). Then, the information processing device 100 provides the user with information about the facilities of interest (step S302).

Next, FIG. 14 will be described. For example, FIG. 14 shows an example of a processing flow for providing map content based on a user's interests performed by the information processing device 100.

In FIG. 14, the information processing device 100 acquires information on a target of interest indicating a target of interest in which the user has shown an interest (step S401). The information processing device 100 then provides the user with map content that displays, on a map, information on the target of interest and a location of interest that is a position corresponding to the target of interest indicated by the target of interest information (step S402).

Next, FIG. 15 will be described. For example, FIG. 15 shows an example of a processing flow for providing information content based on a user's interests and messages in a messenger application, which is performed by the information processing device 100.

In FIG. 15, the information processing device 100 acquires interest subject information indicating an interest subject that is an object in which the user has shown an interest, and message information indicating a message in a messenger application (step S501). Then, the information processing device 100 provides the user with information on the interest subject that is displayed together with the message in the messenger application based on the interest subject information and the message information (step S502).

5. Modifications
The above-described information processing device 100 may be implemented in various different forms other than the above embodiment. Therefore, other embodiments of the information processing device 100 will be described below.

[5-1. user〕
The user to whom the information is to be provided may be any user whose interests can be estimated. For example, the user to whom the information is to be provided may be a member of a specific service. The user may be a registered user, or may be a user who is not registered as a member of a specific service.

[5-2. Target]
In the above example, products and stores are described as examples of the user's interest, but the estimated interest may be any object, not limited to products and stores. For example, the interest may be a transaction object such as a service used by the user, or a facility such as a stadium that serves as an event venue. Note that the above is merely an example, and the interest may be any object as long as it is an object of interest to the user.

[5-3. Providing information on the subject of a transaction]
In the above embodiment, an example of information processing in which the information processing device 100 provides various information such as content to the terminal device 10 has been described, but the present invention is not limited to the above example. For example, an external server may provide various information such as content to the terminal device 10. In this case, the information processing device 100 may provide information related to the user's interests to the external server.

For example, the external server provides a service of sending reminders (notifications) of the user's interests. In this case, the information processing device 100 may provide the external server with information about the user's interests, such as an image of the interest or information specifying the timing for reminding the user of the interest.

5-4. Program
The information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 having a configuration as shown in Fig. 16. Fig. 16 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected by a bus 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), a HDD (Hard Disk Drive), a flash memory, or the like.

The output IF 1060 is an interface for transmitting information to be output to an output device 1010 such as a monitor or printer, which outputs various types of information, and is realized, for example, by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input IF 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, scanner, etc., and is realized, for example, by a USB.

The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

The network IF 1080 receives data from other devices via the network N and sends it to the computing device 1030, and also transmits data generated by the computing device 1030 to other devices via the network N.

The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040.

[5-5. Other]
Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

The above-described embodiments and variations can be combined as appropriate to the extent that they do not cause inconsistencies in the processing content.

In addition, the above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

6. Effects
As described above, the information processing device 100 according to the embodiment includes the acquisition unit 131 and the provision unit 136. The acquisition unit 131 acquires interest subject information indicating an interest subject that is an object in which the user has shown an interest, and message information indicating a message in a messenger application. The provision unit 136 provides the user with information on the interest subject displayed together with the message in the messenger application based on the interest subject information and the message information.

In this way, the information processing device 100 according to the embodiment can provide appropriate information to a user by providing information about the subject of interest that is displayed together with a message in a messenger application based on the subject of interest information and message information.

In addition, in the information processing device 100 according to the embodiment, the providing unit 136 provides information regarding the subject of interest corresponding to the message indicated by the message information.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information about the subject of interest that corresponds to the message indicated by the message information.

In addition, in the information processing device 100 according to the embodiment, the providing unit 136 provides information regarding the subject of interest corresponding to the message from the user.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information on the subject of interest corresponding to the message from the user.

In addition, in the information processing device 100 according to the embodiment, the providing unit 136 provides information on subjects of interest corresponding to messages from users other than the user among multiple users including the user.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information on subjects of interest corresponding to messages from users other than the user among multiple users including the user.

The information processing device 100 according to the embodiment also includes an estimation unit 134. The estimation unit 134 estimates a subject of interest in which the user has shown an interest, based on content in which the user has performed a predetermined operation indicating the user's interest. The provision unit 136 provides information related to the subject of interest estimated by the estimation unit 134.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information indicating the subject of interest that is estimated based on the content in which the user has performed a specific operation that indicates the user's interest.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest based on the subject included in the content.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests based on the subjects contained in the content.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest using a capture image generated by the user's operation.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests using a capture image generated by the user's operation.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest based on the object included in the captured image.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests based on the objects contained in the captured image.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest based on the results of an analysis process that analyzes the captured image.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests based on the results of the analysis process that analyzes the captured image.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest based on the domain of the captured image.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests based on the domain of the captured image.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest based on the domain estimated using a domain estimation model that estimates the domain of a captured image.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests based on the domain estimated using a domain estimation model that estimates the domain of a captured image.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the domain of the captured image using a domain estimation model that takes a screen as input and outputs information indicating the domain of the input image.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the domain of a captured image using a domain estimation model that takes a screen as input and outputs information indicating the domain of the input image.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest using sensor information obtained when the user performs a specific operation.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests using sensor information when the user performs a specific operation.

In addition, in the information processing device 100 according to the embodiment, the estimation unit 134 estimates the user's subject of interest using the user's location information when the user performs a predetermined operation.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by estimating the user's interests using the user's location information when the user performs a specific operation.

In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires interest target information indicating an interest target, which is a transaction target or facility in which the user has expressed an interest. The provision unit 136 provides information on the transaction target or facility that is displayed together with a message in the messenger application.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information about the transaction object or facility that is displayed together with the message in the messenger application.

In the information processing device 100 according to the embodiment, the acquisition unit 131 acquires interest target information indicating an interest target, which is a product or store in which the user has shown an interest. The provision unit 136 provides information on the product or store that is displayed together with a message in the messenger application.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by providing information about products or stores that are displayed together with messages in a messenger application.

In addition, in the information processing device 100 according to the embodiment, the providing unit 136 displays information related to the subject of interest together with a message on the terminal device used by the user.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by displaying information related to a subject of interest together with a message on the terminal device used by the user.

In addition, in the information processing device 100 according to the embodiment, the providing unit 136 transmits information related to the subject of interest displayed together with the message to the terminal device used by the user.

In this way, the information processing device 100 according to the embodiment can provide appropriate information by transmitting information about the subject of interest displayed together with the message to the terminal device used by the user.

Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the invention.

N network 1 information processing system 10 terminal device 20 external device 100 information processing device 110 communication unit 120 storage unit 121 user information storage unit 122 target information storage unit 123 content storage unit 124 model information storage unit 130 control unit 131 acquisition unit 132 reception unit 133 determination unit 134 estimation unit 135 generation unit 136 provision unit

Claims (20)

An acquisition unit that acquires interest subject information indicating an interest subject that is an object of interest of a user and message information indicating a message in a messenger application;
a providing unit that provides the user with information about the subject of interest to be displayed together with the message in the messenger application based on the subject of interest information and the message information;
An information processing device comprising: The providing unit is
The information processing device according to claim 1 , further comprising: providing information about the subject of interest corresponding to the message indicated by the message information. The providing unit is
The information processing device according to claim 2 , further comprising: providing information about the subject of interest corresponding to the message by the user. The providing unit is
The information processing device according to claim 2 , further comprising: providing information about the subject of interest corresponding to the message by a user other than the user among a plurality of users including the user. an estimation unit that estimates an interest subject in which the user has shown an interest, based on a content in which the user has performed a predetermined operation indicating the user's interest;
Further equipped with
The providing unit is
The information processing device according to claim 1 , further comprising: providing information about the subject of interest estimated by the estimation unit. The estimation unit is
The information processing device according to claim 5 , wherein the user's subject of interest is estimated based on a subject included in the content. The estimation unit is
The information processing device according to claim 5 , wherein the user's subject of interest is estimated using a capture image of the content generated by an operation of the user. The estimation unit is
The information processing device according to claim 7 , wherein the user's subject of interest is estimated based on a subject included in the captured image. The estimation unit is
The information processing device according to claim 8 , further comprising: an information processing apparatus for estimating the user's subject of interest based on a result of an analysis process for analyzing the captured image. The estimation unit is
The information processing device according to claim 9 , wherein the user's subject of interest is estimated based on a domain of the captured image. The estimation unit is
The information processing device according to claim 10, characterized in that the interest of the user is estimated based on a domain estimated using a domain estimation model that estimates the domain of the captured image. The estimation unit is
The information processing apparatus according to claim 11, wherein the domain estimation model receives a screen as an input and outputs information indicating a domain of the input image, to estimate the domain of the captured image. The estimation unit is
The information processing device according to claim 5 , wherein the user's subject of interest is estimated using sensor information when the user performs the predetermined operation. The estimation unit is
The information processing device according to claim 13 , wherein the user's subject of interest is estimated using location information of the user at the time when the user performed the predetermined operation. The acquisition unit is
Acquire the interest subject information indicating the interest subject, which is a business object or facility in which the user has shown an interest;
The providing unit is
The information processing device according to claim 1 , further comprising: providing information about the transaction object or the facility that is displayed together with the message in the messenger application. The acquisition unit is
Acquire the interest target information indicating the interest target, which is a product or store in which the user has shown an interest;
The providing unit is
The information processing device according to claim 15, further comprising: providing information about the product or the store that is to be displayed together with the message in the messenger application. The providing unit is
The information processing device according to claim 1 , further comprising: displaying information about the subject of interest together with the message on a terminal device used by the user. The providing unit is
The information processing device according to claim 1 , further comprising: transmitting information about the subject of interest displayed together with the message to a terminal device used by the user. 1. A computer-implemented information processing method, comprising:
An acquisition step of acquiring interest subject information indicating an interest subject that is an object in which the user has shown an interest, and message information indicating a message in a messenger application;
providing the user with information about the subject of interest to be displayed together with the message in the messenger application based on the subject of interest information and the message information;
13. An information processing method comprising: An acquisition step of acquiring interest subject information indicating an interest subject that is an object in which the user has shown an interest, and message information indicating a message in a messenger application;
providing the user with information about the subject of interest to be displayed together with the message in the messenger application based on the subject of interest information and the message information;
An information processing program characterized by causing a computer to execute the above.

Images