JP2020107288A - Display program, display device, display method, and information processing device - Google Patents

Abstract

To provide information on the basis of an action guideline of a user.SOLUTION: A display program causes a computer to execute an estimation procedure, a search procedure, and a display procedure. The estimation procedure estimates a subject corresponding to a user action on the basis of action information (for example, action information used by the user and collected by a terminal device) on the user action. The search procedure searches for information corresponding to the subject estimated by the estimation procedure. The display procedure displays content based on at least the information searched by the search procedure.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display program, a display device, a display method, and an information processing device.

Conventionally, a technique of collecting information on an application is known. For example, in the SNS service, an object corresponding to the attribute information of the user who has issued the browsing request and the relationship information of the user with other users, and an object corresponding to the metadata of the object are acquired, and the acquired object is acquired. A technique has been proposed in which the data is reconfigured so as to be aggregated and transmitted to the user terminal.

JP, 2005-11660, A

However, in the above-mentioned conventional technology, it is not always possible to provide information based on the guideline of the behavior of the user. For example, in the above-mentioned conventional technique, the objects are simply aggregated and transmitted based on the attribute information of the user, and the information cannot always be provided based on the guideline of the behavior of the user.

The present application has been made in view of the above, and an object thereof is to provide a display program, a display device, a display method, and an information processing device capable of providing information based on a guideline of a user's action.

A display program according to the present application, an estimation procedure for estimating a subject corresponding to the user's action, based on action information about the user's action, and a search procedure for searching information corresponding to the subject estimated by the estimation procedure. At least a display procedure for displaying the content based on the information retrieved by the retrieval procedure is executed by the computer.

According to the aspect of the embodiment, it is possible to provide the information based on the guideline of the action of the user.

FIG. 1 is a diagram (1) illustrating an example of the terminal device according to the embodiment. FIG. 2 is a diagram (2) illustrating an example of the terminal device according to the embodiment. FIG. 3 is a diagram (3) illustrating an example of the terminal device according to the embodiment. FIG. 4 is a diagram (4) illustrating an example of the terminal device according to the embodiment. FIG. 5 is a diagram illustrating a configuration example of the display processing system according to the embodiment. FIG. 6 is a diagram illustrating a configuration example of the content distribution server according to the embodiment. FIG. 7 is a diagram illustrating an example of the content information storage unit according to the embodiment. FIG. 8 is a diagram illustrating a configuration example of the terminal device according to the embodiment. FIG. 9 is a diagram illustrating an example of the behavior information storage unit according to the embodiment. FIG. 10 is a diagram (5) illustrating an example of the terminal device according to the embodiment. FIG. 11 is a diagram (6) illustrating an example of the terminal device according to the embodiment. FIG. 12 is a flowchart (1) showing an example of the display processing procedure according to the embodiment. FIG. 13 is a flowchart (2) showing an example of the display processing procedure according to the embodiment. FIG. 14 is a flowchart (3) showing an example of the display processing procedure according to the embodiment. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the terminal device.

Hereinafter, modes (hereinafter, referred to as “embodiments”) for implementing a display program, a display device, a display method, and an information processing device according to the present application will be described in detail with reference to the drawings. Note that the display program, display device, display method, and information processing device according to the present application are not limited by this embodiment. Also, in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicated description will be omitted.

[1. Embodiment)
Display processing realized by the display program and the like of the present embodiment will be described with reference to FIGS. 1 to 4. In the following description, the terminal device 100 used by the user is equipped with an application (hereinafter, may be referred to as an “information collection application”) that collects (stock) behavior information about the user's behavior. Here is an example. Further, in the following description, it is assumed that the display processing according to the embodiment is realized by the function of the information collection application.

The terminal device 100 shown in FIGS. 1 to 4 is a smart device such as a smartphone or a tablet used by a user, and is an arbitrary server device via a wireless communication network such as 3G (Generation) or LTE (Long Term Evolution). It is a mobile terminal device capable of communicating with. In the example illustrated in FIGS. 1 to 4, the terminal device 100 is a smartphone used by the user U1. Further, it is assumed that the terminal device 100 has an output unit such as a liquid crystal display (hereinafter, may be referred to as “screen”) and a touch panel is adopted. That is, the user U1 performs various operations of the terminal device 100 by touching the display surface of the output unit with a finger or a dedicated pen. The terminal device 100 may be not only a smart device, but also an information processing device such as a notebook PC (Personal Computer), a desktop PC, a mobile phone, and a PDA (Personal Digital Assistant).

Hereinafter, the display process of the information collection application performed by the terminal device 100 will be described with reference to FIGS. 1 to 4. In addition, in the following description, when illustrating the state transition of the screen of the terminal device 100, they are sequentially described as a first state, a second state,..., An Nth state (N is an arbitrary number). In addition, in the example illustrated in FIGS. 1 to 4, it is assumed that the action information of the user U1 is collected by the information collection application installed in the terminal device 100. Here, in the following description, the action information is a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, or a user. Images and videos browsed by U1 using the terminal device 100, e-mails sent and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services, and the like. Show.

First, the display process of the subject of a user's action estimated based on the user's action information will be described with reference to FIG. FIG. 1 is a diagram (1) illustrating an example of the terminal device according to the embodiment.

The 1st state of Drawing 1 shows the state of the screen which displays the subject of the behavior of user U1 estimated in the past. In the first state of FIG. 1, the terminal device 100 displays the content corresponding to the subject of the behavior of the user U1 estimated in the past in the area AR11. Further, in the first state of FIG. 1, the terminal device 100 displays the area AR12 for instructing the terminal device 100 to perform the process of estimating the subject of the action.

Subsequently, the second state of FIG. 1 shows the state of the screen displayed when the instruction of the subject estimation process is received from the user U1. In the second state of FIG. 1, the terminal device 100 displays a thumbnail corresponding to the behavior information of the user U1 collected by the information collection application in the area AR13. In the following description, a thumbnail refers to a screenshot itself that is action information, an image included in bookmarked content, and the like.

For example, in the second state of FIG. 1, the terminal device 100 displays the thumbnails D1 to D9 corresponding to the behavior information of the user U1 in the area AR13. Further, the terminal device 100 displays the area AR14 for the user U1 to perform the thumbnail selection operation displayed in the area AR13, by superimposing the area AR14 on each thumbnail. In addition, the terminal device 100 displays the area AR15 for performing an operation of confirming the thumbnail selected by the user U1.

Subsequently, the third state of FIG. 1 shows a state of the screen in which the thumbnail selection operation is received from the user U1. In the example of FIG. 1, it is assumed that the user U1 selects the thumbnails D1 to D3. Here, the thumbnail D1 corresponds to the content about the restaurant A in Sendai city bookmarked by the user U1 in the restaurant introduction service. Further, the thumbnail D2 is the search result of the route search service in which the user U1 has set the departure date and time to "January 13, 7:56" and searched for the transfer guidance from the departure place "Tokyo" to the arrival place "Sendai". Corresponds to the screen shot shown. Further, the thumbnail D3 corresponds to the email regarding the reservation confirmation of the hotel B in Sendai city on “January 13” received by the user U1 using the terminal device 100. Here, the terminal device 100 analyzes information such as text information, time information, and position information included in the action information corresponding to each thumbnail by appropriately using various conventional techniques such as image analysis and morphological analysis. To extract.

Then, the terminal device 100 transitions the screen from the third state to the fourth state by selecting the area AR15.

Subsequently, the fourth state of FIG. 1 shows a state in which the subject of the behavior of the user U1 estimated by the terminal device 100 based on the behavior information corresponding to the thumbnail selected by the user U1 is displayed on the screen. In the fourth state of FIG. 1, the terminal device 100 displays the subject of the behavior of the user U1 estimated based on the behavior information corresponding to the thumbnails D1 to D3 in the area AR16. Further, in the fourth state of FIG. 1, the terminal device 100 displays the area AR17 for the user U1 to perform the operation of determining the subject of the action.

For example, in the example of FIG. 1, in the terminal device 100, the action information corresponding to the thumbnails D1 to D3 is related to Sendai city, and the search for transfer guidance from the departure place “Tokyo” to the arrival place “Sendai” is performed, and Since the reservations for Hotel B in Sendai city are for the same date, the subject of the behavior of user U1 is estimated to be "Sendai trip" on January 13, and the estimated result is displayed in area AR16. .. Then, the terminal device 100 transitions the screen from the fourth state to the fifth state by selecting the area AR17.

Subsequently, the fifth state of FIG. 1 shows a state in which the content corresponding to the estimated subject of the behavior of the user U1 is added to the screen displayed in the first state. As shown in the fifth state in FIG. 1, the terminal device 100 displays the content corresponding to “Sendai trip” in the area AR11.

Next, a display process of content corresponding to the subject of the user's action will be described with reference to FIG. FIG. 2 is a diagram (2) illustrating an example of the terminal device according to the embodiment. FIG. 2 shows an example in which the content C1 (content relating to the schedule of “Sendai trip”) corresponding to the subject “Sendai trip” of the behavior of the user U1 estimated by the terminal device 100 in FIG. 1 is edited. It is also assumed that the content C1 is content optimized for smart devices, and that the horizontal display size is the same as the horizontal display size of the screen of the terminal device 100. Further, the content C1 is assumed to be content whose vertical display size is longer than the vertical display size of the screen of the terminal device 100.

In addition, as illustrated in FIG. 2, the terminal device 100 includes the contents C11, C12, and C14 indicating the behavior information corresponding to the thumbnails D1 to D3 selected by the user U1 in the third state of FIG. The content C1 including the content C13 for recommending information related to “Sendai trip” to the user U1 is displayed on the screen. Note that in the example of FIG. 2, the terminal device 100 includes an area AR21 in which the contents included in the content C1 are arranged in chronological order and displayed on the screen. For example, the terminal device 100 extracts time information by analyzing the action information corresponding to the content included in the content C1 by appropriately using various conventional techniques such as image analysis and morphological analysis. Then, the terminal device 100 arranges and displays the content in the display position corresponding to the extracted time information in the area AR21.

Here, in the following description, the terminal device 100 uses the departure date and time "January 13, 7:56" and the arrival date and time "January 13, 9:30" as time information from the action information corresponding to the content C12. It shall be extracted. In this case, the terminal device 100 arranges and displays it in the display position corresponding to the time information extracted from the content C12 in the area AR21.

In addition, in the following description, it is assumed that the terminal device 100 has extracted the check-in possible time “15:30 to 29:00” to the hotel B as the time information from the behavior information corresponding to the content C14. In this case, since it is assumed that the behavior of the user U1 (check-in to the hotel B) corresponding to the content C14 is performed at the end of the schedule in the subject “Sendai trip” of the behavior of the user U1, the terminal device 100 , The content C14 is arranged and displayed at the bottom end of the area AR21. Then, the terminal device 100 accepts a selection operation (for example, a tap) on the content C14, and displays the display position of the content C14 in the area AR21 in accordance with the drag operation of the finger F10 of the user U1 by checking the checkable time “15”. It is moved within the range of the display position corresponding to "30 minutes to 29:00".

Further, in the following description, the terminal device 100 extracts only the business hours of the “restaurant A” as the time information from the behavior information corresponding to the content C11, and extracts the fixed time (for example, the reserved date and time). It was not done. In this case, the terminal device 100 determines that the time when the action of the user U1 (visiting the restaurant A) corresponding to the content C12 is performed is undetermined, and the content C12 is placed in an area other than the area AR21 (the uppermost end of the content C1). Place and display. In other words, the schedule of the "Sendai trip" in the initial state does not include the action regarding the visit to the restaurant A.

Further, in the description with reference to FIG. 2, the terminal device 100 arranges the content C13 in the area AR21 in which no other content is arranged (that is, the position corresponding to the blank time when the schedule is not fixed). indicate.

The first state of FIG. 2 shows an initial state in which the terminal device 100 displays the content C1 on the screen. In the first state of FIG. 2, the terminal device 100 displays the content C11 and the content C12 from the uppermost end of the content C1 on the screen. Then, the terminal device 100 superimposes and displays the area AR22 for instructing the terminal device 100 to change the display mode of the content C1 on the content C1. Further, the terminal device 100 superimposes and displays the area AR23 for instructing the terminal device 100 to save the content C1 (board) on the content C1. When the area AR23 is selected by the user U1, the terminal device 100 ends the editing process of the content C1 and displays the content arranged in the area AR21 and the information on the time corresponding to each content in the “Sendai trip” schedule. Is stored in the storage unit of the terminal device 100. Note that the terminal device 100 may set an alarm to notify the user U1 based on the information regarding the time corresponding to each content.

Here, in the first state of FIG. 2, it is assumed that the user U1 desires to visit the restaurant A. In this case, the terminal device 100 receives a selection operation on the content C11 by the finger F10 of the user U1, and moves the display position of the content C11 according to the drag operation of the finger F10 of the user U1. Then, the terminal device 100 arranges the content C11 at the position where the selection of the content C11 by the finger F10 of the user U1 is released, and displays the content C11 on the screen. In the description of FIG. 2, it is assumed that the content C11 is arranged in a region corresponding to “12:00” to “12:00” in the region AR21.

Subsequently, in the second state of FIG. 2, the content C1 is moved downward by a scrolling operation of the user U1 or the like, so that the area of the content C1 which is hidden in the first state is displayed on the screen. Indicates. In the second state of FIG. 2, the terminal device 100 displays the content C11, the content C12, and the content C13 on the screen. Also in the second state, the terminal device 100 displays the area AR22 and the area AR23 by superimposing them on the content C1.

Here, in the second state of FIG. 2, it is assumed that the user U1 desires to incorporate the schedule in the blank time. In this case, the terminal device 100 receives a selection operation for the content C13 by the user U1 and transitions the screen to the third state.

Subsequently, the third state of FIG. 2 shows a state in which the content indicating the information related to the action subject “Sendai trip” is displayed on the screen in place of the content C1. In the third state of FIG. 2, the terminal device 100 searches for information corresponding to the subject of the action, and displays the content corresponding to the searched information on the screen as content recommended for the user U1. For example, the terminal device 100 searches the web (a web page or various applications installed in the terminal device 100) for information corresponding to the action subject “Sendai Travel”. In addition, the terminal device 100 is the action information of the user U1 other than the action information corresponding to the content C11, the content C12, and the content C13 collected by the information collection application installed in the terminal device 100, and the subject of the action “Sendai Travel”. Search for behavior information related to. In addition, the terminal device 100 searches for information corresponding to the subject “Sendai trip” of the action based on the demographic attribute or the psychographic attribute based on the action information of the user U1. Then, the terminal device 100 displays, among the retrieved information, the content related to the action that the user U1 can perform in the blank time, as the content indicating the information recommended to the user U1. In addition, in the third state of FIG. 2, it is assumed that the terminal device 100 displays the content C15 and the content C16 on the screen as the content indicating the information recommended to the user U1.

In the third state of FIG. 2, it is assumed that it is desired to incorporate the schedule regarding the content C15 in the blank time. In this case, the terminal device 100 receives the selection operation for the content C15 by the user U1, and transitions the screen to the fourth state.

Subsequently, the fourth state of FIG. 2 shows the state of the screen in which the new content is displayed in place of the content C13. In the fourth state of FIG. 2, the terminal device 100 displays the content C17 corresponding to the content C15 selected by the user U1 in the third state on the screen instead of the content C13. Also in the second state, the terminal device 100 displays the area AR22 and the area AR23 by superimposing them on the content C1.

Here, in the fourth state of FIG. 2, when the area AR22 is selected, the terminal device 100 transitions the screen to the fifth state or the sixth state.

Subsequently, the fifth state of FIG. 2 shows a state of the screen in which the content arranged in the area AR21 of the content C1 is arranged at the position corresponding to each content on the map. For example, in the example of FIG. 2, the terminal device 100 appropriately uses various conventional techniques such as image analysis and morphological analysis for the action information corresponding to the content C11 and the content C14 and the information regarding the content C17. The position information is extracted by analyzing. Then, the terminal device 100 arranges each content at the position P1 corresponding to the content C11, the position P2 corresponding to the content C17, and the position P3 corresponding to the content C14 on the map of Sendai city, and displays them on the screen.

The terminal device 100 may arrange and display each content in the content C1 based on the position information extracted from the action information corresponding to the content included in the content C1. For example, the terminal device 100 sets “Sendai Station” as the departure point. Then, the terminal device 100 extracts position information from the action information corresponding to each content included in the content C1, and arranges each content in the content C1 according to the distance from the “Sendai Station” of the extracted position information. indicate.

Further, the terminal device 100, based on the information on the time corresponding to each content arranged in the area AR21 (the information on the time when the user U1 moves to the position corresponding to each content), the position P1 on the map of Sendai city. Information about the order in which the user U1 visits may be displayed on to P3.

Subsequently, the sixth state of FIG. 2 shows a state of a screen for displaying thumbnails corresponding to the content arranged in the area AR21 of the content C1 in the form of an album. In the sixth state of FIG. 2, the terminal device 100 displays thumbnails corresponding to the content C11, the content C12, the content C14, and the content C17 arranged in the area AR21 of the content C1 on the screen in the form of an album.

In the example of FIG. 2, as shown in the sixth state, the content C18 relating to the belongings of the user U1 in the action subject “Sendai trip” may be displayed. For example, the terminal device 100 receives a selection operation for the content C18 by the user U1, and transitions the screen to the edit screen of the property of the user U1 in the action subject “Sendai trip”. In addition, the terminal device 100 may superimpose and display an icon image or the like for the user U1 to instruct the terminal device 100 to transition to the editing screen of the property, in the first state to the sixth state of FIG.

Further, in the example of FIG. 2, the terminal device 100 may search for information regarding a transportation means between positions corresponding to the content included in the content C1, and display the content corresponding to the searched information on the content C1. For example, in the second state of FIG. 2, the terminal device 100 searches for a means of transportation from Sendai Station to the location of the restaurant A by a route search service or the like, and a content indicating the search result is displayed between the content C11 and the content C12. Deploy.

In addition, in the example of FIG. 2, when the behavior information related to the subject “Sendai trip” of the behavior is newly collected after the content C1 is displayed, the terminal device 100 displays the content corresponding to the newly collected behavior information. May be notified as to whether or not to place in the content C1.

Next, with reference to FIG. 3, description will be given of a display process for displaying information regarding action information by superimposing it on the thumbnail. FIG. 3 is a diagram (3) illustrating an example of the terminal device according to the embodiment. In FIG. 3, when the thumbnail corresponding to the behavior information of the user U1 collected by the information collection application is displayed on the screen (corresponding to the second state in FIG. 1), the information regarding the behavior information (hereinafter referred to as “target information”) is displayed in each thumbnail. May be described) is displayed in an overlapping manner. Here, in the following description, the target information is information acquired by analyzing the action information by appropriately using various conventional techniques such as image analysis and morphological analysis (for example, text information, time information, position Information), information for identifying the source of behavior information (for example, application, web service, etc.), icon image corresponding to the source of behavior information (for example, app icon, set to web service) Indicates an icon image (web click icon), etc. In addition, in the following description, it is assumed that the information collection application acquires target information in association with behavior information.

In the following description, an example in which the terminal device 100 displays the thumbnails D1 to D9 corresponding to the action information collected by the information collection application will be shown. Since the thumbnails D1 to D3 are the same as those shown in FIG. 1, their description will be omitted. The thumbnail D4 corresponds to a screen shot of a screen displaying the ticket issuing code (QR code (registered trademark)) of the ticket of the concert C purchased by the user U1 using the terminal device 100 in the ticket sales service. The thumbnail D5 corresponds to the screen shot in the map application. The thumbnails D6 to D9 correspond to products bookmarked by the user U1 in the electronic commerce service.

The first state of FIG. 3 shows a state of a screen displaying thumbnails corresponding to the behavior information of the user U1 collected by the information collection application. In the example of FIG. 3, the terminal device 100 superimposes and displays the area AR18 for displaying the target information on each thumbnail. Since the areas AR13 to AR15 are the same as those shown in the second state of FIG. 1, the description thereof will be omitted.

In the example of FIG. 3, the terminal device 100 displays the text information acquired from the behavior information and the icon image corresponding to the provider of the behavior information in the area AR18 as the target information. For example, in the example of FIG. 3, the terminal device 100 superimposes and displays the text information on the thumbnails D1 and D5 together with the icon image corresponding to the source of the action information. Further, when the behavior information includes time information (for example, departure date and time and reservation date and time), the terminal device 100 superimposes and displays the target information including the time information as shown in the thumbnails D2 to D5. Further, when the behavior information includes the product and product information such as the price of the product, the terminal device 100 superimposes and displays the target information including the product information as shown in the thumbnails D6 to D9.

The terminal device 100 may execute the process corresponding to the character string acquired as the target information. For example, when the target information including the time information is displayed by being superimposed on the thumbnail, the terminal device 100 notifies the user U1 of the alarm based on the time information (for example, the restaurant bookmarked by the user in the restaurant introduction service is Alerts indicating that it is outside business hours or closed days) may be set. Further, the terminal device 100 may set an alarm for notifying the user U1 based on the current position of the user U1. For example, when the position information is included in the behavior information (for example, the content about the restaurant bookmarked by the user in the restaurant introduction service), the terminal device 100 determines that the current position of the user U1 is within a predetermined range from the position information (for example, the position information). , 100 meters, 500 meters, etc.), the user U1 may be notified of the action information. Further, for example, the terminal device 100 may collect information about stores around the current position of the user U1 from an external device such as a service providing device that provides a service to the user. Then, when there is a store that handles products corresponding to the behavior information of the user (for example, products corresponding to the thumbnails D6 to 9) in a store within a predetermined range from the current position of the user U1, the terminal device 100, the user U1. May be notified to the action information.

In addition, the terminal device 100 may display the target information updated based on the information provided by the behavior information provider. For example, the terminal device 100 may display the thumbnail D1 by superimposing the target information including the information about the business hours of the restaurant A, the change of holidays, and the like. Further, the terminal device 100 may superimpose and display target information including train delay information on the thumbnail D2. Further, the terminal device 100 may superimpose and display the target information including the information regarding the change in the price of the product (for example, the information regarding the discount) on the thumbnails D6 to D9. Further, the terminal device 100 may notify the user U1 that the target information has been updated based on the information provided by the source of the behavior information.

Subsequently, the second state of FIG. 3 shows the state of the screen displayed when the thumbnail displayed in the first state is selected by the user U1. In the example of FIG. 3, when the thumbnail displayed in the first state is selected by the user U1, the terminal device 100 is content that is the source of the action information corresponding to the thumbnail and that corresponds to the action information. Is displayed on the screen. Here, in the example of FIG. 3, it is assumed that the thumbnail D2 is selected. In this case, the terminal device 100 displays the content (search result) on the route search service at the time when the user U1 acquires the screen shot, which is the action information corresponding to the thumbnail D2, on the screen. That is, the terminal device 100 displays the application or the web service that is the source of the action information in a state where the same parameter information as that input when the action information is acquired is input.

In addition, when the thumbnail corresponding to the image of the product taken by the user U1 using the imaging unit of the terminal device 100 in the actual shop is selected, the terminal device 100 selects a predetermined e-commerce service corresponding to the product. Content may be displayed.

Next, with reference to FIG. 4, a display process of content (hereinafter, may be referred to as “related content”) indicating information related to the subject of the user's action will be described. FIG. 4 is a diagram (4) illustrating an example of the terminal device according to the embodiment. 4, an example in which related content is displayed in the editing process of the content C2 (content related to the schedule of "Sendai trip") corresponding to the subject "Sendai trip" of the behavior of the user U1 estimated by the terminal device 100 in FIG. Show. It is also assumed that the content C2 is content optimized for smart devices, and that the horizontal display size is the same as the horizontal display size of the screen of the terminal device 100. The content C2 is assumed to be content whose vertical display size is longer than the vertical display size of the screen of the terminal device 100.

Further, as shown in FIG. 4, the terminal device 100 displays on the screen the content C2 including the content C12 and C14 indicating the action information corresponding to the thumbnails D2 and D3 selected by the user U1 in the third state of FIG. To do. The contents C12 and C14 in the example of FIG. 4 are the same as those shown in FIG.

The first state of FIG. 4 shows an initial state in which the terminal device 100 displays the content C2 on the screen. In the first state of FIG. 4, the terminal device 100 displays the content C12 on the screen from the uppermost end of the content C2. Then, the terminal device 100 superimposes and displays the areas AR22 and AR23 on the content C2. The areas AR22 and AR23 in the example of FIG. 4 are the same as those shown in FIG.

Here, in the first state of FIG. 4, it is assumed that the user F1's finger F10 has performed an upward sliding operation on the lowermost portion of the screen (not the area AR23). In this case, the terminal device 100 transitions the screen to the second state.

Subsequently, the second state of FIG. 4 shows a state in which an area for displaying related content (hereinafter, sometimes referred to as “related content area”) is displayed by being superimposed on the content C2. In the example of FIG. 4, in the related content area, the behavior information corresponding to the thumbnail D1 selected by the user U1 in the third state of FIG. 1 (in other words, of the behavior information selected by the user U1 in the third state of FIG. 1). Among them, the content C11 indicating the action information in which the time of the action of the user U1 corresponding to the action information is undetermined), and the information related to the subject “Sendai trip” of the action and recommended to the user U1. Content C19 and content C20 indicating information are included. Note that the content C11 in the example of FIG. 4 is the same as that shown in FIG. 2, so description thereof will be omitted.

Here, in the second state of FIG. 4, it is assumed that the selection operation for the content C11 is performed by the finger F10 of the user U1. In this case, the terminal device 100 transitions the screen to the third state.

Subsequently, the third state of FIG. 4 shows a state in which the related content area is reduced. In addition, in the third state of FIG. 4, it is assumed that the selection operation for the content C11 by the finger F10 of the user U1 is continued. In this case, the terminal device 100 reduces the related content area while the display position of the content C11 is held by the finger F10 of the user U1. That is, the terminal device 100 superimposes the content C11 on the content C2 and displays it while being held by the finger F10 of the user U1.

Subsequently, the fourth state of FIG. 4 shows a state in which the related content area is reduced and related contents other than the content C11 are hidden from the screen of the terminal device 100. In addition, in the third state of FIG. 4, it is assumed that the selection operation for the content C11 by the finger F10 of the user U1 is continued. Here, the terminal device 100 moves the display position of the content C11 according to the drag operation to the content C11 by the finger F10 of the user U1.

Subsequently, the fifth state of FIG. 4 shows a state where the display position of the content C11 is moved. As shown in the fifth state in FIG. 4, the terminal device 100 displays the content C11 at a position corresponding to the operation by the finger F10 of the user U1.

Subsequently, the sixth state of FIG. 4 shows a state in which the selection operation on the content C11 by the finger F10 of the user U1 is released. As shown in the sixth state of FIG. 4, the terminal device 100 incorporates the schedule related to the content C11 at the position (time zone) where the selection operation for the content C11 is released.

Note that the terminal device 100 may display the related content area by superimposing it on a predetermined area of the content C2 displayed on the screen. For example, the terminal device 100 superimposes and displays the related content area on the area from the bottom of the screen to the center of the screen (that is, the lower half area of the screen) in the content C2 displayed on the screen. Further, in the first state of FIG. 4, the terminal device 100 sets the related content area in the area from the bottom of the screen to the position where the upward sliding operation on the bottom of the screen by the finger F10 of the user U1 is released. ..

Further, when the terminal device 100 receives a predetermined operation (for example, a flick operation) on the content arranged in the content C2, the terminal device 100 may delete the content from the content C2 and display the content in the related content area.

When the terminal device 100 receives a selection operation (for example, a touch operation, which may be referred to as a “move operation” below) with the finger F10 of the user U1 with respect to the content arranged in the content C2, the terminal device 100 selects the content. The content may be displayed at a position corresponding to the operation of the finger F10 of the user U1, and the content may be displayed at a position of the content C2 where the moving operation is released. Here, the terminal device 100 may update the action information (for example, time information) corresponding to the content according to the position (time zone) where the movement operation of the content is released.

In addition, when the terminal device 100 receives a selection operation (for example, a long tap) different from the movement operation, which is a selection operation performed by the finger F10 of the user U1 on the content arranged in the content C2, the terminal device 100 receives the content. The related content area may be displayed superimposed on the content C2 while being held by the finger F10 of U1. Then, when the selection of the content is canceled in the related content area, the terminal device 100 may delete the content from the content C2 and display the content in the related content area.

Further, the screen shot of the screen of the terminal device 100 in the above description may be acquired by the function of the information collecting application or the function of the OS (Operating System) of the terminal device 100.

In addition, the user behavior information in the above description is not limited to the information collected by the function of the information collection application, and may include information added to the information collection application according to the user's operation.

In general, a user investigates things based on some action guideline, uses the collected information to make a schedule, and acts. At this time, if the collected information results are aggregated and provided to the user in the form of a schedule or the like, the user experience is improved. Here, the terminal device 100 according to the embodiment estimates the subject of the action of the user based on the action information of the user as described above, and displays the content corresponding to the subject of the action. That is, the terminal device 100 according to the embodiment can provide information based on the guideline of the behavior of the user.

Further, conventionally, a technique of taking a screen shot of a display screen to acquire image data while operating an application of a mobile terminal, and a technique of displaying a screen shot image by superimposing a title or a URL on the web page are disclosed. Has been done. However, in the conventional technology, the application or IT service that is the basis of the screenshot obtained in the past cannot be intuitively understood, and there is room for improving usability. On the other hand, the terminal device 100 according to the embodiment can display the text information included in the screenshot and the information regarding the provider of the screenshot by superimposing them on the screenshot. That is, the terminal device 100 according to the embodiment can improve usability.

Further, conventionally, there has been proposed a technique for adjusting the layout of contents (main information and sub information) based on the arrangement of a display area for sub display, which is different from the display area for main information. However, in the conventional technology, operability when the user changes the arrangement of the contents is not taken into consideration, and there is room for improving usability. On the other hand, the terminal device 100 according to the embodiment can arrange the information (sub-information) displayed in the related content area as described above in the main content by a user's intuitive operation. That is, the terminal device 100 according to the embodiment can improve usability.

[2. Configuration of display processing system]
Next, the display processing system 1 for realizing the above-mentioned display processing will be described with reference to FIG. FIG. 5 is a diagram illustrating a configuration example of the display processing system according to the embodiment. As shown in FIG. 5, the display processing system 1 includes a terminal device 100, a content distribution server 10, and an external device 20. The terminal device 100, the content distribution server 10 and the external device 20 are communicably connected to each other via a network N in a wired or wireless manner. The network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (a mobile telephone network, a fixed telephone network, etc.), a regional IP (Internet Protocol) network, and the Internet. The network N may include a wired network or a wireless network. In addition, the display processing system 1 illustrated in FIG. 5 may include a plurality of terminal devices 100, a plurality of content distribution servers 10, and a plurality of external devices 20.

The content distribution server 10 is a server device that distributes content related to the information collection application to the terminal device 100. The content distribution server 10 may distribute the data itself of the information collection application as the content to the terminal device 100. The content distribution server 10 may function as a distribution device that distributes control information to the terminal device 100. Here, the control information is described in a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). The information collection application itself distributed from the content distribution server 10 may be regarded as the control information. The configuration of the content distribution server 10 will be described later.

The external device 20 is a server device that distributes information of various contents to the terminal device 100. The external device 20 is related to, for example, a portal site, a news site, an auction site, a weather forecast site, a shopping site, a finance (stock price) site, a route search site, a map providing site, a travel site, a restaurant introduction site, a web blog, and the like. It is a device for delivering to the terminal device 100 a web page on which various types of information are placed. The external device 20 is, for example, various applications installed in the terminal device 100 (for example, portal application, news application, auction site, weather forecast application, shopping application, finance (stock price) application, route search application, map providing application). , A travel application, a restaurant introduction application, a blog browsing application, etc.) may be a device that distributes information to the terminal device 100. The external device 20 may also be a server that delivers the application data itself. The external device 20 may also function as a distribution device that distributes control information to the terminal device 100. Here, the control information is described in a script language such as JavaScript (registered trademark) or a style sheet language such as CSS. The application itself delivered from the external device 20 may be regarded as the control information.

The terminal device 100 is an information processing device used by a user. The terminal device 100 acquires information about the information collection application from the content distribution server 10 according to the user's operation. In addition, the terminal device 100 acquires information of various contents from the external device 20 according to a user's operation. Further, when the terminal device 100 receives the control information for realizing the above-mentioned display process from the content distribution server 10, the terminal device 100 realizes the display process according to the control information. The configuration of the terminal device 100 will be described later.

[3. Content distribution server configuration]
Next, the configuration of the content distribution server 10 according to the embodiment will be described with reference to FIG. FIG. 6 is a diagram illustrating a configuration example of the content distribution server according to the embodiment. As shown in FIG. 6, the content distribution server 10 includes a communication unit 11, a storage unit 12, and a control unit 14.

(About communication unit 11)
The communication unit 11 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is connected to the network N in a wired or wireless manner, and transmits/receives information to/from the terminal device 100, the external device 20, and the like.

(About storage unit 12)
The storage unit 12 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. As shown in FIG. 6, the storage unit 12 has a content information storage unit 13.

(About the content information storage unit 13)
The content information storage unit 13 stores information about content. For example, the content information storage unit 13 stores content (corresponding to “content C1” in the example of FIG. 2) corresponding to the subject of the user's action. The content information storage unit 13 also stores content that is information related to the subject of the user's action and that indicates information recommended to the user. Further, the content information storage unit 13 stores the content corresponding to the subject of the action of the user that is recommended according to the action information and the attribute information of the user. Here, an example of information about the content stored in the content information storage unit 13 will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating an example of the content information storage unit according to the embodiment. In the example of FIG. 7, the content information storage unit 13 has items such as “content ID”, “content information”, “attribute information”, and “action history”.

The “content ID” indicates identification information for identifying the content. “Content information” indicates information that constitutes the content. In FIG. 7, an example in which conceptual information such as “contents information 1” is stored in “contents information” is shown. However, in reality, text information indicating the contents, image information, video information, or A file path name indicating the storage location is stored.

The “attribute information” stores information such as demographic attributes and psychographic attributes of the users who are the distribution targets of the content stored in the “content information”. The “action history” stores the history of the action information of the user who is the distribution target of the content stored in the “content information”.

That is, in FIG. 7, the “content information 1” identified by the content ID “C101” is distributed to the user who has the attribute corresponding to the “attribute information 1” and who has performed the action shown in the “action history 1”. The following is an example.

(About control unit 14)
The control unit 14 is a controller, and various programs stored in a storage device inside the content distribution server 10 use a RAM as a work area by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by being executed as. The control unit 14 is a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in FIG. 6, the control unit 14 according to the embodiment includes a reception unit 15, an acquisition unit 16, an estimation unit 17, and a distribution unit 18, and has functions and actions of information processing described below. Realize or execute.

(About reception section 15)
The receiving unit 15 receives a content distribution request and action information from the terminal device 100.

(About acquisition unit 16)
The acquisition unit 16 analyzes the action information received by the receiving unit 15 from the terminal device 100 by appropriately using various conventional techniques such as image analysis and morphological analysis to obtain information (target information) about the action information. get. For example, the acquisition unit 16 acquires target information including an icon image (for example, an application icon or an icon image (web click icon) set in a web service) corresponding to a source of behavior information. In addition, the acquisition unit 16 acquires target information including text information, time information, position information, and the like by analyzing action information by appropriately using various conventional techniques such as image analysis and morphological analysis.

(About the estimation unit 17)
The estimation unit 17 estimates the subject corresponding to the user's action based on the action information regarding the user's action. For example, the estimating unit 17 analyzes the action information received by the receiving unit 15 from the terminal device 100 by appropriately using various conventional techniques such as image analysis and morphological analysis, and thereby the text information included in the action information. Also, information such as time information and position information is extracted. Then, the estimation unit 17 estimates the subject of the user's action based on the information extracted from the action information.

(About distribution unit 18)
When the receiving unit 15 receives a content distribution request, the distribution unit 18 distributes the content stored in the content information storage unit 13 according to the attribute information and the behavior information of the user who uses the terminal device 100. Further, the distribution unit 18 distributes the target information acquired by the acquisition unit 16 to the terminal device 100. The distribution unit 18 is a terminal device used by the user, and distributes information about a subject to a terminal device that displays information corresponding to the subject estimated by the estimation unit 17. For example, the delivery unit 18 delivers the information on the subject of the user's action estimated by the estimation unit 17 to the terminal device 100.

[4. Configuration of terminal device]
Next, the configuration of the terminal device 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a diagram illustrating a configuration example of the terminal device according to the embodiment. As shown in FIG. 8, the terminal device 100 includes a communication unit 110, a storage unit 120, an input unit 130, an output unit 140, an imaging unit 150, and a control unit 160.

(About communication unit 110)
The communication unit 110 is realized by, for example, a NIC. Then, the communication unit 110 is connected to the network N by wire or wirelessly, and transmits/receives information to/from the content distribution server 10, the external device 20, and the like.

(About storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 8, the storage unit 120 has a behavior information storage unit 121.

(About the behavior information storage unit 121)
The behavior information storage unit 121 stores behavior information regarding the behavior of the user. Here, an example of the action information stored in the action information storage unit 121 will be described with reference to FIG. 9. FIG. 9 is a diagram illustrating an example of the behavior information storage unit according to the embodiment. In the example of FIG. 9, the behavior information storage unit 121 has items such as “date”, “action information”, and “analysis result information”.

The “date” indicates the date when the user who uses the terminal device 100 takes the action indicated by the information stored in the “action information”. The “action information” indicates information regarding an action performed by a user who uses the terminal device 100. Note that FIG. 9 shows an example in which conceptual information such as “contents information 1” is stored in the “behavior information”. Images and videos captured by using the terminal device 100, images and videos viewed by the user using the terminal device 100, emails sent and received by the user using the terminal device 100, applications used by the user via the terminal device 100, and web services. Information such as bookmarked content in the file etc., or a file path name indicating the storage location is stored.

“Analysis result information” is information such as time information and position information extracted by analyzing the information stored in “behavior information” by appropriately using various conventional techniques such as image analysis and morphological analysis. The information (target information in the example of FIG. 3) is shown.

That is, in FIG. 9, an example in which the user who uses the terminal device 100 performs the action shown in “action information 1” on the date “20XX/1/9” and “analysis result 1” is extracted from “action information 1” Indicates.

(Regarding the input unit 130 and the output unit 140)
The input unit 130 is an input device that receives various operations from a user. For example, the input unit 130 is realized by a keyboard, a mouse, operation keys, or the like. The output unit 140 is a display device for displaying various information. That is, the output unit 140 corresponds to the display screen of the terminal device 100. For example, the output unit 140 is realized by a liquid crystal display or the like. When the touch panel is used as the terminal device 100, the input unit 130 and the output unit 140 are integrated.

(About the imaging unit 150)
The image pickup unit 150 has an optical system such as a lens, and an image pickup element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

(About control unit 160)
The control unit 160 is a controller, and various programs (corresponding to an example of a display program) stored in a storage device inside the terminal device 100 are executed by a RAM as a work area by a CPU, an MPU, or the like, for example. Will be realized. When the control information or application for realizing the above-mentioned display processing is distributed from the content distribution server 10, the control unit 160 operates according to the control information or application to realize the display processing.

The control unit 160 is a controller and is realized by an integrated circuit such as an ASIC or FPGA. As illustrated in FIG. 8, the control unit 160 according to the embodiment includes an acquisition unit 161, a processing unit 162, an estimation unit 163, a search unit 164, a display unit 165, and a display control unit 166. It realizes or executes the functions and actions of information processing described below.

(About the acquisition unit 161)
The acquisition unit 161 acquires, using the captured image, target information that is information regarding the capturing target captured in the captured image. For example, in the example of FIG. 3, the acquisition unit 161 includes a screen shot of the screen of the terminal device 100, images and videos captured by the user U1 using the terminal device 100, and images browsed by the user U1 using the terminal device 100. Image analysis, morpheme to action information including moving images, e-mails transmitted and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services, and the like. The target information is acquired by performing analysis by appropriately using various conventional techniques such as analysis.

The acquisition unit 161 may also acquire target information indicating information about an application that is a provider of a shooting target. For example, in the example of FIG. 3, the acquisition unit 161 acquires target information including information for specifying the source of the behavior information (for example, application, web service, etc.).

The acquisition unit 161 may also acquire target information including an icon image indicating an application that is a provider of a shooting target. For example, in the example of FIG. 3, the acquisition unit 161 includes a target including an icon image (for example, an application icon or an icon image (web click icon) set in a web service) corresponding to the source of the behavior information. Get information.

In addition, the acquisition unit 161 may acquire the analysis result of the character string to be captured as the target information. For example, in the example of FIG. 3, the acquisition unit 161 analyzes text information, time information, position information, and the like by analyzing action information by appropriately using various conventional techniques such as image analysis and morphological analysis. Acquire the target information that includes.

In addition, the acquisition unit 161 acquires target information indicating information indicated by the two-dimensional code to be captured. Normally, a ticket issuing code (eg, QR code (registered trademark)) display screen provided by a ticket sales service or the like displays time information about the date and time of the event corresponding to the ticket issuing code together with the ticket issuing code and the venue. It is assumed that information such as location information regarding is displayed. Therefore, the acquisition unit 161 analyzes the screen shot of the screen displaying the two-dimensional code using image analysis to obtain target information such as time information indicating the information indicated by the two-dimensional code and position information. get.

The acquisition unit 161 may acquire the target information from the content distribution server 10. For example, the acquisition unit 161 transmits the behavior information to the content distribution server 10 together with the information for identifying the provider of the behavior information. Then, the acquisition unit 161 acquires the target information by receiving from the content distribution server 10 the target information extracted by the content distribution server 10 analyzing the information for identifying the source of the behavior information and the behavior information. May be.

(About processing unit 162)
The processing unit 162 executes a process corresponding to the character string acquired as the target information. For example, the processing unit 162 executes processing corresponding to the character string acquired as the target information, based on the current position of the user U1. For example, in the example of FIG. 3, when the action information (for example, the content related to the restaurant bookmarked by the user in the restaurant introduction service) includes the position information, the processing unit 162 determines that the current position of the user U1 is from the position information. When it is within a predetermined range (for example, 100 meters, 500 meters, etc.), the user U1 is notified of the action information.

Further, for example, in the example of FIG. 3, the processing unit 162 collects information about stores around the current position of the user U1 from the external device 20 or the like such as a service providing device that provides a service to the user U1. Then, when there is a store that handles products corresponding to the behavior information of the user (for example, products corresponding to the thumbnails D6 to 9) in a store within a predetermined range from the current position of the user U1, the terminal device 100, the user U1. To the relevant action information.

In addition, the processing unit 162 may notify the user U1 that the target information has been updated based on the information provided by the behavior information provider. For example, in the example of FIG. 3, the processing unit 162 includes information about changes in the business hours of the restaurant A corresponding to the behavior information of the thumbnail D1, holidays, and train delay information corresponding to the behavior information of the thumbnail D2. Alternatively, the user U1 may be notified of information such as information regarding price changes of products corresponding to the behavior information of the thumbnails D6 to D9. Further, the processing unit 162 may notify the user U1 of the information regarding the detour plan as well as the notification of the delay information of various transportation means.

In addition, when the character string acquired as the target information indicates the time, the processing unit 162 may execute a process of notifying the user of the time. For example, in the example of FIG. 3, the processing unit 162 may notify the action information corresponding to the thumbnails D2 to D4 whose target information includes time information based on the time information.

(About the estimation unit 163)
The estimation unit 163 estimates the subject corresponding to the user's action based on the action information regarding the user's action. For example, in the example of FIG. 1, the estimation unit 163 estimates the subject corresponding to the action of the user U1 based on the action information of the user U1 collected by the information collection application installed in the terminal device 100.

In addition, the estimation unit 163 may estimate the subject based on a part of the behavior information among the behavior information. For example, in the example of FIG. 1, the estimation unit 163 estimates the subject of the action of the user U1 based on the action information related to “Sendai” (action information corresponding to the thumbnails D1 to D3).

In addition, the estimation unit 163 may estimate the subject based on the action information selected by the user from the action information. For example, in the example of FIG. 1, the estimation unit 163 estimates the subject of the behavior of the user U1 based on the behavior information corresponding to the thumbnails D1 to D3 selected by the user U1.

In addition, the estimation unit 163 may estimate the subject based on the action information that is information stored according to the user's operation. For example, in the example of FIG. 1, the estimation unit 163 uses a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, images and videos captured by the user U1 using the terminal device 100, and the user. Images and videos viewed by the U1 using the terminal device 100, e-mails transmitted and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services, etc. , The subject of the action of the user U1 is estimated based on the action information acquired by the active operation of the user U1.

In addition, the estimation unit 163 may estimate the subject based on the action information including the information regarding the electronic mail. For example, in the example of FIG. 1, the estimation unit 163 is based on the behavior information that the user U1 receives using the terminal device 100 and that includes the email regarding the reservation confirmation of the hotel B in Sendai city on “January 13”. Then, the subject of the behavior of the user U1 is estimated.

In addition, the estimation unit 163 may estimate the subject based on the action information including the information based on the image recognition result of the image regarding the display screen of the application. For example, in the example of FIG. 1, the estimation unit 163 analyzes text information, time information, and position by analyzing screen shots of the display screen of the route search service on a predetermined application by appropriately using various conventional techniques. Extract information such as information. Then, the estimation unit 163 estimates the subject of the behavior of the user U1 based on the behavior information including the extracted information.

In addition, the estimation unit 163 may acquire the behavior information including the result of the route search performed by the user, estimate the region visited by the user based on the result of the route search, and use the estimated region as the subject. For example, in the example of FIG. 1, the estimation unit 163 acquires action information including a screen shot of a search result display screen in the route search service performed by the user U1. Then, the estimation unit 163 sets the subject “Sendai trip” related to the destination “Sendai” extracted by analyzing the screen shot of the display screen by appropriately using various conventional techniques as the subject of the behavior of the user U1. To do.

In addition, the estimation unit 163 may estimate the subject based on the action information selected by the user from the action information. For example, in the example of FIG. 1, the estimation unit 163 estimates the subject of the action of the user U1 based on the action information corresponding to the thumbnails D1 to D3 selected by the user U1.

The estimating unit 163 may acquire information such as text information, time information, and position information included in the behavior information from the content distribution server 10. For example, the estimation unit 163 transmits the behavior information to the content distribution server 10. Then, the estimation unit 163 receives information such as text information extracted by the content distribution server 10 by analyzing the behavior information, time information, and position information from the content distribution server 10, and the user's behavior based on the received information. Estimate the subject of. In addition, the estimation unit 163 may acquire information on the subject of the user's action estimated by the content delivery server 10 (estimation unit 17) based on the action information transmitted to the content delivery server 10. The estimating unit 163 may synchronize the subject of the user's action estimated by the estimating unit 163 and the subject of the user's action estimated by the content distribution server 10.

(About the search unit 164)
The search unit 164 searches for information corresponding to the subject estimated by the estimation unit 163. For example, in the example of FIG. 2, the search unit 164 obtains information corresponding to the subject “Sendai trip” of the action from the action information storage unit 121, the external device 20 such as the service providing device that provides the service to the user U1, and the like. Search for.

Further, the search unit 164 may search the action information for information corresponding to the subject. For example, in the example of FIG. 2, the search unit 164 searches the behavior information collected by the information collection application installed in the terminal device 100 and related to the behavior subject “Sendai trip”.

The search unit 164 may search for information on the web corresponding to the subject. For example, in the example of FIG. 2, the search unit 164 searches the web page and various applications installed in the terminal device 100 for information corresponding to the subject “Sendai trip” of the action.

Further, the search unit 164 may search for information on facilities that correspond to the subject and that are within a predetermined range from the position that corresponds to the behavior information selected by the user. For example, when the action information selected by the user includes location information, the search unit 164 retrieves information on facilities corresponding to subjects within a predetermined range (for example, 100 meters, 500 meters, etc.) from the location information. Search for.

In addition, the search unit 164 searches for facility information based on the search history of the user. For example, the search unit 164 searches facility information based on the user's search history included in the behavior information collected by the information collection application installed in the terminal device 100.

The search unit 164 may search for information corresponding to the subject of the user's action indicated by the information, based on the information on the subject of the user's action acquired by the estimation unit 163 from the content distribution server 10. For example, when the information about the subject of the user's action acquired from the content distribution server 10 by the estimation unit 163 indicates “Sendai trip”, the search unit 164 stores information corresponding to “Sendai trip” in the action information storage unit. 121 or an external device 20 such as a service providing device that provides a service to the user U1.

(About display unit 165)
The display unit 165 displays at least the content based on the information retrieved by the retrieval unit 164. For example, in the example of FIG. 2, the display unit 165 displays the external device 20 such as a service providing device that provides a service to the user U1, the action information of the user U1 collected by the information collecting application installed in the terminal device 100, and the like. The content C1 based on the information corresponding to the action subject “Sendai trip” retrieved by the retrieval unit 164 is displayed.

The display unit 165 may also display content based on the behavior information regarding the behavior of the user and the information retrieved by the retrieval unit 164. For example, in the example of FIG. 2, the display unit 165 displays the content C11, the content C12, and the content C13 corresponding to the behavior information collected by the information collection application installed in the terminal device 100, and the information searched by the search unit 164. The content C1 based on the corresponding content C17 is displayed.

In addition, the display unit 165 may display the content in a mode according to the situation of the user. For example, in the example of FIG. 2, when the current time is morning (for example, 5:00 to 8:00) or night (for example, 18:00 to 24:00), the display unit 165 displays the content C1 by the user U1. Since it is assumed that the content C1 is browsed for the purpose of confirming the schedule of the current day or the schedule of the next day, the content C1 is displayed in a timeline format (first state, second state and fourth state in FIG. 2) or , Album format (sixth state in FIG. 2).

Further, for example, the display unit 165 displays the content in a mode according to the situation of the user estimated based on the sensor information detected by various sensors mounted on the terminal device 100. For example, in the example of FIG. 2, when the user's situation is estimated to be the “walking state” by the various sensors mounted on the terminal device 100, the display unit 165 displays the content C1 in the map format (see FIG. 2). 5 states).

Further, the display unit 165 displays the content in a mode according to the subject. For example, in the example of FIG. 2, the display unit 165 displays the content C1 in the format of the imline according to the trip and accepts the editing operation because the estimation unit 163 has estimated that the subject of the user U1's action is “Sendai trip”. .. The display unit 165 may display the content in various modes according to the subject of the behavior of the user estimated by the estimation unit 163. Here, an example of displaying the content in the form of an album will be described with reference to FIG. FIG. 10 is a diagram (5) illustrating an example of the terminal device according to the embodiment.

In the example of FIG. 10, the behavior information regarding the product (shoes H to shoes K) bookmarked by the user U1 in the electronic commerce service is selected, and the estimation unit 163 sets the subject of the behavior of the user U1 as “under consideration of shoes to purchase”. (Note of what you want)”. Then, in the first state of FIG. 10 (corresponding to the fifth state of FIG. 1), it is assumed that the display unit 165 displays content corresponding to the subject “wanted memo” of the behavior of the user U1 in the area AR11.

In the first state of FIG. 10, the display unit 165 transitions the screen to the second state by accepting the selection operation of the user U1 with respect to the content corresponding to the action subject “wanted memo”. The second state of FIG. 10 shows a state in which information corresponding to the action information selected by the user is displayed. In the example of FIG. 10, the display unit 165 displays the contents C21 to C24 corresponding to the action information selected by the user on the screen. Further, the display unit 165 displays an area AR22 for the user U1 to instruct to change the display mode of the contents C21 to C24 on the screen.

In the second state of FIG. 10, the display unit 165 displays the content C21 to C24 including the image of the product acquired from the source of the action information, the price of the product, the information about the discount, and the like. Note that, in the second state of FIG. 10, the display unit 165 relates to the shoes that are considered for purchase in the subject of the behavior of the user U1, based on the demographic attribute or the psychographic attribute based on the behavior information of the user U1. Information may be displayed.

Further, in the second state of FIG. 10, when the area AR22 is selected, the display unit 165 displays map information indicating the position of the actual store that sells the products corresponding to the contents C21 to C24 on the screen. Good. For example, the display unit 165 displays map information indicating the position of a physical store that sells products corresponding to the contents C21 to C24 within a predetermined range (for example, 100 meters, 500 meters, etc.) from the current position of the user U1. Display on screen.

Subsequently, the third state of FIG. 10 shows a state in which information corresponding to the action information selected by the user U1 is displayed. For example, in the example of FIG. 10, the display unit 165 displays the page of the product corresponding to the behavior information selected by the user U1 in the electronic commerce service on the browser included in the information collection application.

Further, the display unit 165 may extract time information or position information from the information corresponding to the subject, and display the content in which the information corresponding to the subject is arranged at the display position based on the time information or the position information. For example, in the example of FIG. 2, the display unit 165 extracts the time information from the action information corresponding to the content included in the content C1, arranges the content at the display position corresponding to the extracted time information, and displays the content on the screen. .. Further, in the example of FIG. 2, the display unit 165 sets “Sendai Station” as a departure point and extracts position information from action information or the like corresponding to each content included in the content C1. Then, the display unit 165 arranges each content in the content C1 according to the distance from the “Sendai Station” of the extracted position information and displays it on the screen.

Further, the display unit 165 may display the content including the map information in which the information corresponding to the subject is arranged based on the position information. For example, in the example of FIG. 2, the display unit 165 extracts the position information from the action information or the like corresponding to the content included in the content C1, and displays each content at the position corresponding to the extracted position information on the map of Sendai city. Place and display on the screen.

The display unit 165 may also display content in which information corresponding to the subject is arranged in chronological order based on the time information or the position information. For example, in the example of FIG. 2, the display unit 165 extracts time information or position information from the action information or the like corresponding to the content included in the content C1 and arranges each content included in the content C1 in the area AR21. Display on screen.

Further, the display unit 165 may display the action information selected by the user and the information on the web. For example, in the example of FIG. 2, the display unit 165 displays the content C11, the content C12, and the content C13 corresponding to the action information selected by the user U1, and the content C1 including the content corresponding to the information searched by the search unit 164. indicate.

The display unit 165 may display the recommendation of the subject of the action. Here, with reference to FIG. 11, an example of displaying the recommendation of the subject of the action will be described. FIG. 11 is a diagram (6) illustrating an example of the terminal device according to the embodiment.

In addition, in the example of FIG. 11, it is assumed that the action information of the user U1 regarding the movie L is collected by the information collection application installed in the terminal device 100. For example, in the example of FIG. 11, as the action information regarding the movie L, an image and a movie regarding the movie L browsed by the user U1 using the terminal device 100, a bookmark in a web service regarding the movie theater M showing the movie L, and the like. It is supposed to be collected by the information collection app. Further, in the example of FIG. 11, it is assumed that the estimation unit 163 estimates the subject of the action of the user U1 as “movie L” based on the action information regarding the movie L. Then, in the first state of FIG. 11 (corresponding to the fifth state of FIG. 1), it is assumed that the display unit 165 displays content corresponding to the subject “movie L” of the action recommended to the user U1 in the area AR11.

The first state of FIG. 11 shows a state in which recommendations of the subject of action are displayed. In the first state of FIG. 11, the display unit 165 reduces the visibility of the content corresponding to the action subject “Movie L” and displays the text “Why not go see a movie this weekend?” indicate. Here, the display unit 165 transitions the screen to the second state by accepting the selection operation of the user U1 with respect to the content corresponding to the action subject “movie L”.

Subsequently, the second state of FIG. 11 shows a state in which the content related to the action subject “movie L” is displayed. For example, in the example of FIG. 11, the search unit 164 searches for information related to the action subject “movie L”. Then, the display unit 165 extracts the time information from the information searched by the search unit 164, and displays it at the display position corresponding to the extracted time information while reducing the visibility of the contents C25 to C27 indicating the information. Further, also in the first state of FIG. 11, the terminal device 100 displays the area AR22 and the area AR23. Here, the content C25 indicates information regarding transfer guidance from the residential area "Shimokitazawa" of the user U1 to the location "Shinjuku" of the movie theater M in the route search service. Also, the content C26 indicates information regarding the show time of the movie L in the movie theater M. In addition, the content C27 is a result of the search unit 164 searching for information related to the area “Shinjuku” that the user U1 visits in the action subject “movie L” (information about the restaurant N whose location is “Shinjuku”). Show.

In the second state of FIG. 11, when a predetermined operation (for example, tap, long tap, etc.) is performed on the contents C25 to C27, an application corresponding to each content, a web service, contents, etc. are displayed. Good. For example, when a predetermined operation is performed on the content C25, the display unit 165 launches the route search service with the parameter information corresponding to the content C25 input. Then, when the parameter information is changed (for example, the arrival time is changed), the display unit 165 displays the content C25 at the display position corresponding to the changed parameter information in the second state of FIG. Further, for example, when a predetermined operation is performed on the content C26 or C27, the display unit 165 displays a page relating to the reservation of the movie L or the restaurant N in the facility reservation service. Further, for example, when a predetermined operation is performed on the content C27, the display unit 165 corresponds to information related to the area “Shinjuku” visited by the user U1 and different from the information on the restaurant N. The content is displayed on the screen as content recommended to the user U1.

Subsequently, the third state of FIG. 11 shows a state in which the content displayed in the second state is selected by the user U1. In the example of FIG. 11, it is assumed that the contents C25 to C27 are selected by the user U1. In this case, the display unit 165 displays the contents C25 to C27 with enhanced visibility. Then, since the area AR23 is selected in the third state of FIG. 11, the display unit 165 stores the content related to the subject “movie L” of the action in the storage unit 120 and shifts the screen to the fourth state. ..

Then, the 4th state of FIG. 11 shows the state in which the recommendation of the subject of action was selected by the user U1. As shown in the fourth state in FIG. 11, the display unit 165 displays the content corresponding to the subject “movie L” of the action in a highly visible state.

In addition, the display unit 165 may display the target information acquired by the acquisition unit 161 by superimposing it on the captured image. For example, in the example of FIG. 3, the display unit 165 displays the area AR18 displaying the target information acquired by the acquisition unit 161 so as to be superimposed on each thumbnail.

Further, the display unit 165 may superimpose the target information updated based on the information provided by the application that is the source of the shooting target on the captured image and display the target information. For example, in the example of FIG. 3, the display unit 165 displays information about changes in the business hours of restaurants, holidays, etc. provided by the provider of the shooting target, delay information on various transportation means, and price changes of products. The target information updated by the information regarding the above is superimposed and displayed on each thumbnail.

In addition, the display unit 165 may display a screen corresponding to the shooting target, which is provided by an application that is a provider of the shooting target, when the shot image is selected. For example, in the example of FIG. 3, the display unit 165 has input the same parameter information as the parameter information input when the action information corresponding to the thumbnail selected by the user U1 is input, and the source of the action information is displayed. Display applications and web services that are.

In addition, the display unit 165 may superimpose and display the third content in which the second content is arranged on the first content. For example, in the example of FIG. 4, the display unit 165 displays the related content area for displaying the related content by superimposing it on the content C2.

In addition, the display unit 165 may superimpose and display the third content in which the plurality of second contents are arranged on the first content. For example, in the example of FIG. 4, the display unit 165 displays the related content area including the content C19 and the content C20 indicating the information recommended for the user U1 by being superimposed on the content C2.

The display unit 165 may also display the third content when the fourth content included in the first content is selected. For example, in the example of FIG. 4, when the display unit 165 receives a selection operation by the finger F10 of the user U1 for the content arranged in the content C2, the display unit 165 holds the content with the finger F10 of the user U1 and the related content. The area is displayed so as to be superimposed on the content C2.

Further, the display unit 165 may display the third content while holding the position of the selected fourth content. For example, in the example of FIG. 4, when the display unit 165 receives a selection operation different from the moving operation, which is a selection operation by the finger F10 of the user U1 with respect to the content placed in the content C2, the content is displayed by the user U1. The related content area is displayed while being superposed on the content C2 while being held by the finger F10.

(About display control unit 166)
When the second content is selected, the display control unit 166 reduces the area in which the third content is displayed while displaying the first content and the selected second content. For example, in the example of FIG. 4, the display control unit 166 reduces the related content area while displaying the content C11 that has been selected by the finger F10 of the user U1.

Also, the display control unit 166 may display the second content by superimposing it on the first content. For example, in the example of FIG. 4, the display control unit 166 displays the content C11 on which the selection operation by the finger F10 of the user U1 has been performed by superimposing it on the content C2.

Further, the display control unit 166 may display the selected second content in a state where the display mode of the selected second content is changed so as to be superimposed on the first content. For example, the display control unit 166 displays the second content in a state where vibration, blinking, coloring, size up, size down, brightness up, brightness down, marking addition, and the like are superimposed on the first content. Let

Further, when the second content is selected, the display control unit 166 may display the first content and the third content in a changed display mode. For example, the display control unit 166 displays the first content and the third content in a state in which vibration, blinking, coloring, size up, size down, brightness up, brightness down, marking addition, and the like have been performed.

Further, the display control unit 166 may superimpose and display the first content in a state where the visibility of the selected second content is enhanced. For example, in the example of FIG. 4, the display control unit 166 superimposes and displays the content C11 on which the selection operation is performed with the finger F10 of the user U1 in a state in which the visibility is enhanced.

Further, the display control unit 166 may display the selected second content in a state in which the visibility of the selected second content is reduced and superimposed on the first content. For example, in the example of FIG. 4, the display control unit 166 displays the content C11 superimposed on the content C2 in a state in which the visibility of the content C11 that has been selected by the finger F10 of the user U1 is reduced.

Further, the display control unit 166 may display the first content and the third content in a state in which the visibility is reduced. For example, in the example of FIG. 4, the display control unit 166 displays the content C2 and the related content area with reduced visibility after the selection operation of the content C11 is performed by the finger F10 of the user U1.

Further, the display control unit 166 may gradually reduce the area in which the third content is displayed. For example, in the example of FIG. 4, the display control unit 166 gradually reduces the related content area after the selection operation to the content C11 is performed by the finger F10 of the user U1.

Further, the display control unit 166 may gradually expand the area in which the first content is displayed. For example, in the example of FIG. 4, the display control unit 166 gradually enlarges the content C2 after the selection operation for the content C11 is performed by the finger F10 of the user U1.

Further, the display control unit 166 may reduce the area in which the third content is displayed while maintaining the position of the second content. For example, in the example of FIG. 4, the display control unit 166 gradually reduces the related content area while the content C11 is held by the finger F10 of the user U1.

In addition, the display control unit 166 may reduce the area in which the third content is displayed when the second content is continuously selected. For example, in the example of FIG. 4, the display control unit 166 reduces the related content area when the content C11 is operated by a long tap or the like with the finger F10 of the user U1.

In addition, the display control unit 166 may reduce the area in which the third content is displayed when the selection position where the second content is selected is moved in the predetermined direction. For example, in the example of FIG. 4, the display control unit 166 reduces the related content area when the selection operation for the content C11 is performed by the finger F10 of the user U1 and the content C11 is moved in the predetermined direction.

Further, the display control unit 166 may move the display position of the second content according to the movement of the selection position where the second content is selected. For example, in the example of FIG. 4, the display control unit 166 moves the display position of the content C11 according to the drag operation of the finger F10 of the user U1 onto the content C11.

In addition, when the selection of the second content is released, the display control unit 166 may display the second content by superimposing the second content on a position corresponding to the position of the first content where the selection of the second content is released. Good. For example, in the example of FIG. 4, the display control unit 166 displays the content C11 at a position in the content C2 where the selection operation for the content C11 by the finger F10 of the user U1 is released.

In addition, when any of the second contents is selected, the display control unit 166 reduces the area in which the third content is displayed while displaying the first content and the selected second content. Good. For example, in the example of FIG. 4, when the content C11, the content C19, or the content C20 displayed in the related content area is selected, the display control unit 166 associates the selected content with the selected content displayed. Shrink the content area.

In addition, the display control unit 166 may hide the four contents in the area where the first content is displayed when a predetermined operation is performed on the fourth content included in the first content. For example, in the example of FIG. 4, when the display control unit 166 receives a predetermined operation (for example, a flick operation) on the content placed in the content C2, the display control unit 166 deletes the content from the content C2.

Further, when the selection of the fourth content is canceled in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed and displays the third content. The fourth content may be displayed in the displayed area. For example, in the example of FIG. 4, the display control unit 166 selects the content on the related content area displayed by the display unit 165 in response to the selection operation by the finger F10 of the user U1 on the content arranged in the content C2. When the content is released, the content is deleted from the content C2 and displayed in the related content area.

Further, when the selection of the fourth content is canceled in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed and displays the third content. The area in which the third content is displayed may be reduced while the fourth content is displayed in the area. For example, in the example of FIG. 4, when the selection of the content placed in the content C2 by the finger F10 of the user U1 is canceled in the related content area, the display control unit 166 deletes the content from the content C2 and The related content area is reduced while being displayed in the content area.

[5. Display processing flow]
The procedure of content display processing of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 12 is a flowchart (1) showing an example of the display processing procedure according to the embodiment.

As shown in FIG. 12, the terminal device 100 estimates the subject of the user's action based on the user's action information (step S101). Subsequently, the terminal device 100 searches for information corresponding to the subject of the user's action (step S102). For example, the terminal device 100 determines the subject of the user's action on the web (web page or various applications installed in the terminal device 100) or from the user's action information collected by the information collection application installed in the terminal device 100. Search for information corresponding to.

Subsequently, the terminal device 100 displays the content based on the retrieved information (step S103), and ends the process.

Next, a procedure of display processing of target information of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 13 is a flowchart (2) showing an example of the display processing procedure according to the embodiment.

As illustrated in FIG. 13, the terminal device 100 acquires target information from the behavior information (step S201). For example, the terminal device 100 identifies the source of the text information, the time information, the position information, and the action information by analyzing the action information by appropriately using various conventional techniques such as image analysis and morphological analysis. Information, an icon image corresponding to the provider of the action information, and the like are acquired.

Then, the terminal device 100 superimposes and displays the target information on the captured image (step S202). For example, the terminal device 100 superimposes and displays the target information on the screen shot or the like of the terminal device 100 acquired as the behavior information.

Subsequently, the terminal device 100 executes a process corresponding to the target information (step S203), and ends the process. For example, when the target information includes time information, the terminal device 100 sets an alarm to notify the user based on the time information.

Next, a procedure of content display processing of the display processing system 1 according to the embodiment will be described with reference to FIG. FIG. 14 is a flowchart (3) showing an example of the display processing procedure according to the embodiment.

As shown in FIG. 14, the terminal device 100 displays the first content (step S301). For example, in the example of FIG. 4, the terminal device 100 displays the content C2. Subsequently, when the operation for instructing the display processing of the third content is performed (step S302; Yes), the terminal device 100 displays the third content in which the second content is arranged by superimposing it on the first content. (Step S303). For example, in the example of FIG. 4, the terminal device 100 displays the related content area in which the related content is arranged by superimposing it on the content C2. On the other hand, when the operation for instructing the display processing of the third content is not performed (step S302; No), the terminal device 100 displays the first content as it is. Then, when the operation of instructing the storage processing of the first content is performed (step S306; Yes), the terminal device 100 stores the first content (step S307) and ends the processing. In addition, the terminal device 100 continuously displays the first content when the operation for instructing the storage processing of the first content is not performed (step S306; No).

Subsequently, when the second content is selected (step S304; Yes), the terminal device 100 reduces the area in which the third content is displayed while the first content and the second content are displayed (step S305). .. On the other hand, when the second content is not selected (step S304; No), the terminal device 100 reduces the area in which the third content is displayed and displays the first content. For example, when a predetermined operation (for example, a swipe operation) is performed on the third content, the terminal device 100 reduces the area in which the third content is displayed and displays the first content. Then, when the operation of instructing the storage processing of the first content is performed (step S306; Yes), the terminal device 100 stores the first content (step S307) and ends the processing. In addition, the terminal device 100 continuously displays the first content when the operation for instructing the storage processing of the first content is not performed (step S306; No).

Then, when an operation for instructing the storage processing of the first content is performed (step S306; Yes), the first content is stored (step S307), and the processing ends. For example, the terminal device 100 stores the first content in the state where the second content is arranged. On the other hand, when the operation of instructing the storage processing of the first content is not performed (step S306; No), the terminal device 100 continuously displays the first content in which the second content is arranged.

[6. Modification example)
The above-mentioned embodiment shows an example, and various modifications and applications are possible.

[6-1. About behavior information]
In the above-described embodiment, the behavior information is a screen shot of the screen of the terminal device 100 acquired by the user U1 using the terminal device 100, an image and a moving image captured by the user U1 using the terminal device 100, or the user U1. Examples of images and videos browsed using the terminal device 100, emails sent and received by the user U1 using the terminal device 100, applications used by the user U1 via the terminal device 100, contents bookmarked in web services, and the like. However, the action information is not limited to the example described above. The behavior information may be any information as long as it is information about the behavior of the user that can be acquired by the information processing device or the like.

[6-2. Regarding Display Unit 165 and Display Control Unit 166]
In the example of FIG. 4 described above, an example in which the display unit 165 and the display control unit 166 display the related content area displaying the related content by superimposing the related content area on the screen is shown. However, the functions of the display unit 165 and the display control unit 166 are described. Is not limited to the above example. The second content may be any information as long as the information can be added to the first content. For example, the display unit 165 displays the message application as the first content, and superimposes the third content on which the second content, which is information such as text, image, and moving image that can be added to the message, is arranged on the first content. indicate. Then, the display control unit 166 performs the above-described display control in the message application.

[6-3. Displaying content based on user behavior patterns]
The display unit 165 may display the content based on the behavior pattern of the user. For example, when the action information indicating that the user continues to visit the region O is collected, the estimation unit 163 estimates that the subject of the user's action is “a visit to the region O”. Then, the display unit 165 displays the content including the information on the region O searched by the search unit 164 (facility such as restaurants around the region O). In addition, the display unit 165 may change and display the content of the content each time the content related to the predetermined area is displayed. Further, the display unit 165 may display the content to the user based on the information regarding the evaluation of the content after the user has actually performed the action corresponding to the content displayed by the display unit 165.

[6-4. Display of content related to activity information]
The display unit 165 may display the content related to the collected behavior information of the user. For example, when the user's behavior information includes information such as concert viewing and event participation, the display unit 165 displays content related to the next work of the concert and content related to the next year of the event. ..

[7. effect〕
As described above, the terminal device 100 according to the embodiment has the acquisition unit 161, the processing unit 162, the estimation unit 163, the search unit 164, the display unit 165, and the display control unit 166. The acquisition unit 161 acquires, using the captured image, target information that is information regarding the capturing target captured in the captured image. The processing unit 162 executes a process corresponding to the character string acquired as the target information. The estimation unit 163 estimates the subject corresponding to the user's action based on the action information regarding the user's action. The search unit 164 searches for information corresponding to the subject estimated by the estimation unit 163. The display unit 165 displays at least the content based on the information retrieved by the retrieval unit 164. Further, the display unit 165 displays the target information acquired by the acquisition unit 161 so as to be superimposed on the captured image. Further, the display unit 165 displays the first content by superimposing the third content on which the second content is arranged. When the second content is selected, the display control unit 166 reduces the area in which the third content is displayed while displaying the first content and the selected second content.

Accordingly, the terminal device 100 according to the embodiment displays the content based on the information corresponding to the subject of the user's action, and thus can provide the information based on the guideline of the user's action. Moreover, since the terminal device 100 according to the embodiment displays the target information regarding the shooting target in a manner that the target information is superimposed on the captured image, usability can be improved. In addition, the terminal device 100 according to the embodiment controls the display of the content according to the case where the content dynamically changes, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on a part of the action information among the action information. Then, the search unit 164 searches the action information for information corresponding to the subject.

As a result, the terminal device 100 according to the embodiment estimates the subject of the user's action based on the action information having relevance in the action information, and retrieves the information to be provided to the user from the user's action information. It is possible to provide the user with information with high accuracy and high appeal.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on the action information selected by the user from the action information.

With this, the terminal device 100 according to the embodiment estimates the subject of the user's action based on the action information selected by the user, and thus can provide information according to the user's request.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays content based on the action information regarding the action of the user and the information searched by the search unit 164.

Accordingly, the terminal device 100 according to the embodiment displays the content corresponding to the subject of the user's action, and thus can provide the user with highly appealing information.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the content in a mode according to the situation of the user.

As a result, the terminal device 100 according to the embodiment displays the content according to the time period when the user browses the content and the moving state of the user, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the content in a mode according to the subject.

Accordingly, the terminal device 100 according to the embodiment can display the content in a mode according to the subject of the user's action, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 extracts time information or position information from information corresponding to the subject, and information corresponding to the subject at the display position based on the time information or the position information. Display the content with placed.

Thereby, the terminal device 100 according to the embodiment can display the content based on the time information or the position information, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the content including the map information in which the information corresponding to the subject is arranged based on the position information.

As a result, the terminal device 100 according to the embodiment provides the user with information along with the position on the map, so that usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays content in which information corresponding to the subject is arranged in chronological order based on time information or position information.

Accordingly, the terminal device 100 according to the embodiment displays information in chronological order, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on the action information that is the information stored according to the user's operation.

Accordingly, the terminal device 100 according to the embodiment estimates the subject of the user's action based on the action information stored by the user, and thus can provide information according to the user's request.

In addition, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on the action information including the information regarding the email.

As a result, the terminal device 100 according to the embodiment estimates the subject of the user's action including the email, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on the action information including the information based on the image recognition result of the image regarding the display screen of the application.

Accordingly, the terminal device 100 according to the embodiment estimates the subject of the user's action including the display screen of the application, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 acquires the behavior information including the result of the route search performed by the user, and estimates the region visited by the user based on the result of the route search, The subject is the estimated area.

As a result, the terminal device 100 according to the embodiment provides the information about the visited area of the user, and thus can provide the user with highly appealing information.

Further, in the terminal device 100 according to the embodiment, for example, the estimation unit 163 estimates the subject based on the action information selected by the user from the action information. Then, the search unit 164 searches for information on the web corresponding to the subject. Then, the display unit 165 displays the action information selected by the user and the information on the web.

Thus, the terminal device 100 according to the embodiment searches the information related to the subject of the user's action on the web and displays the information, so that usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the search unit 164 searches for information on a facility corresponding to the subject and within a predetermined range from the position corresponding to the behavior information selected by the user. To do.

Thereby, the terminal device 100 according to the embodiment displays the facility related to the subject of the user's action, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the search unit 164 searches facility information based on the search history of the user.

Thereby, the terminal device 100 according to the embodiment displays the facility based on the search history of the user, and thus usability can be improved.

In addition, in the terminal device 100 according to the embodiment, for example, the acquisition unit 161 acquires target information indicating information regarding an application that is a provider of a shooting target. Further, for example, the acquisition unit 161 acquires target information including an icon image indicating an application that is a provider of a shooting target. In addition, for example, the display unit 165 displays the target information updated based on the information provided by the application that is the source of the shooting target, on the captured image in a superimposed manner.

As a result, the terminal device 100 according to the embodiment displays the captured image in a state in which the information regarding the provider of the capturing target can be visually recognized, and thus usability can be improved.

In addition, in the terminal device 100 according to the embodiment, for example, when the captured image is selected, the display unit 165 displays a screen corresponding to the capturing target, which is provided by the application that is the provider of the capturing target. To do.

As a result, the terminal device 100 according to the embodiment displays the screen of the application of the provider of the shooting target, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the acquisition unit 161 acquires the analysis result of the character string to be captured as target information. In addition, for example, the acquisition unit 161 acquires target information indicating the information indicated by the two-dimensional code to be captured.

As a result, the terminal device 100 according to the embodiment displays the captured image in a state where the information of the capturing target is visible, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, when the character string acquired as the target information indicates time, the processing unit 162 executes a process of notifying the user of the time.

As a result, the terminal device 100 according to the embodiment executes the notification regarding the shooting target based on the time information included in the shooting target, and thus usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 causes the second content to be superimposed on the first content and displayed.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 displays the selected second content in a state in which the display mode of the selected second content is changed so as to be superimposed on the first content. In addition, for example, when the second content is selected, the display control unit 166 causes the first content and the third content to be displayed in a changed display mode. In addition, for example, the display control unit 166 displays the selected second content in a state in which the visibility of the selected second content is enhanced and is superimposed on the first content. In addition, for example, the display control unit 166 displays the selected second content in a state where the visibility of the selected second content is reduced and is superimposed on the first content. Further, for example, the display control unit 166 displays the first content and the third content in a state in which the visibility is reduced.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 gradually reduces the area in which the third content is displayed. Further, for example, the display control unit 166 gradually expands the area in which the first content is displayed.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 reduces the area in which the third content is displayed while maintaining the position of the second content. Further, for example, the display control unit 166 reduces the area in which the third content is displayed when the second content is continuously selected. Further, for example, the display control unit 166 reduces the area in which the third content is displayed when the selection position where the second content is selected is moved in the predetermined direction.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, the display control unit 166 moves the display position of the second content according to the movement of the selection position where the second content is selected. Further, for example, when the selection of the second content is canceled, the display control unit 166 displays the second content by superimposing the second content at a position corresponding to the position of the second content where the selection of the second content is canceled. ..

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

In addition, in the terminal device 100 according to the embodiment, for example, the display unit 165 displays the first content by superimposing the third content in which the plurality of second contents are arranged. Then, when any one of the second contents is selected, the display control unit 166 reduces the area in which the third content is displayed while displaying the first content and the selected second content.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, in the terminal device 100 according to the embodiment, for example, when the predetermined operation is performed on the fourth content included in the first content, the display control unit 166 displays the fourth content in the area in which the first content is displayed. Hide content. Further, for example, the display unit 165 displays the third content when the fourth content included in the first content is selected. Then, when the selection of the fourth content is released in the area where the third content is displayed, the display control unit 166 hides the fourth content in the area where the first content is displayed and displays the third content. The fourth content is displayed in the displayed area. Further, for example, the display unit 165 displays the third content while holding the position of the selected fourth content. In addition, for example, the display control unit 166 hides the fourth content in the area in which the first content is displayed, and displays the third content while displaying the fourth content in the area in which the third content is displayed. Reduce the area.

With this, the terminal device 100 according to the embodiment displays the content according to the intuitive operation of the user, and thus the usability can be improved.

Further, as described above, the content distribution server 10 according to the embodiment has the reception unit 15, the acquisition unit 16, the estimation unit 17, and the distribution unit 18. The receiving unit 15 receives a content distribution request and action information from the terminal device 100. The acquisition unit 16 acquires information on the behavior information by analyzing the behavior information received by the reception unit 15 from the terminal device 100 by appropriately using various conventional techniques such as image analysis and morphological analysis. The estimation unit 17 estimates the subject corresponding to the user's action based on the action information regarding the user's action. The distribution unit 18 is a terminal device used by the user, and distributes information about a subject to a terminal device that displays information corresponding to the subject estimated by the estimation unit 17.

As a result, the content distribution server 10 according to the embodiment estimates the subject of the user's action based on the user's action information, and thus can provide information according to the user's request.

[8. Hardware configuration]
Further, the terminal device according to each of the embodiments described above is realized by, for example, a computer 1000 configured as shown in FIG. Hereinafter, the terminal device 100 will be described as an example. FIG. 15 is a hardware configuration diagram illustrating an example of a computer that realizes the function of the terminal device. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

The CPU 1100 operates based on a program stored in the ROM 1300 or the HDD 1400, and controls each unit. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, a program dependent on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. The communication interface 1500 receives data from another device via the communication network 500 (corresponding to the network N of the embodiment) and sends the data to the CPU 1100, and also transmits data generated by the CPU 1100 via the communication network 500 to another device. Send to device.

The CPU 1100 controls output devices such as a display and a printer and input devices such as a keyboard and a mouse via the input/output interface 1600. The CPU 1100 acquires data from the input device via the input/output interface 1600. In addition, the CPU 1100 outputs the data generated via the input/output interface 1600 to the output device.

The media interface 1700 reads a program or data stored in the recording medium 1800 and provides the program or data to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) and a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Etc.

For example, when the computer 1000 functions as the terminal device 100, the CPU 1100 of the computer 1000 realizes the function of the control unit 160 by executing the program loaded on the RAM 1200. Further, the HDD 1400 stores each data in the storage device of the terminal device 100. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from other devices via a predetermined communication network.

[9. Other]
As described above, some of the embodiments of the present application have been described in detail based on the drawings, but these are examples, and various modifications based on the knowledge of those skilled in the art, including the modes described in the section of the disclosure of the invention, It is possible to implement the present invention in other forms with improvements.

Further, the terminal device 100 described above can be flexibly changed in configuration, for example, by implementing an external platform or the like by calling it with an API (Application Programming Interface) or network computing depending on the function.

Further, the “unit” described in the claims can be read as “means” or “circuit”. For example, the acquisition unit can be read as an acquisition unit or an acquisition circuit.

10 Content Distribution Server 11 Communication Unit 12 Storage Unit 13 Content Information Storage Unit 14 Control Unit 15 Reception Unit 16 Acquisition Unit 17 Estimating Unit 18 Distribution Unit 20 External Device 100 Terminal Device 110 Communication Unit 120 Storage Unit 121 Action Information Storage Unit 130 Input Unit 140 Output Unit 150 Imaging Unit 160 Control Unit 161 Acquisition Unit 162 Processing Unit 163 Estimating Unit 164 Searching Unit 165 Display Unit 166 Display Control Unit

Claims (19)

An estimation procedure for estimating a subject corresponding to the user's action based on action information about the user's action;
A search procedure for searching information corresponding to the subject estimated by the estimation procedure;
A display program for causing a computer to execute at least a display procedure for displaying content based on information retrieved by the retrieval procedure. The estimation procedure is
Based on some of the behavior information of the behavior information, the subject is estimated,
The search procedure is
The display program according to claim 1, wherein the behavior information is searched for information corresponding to the subject. The estimation procedure is
The display program according to claim 2, wherein the subject is estimated based on action information selected by the user from the action information. The display procedure is
The display program according to any one of claims 1 to 3, which displays content based on the behavior information regarding the behavior of the user and the information retrieved by the retrieval procedure. The display procedure is
The display program according to any one of claims 1 to 4, wherein the content is displayed in a mode according to the situation of the user. The display procedure is
The display program according to any one of claims 1 to 5, wherein the content is displayed in a mode according to the subject. The display procedure is
The time information or the position information is extracted from the information corresponding to the subject, and the content in which the information corresponding to the subject is arranged at a display position based on the time information or the position information is displayed. The display program according to any one of 1 to 6. The display procedure is
The display program according to claim 7, which displays the content including map information in which information corresponding to the subject is arranged based on the position information. The display procedure is
The display program according to claim 7, further comprising: displaying the content in which information corresponding to the subject is arranged in chronological order based on the time information or the position information. The estimation procedure is
The display program according to any one of claims 1 to 9, wherein the subject is estimated based on the action information that is information stored according to an operation of the user. The estimation procedure is
The display program according to any one of claims 1 to 10, wherein the subject is estimated based on the action information including information about electronic mail. The estimation procedure is
The display program according to any one of claims 1 to 11, wherein the subject is estimated based on the action information including information based on an image recognition result of an image regarding a display screen of an application. The estimation procedure is
The action information including the result of the route search performed by the user is acquired, the region visited by the user is estimated based on the result of the route search, and the estimated region is set as the subject. Item 13. A display program according to any one of items 1 to 12. The estimation procedure is
Based on the behavior information selected by the user of the behavior information, the subject is estimated,
The search procedure is
Search for information on the web that corresponds to the subject,
The display procedure is
The display program according to claim 1, wherein the behavior information selected by the user and the information on the web are displayed. The search procedure is
15. The display program according to claim 14, wherein information on facilities that correspond to the subject and are within a predetermined range from a position that corresponds to the behavior information selected by the user is searched. The search procedure is
The display program according to claim 15, wherein information on the facility is searched based on a search history of the user. An estimation unit that estimates a subject corresponding to the user's action based on action information about the user's action;
A search unit for searching information corresponding to the subject estimated by the estimation unit;
At least a display unit for displaying content based on the information retrieved by the retrieval unit. A display method performed by a computer,
An estimation step of estimating a subject corresponding to the user's action based on action information about the user's action;
A search step of searching information corresponding to the subject estimated by the estimating step;
At least, a display step of displaying content based on the information retrieved by the retrieval step. An estimation unit that estimates a subject corresponding to the user's action based on action information about the user's action;
A terminal unit used by the user, comprising: a distribution unit that distributes information related to the subject to a terminal device that displays information corresponding to the subject estimated by the estimation unit. apparatus.

Images