WO2016084453A1

WO2016084453A1 - Information processing device, control method and program

Info

Publication number: WO2016084453A1
Application number: PCT/JP2015/075268
Authority: WO
Inventors: 勝吉金本; 栗屋　志伸; 拓也藤田; 淳史野田
Original assignee: ソニー株式会社
Priority date: 2014-11-27
Filing date: 2015-09-04
Publication date: 2016-06-02
Also published as: JP2016103079A

Abstract

[Problem] To provide an information processing device, control method and program that can provide, as a topic, a temporal difference of a subject to be recognized. [Solution] An information processing device of the present invention is provided with: a comparison unit that compares a current feature quantity of a subject to be recognized and a feature quantity thereof at another point in time on a time series; and a notification control unit that performs control to provide notice of the difference of the subject to be recognized based on the comparison results of the comparison unit.

Description

Information processing apparatus, control method, and program

The present disclosure relates to an information processing device, a control method, and a program.

In recent years, life logs that digitize and record user life, behavior, experience, etc. into video, audio, location information, movement information, etc. have become popular. The life log is automatically recorded by a wearable device (smart band, smart eye glass, smart watch, etc.) worn by the user or a mobile terminal (smart phone, etc.) possessed by the user, and stored in a predetermined server on the cloud, for example. The

When browsing the life log accumulated in this way, in general, past history is displayed in chronological order, or all recorded data is only displayed. When there is data to be browsed, the user has to search by himself / herself by inputting a search condition.

In addition, as a technique for automatically extracting recommendation information to the user, for example, the following cited document 1 provides information for suggesting other contents to be accessed next based on the user profile information and access history information A method is disclosed.

JP 2002-108923 A

However, if you want to know from the life log what happened before, such as when you meet a person you have met before or when you have visited a place that you have visited before, the image from the date and time at that time If it is necessary to search and the date and time are not clear, it was difficult to find out.

Also, if it is known that the target person or landscape has changed since meeting or visiting before, the difference can be discussed, but it was difficult to notice easily.

Therefore, the present disclosure proposes an information processing apparatus, a control method, and a program that can provide a temporal difference of a recognition target as a topic.

According to the present disclosure, the comparison unit that compares the current feature amount of the recognition target and the feature amount at another time point in time series, and the visualization information that visualizes the difference of the recognition target based on the comparison result of the comparison unit An information processing apparatus including a generation unit that generates

According to the present disclosure, the current feature amount of the recognition target is compared with the feature amount at another time point in time series, and the visualization information that visualizes the difference of the recognition target based on the comparison result is generated. And a control method including the above.

According to the present disclosure, the computer visualizes the difference of the recognition target based on the comparison result of the comparison unit comparing the current feature amount of the recognition target and the feature amount at another time point in time series We propose a program for functioning as a generation unit that generates the visualization information.

As described above, according to the present disclosure, it is possible to provide a temporal difference of a recognition target as a topic.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure explaining the outline of the information processing system by one embodiment of this indication. It is a block diagram which shows an example of a structure of the server by this embodiment. It is a flowchart which shows the log collection process by this embodiment. It is a flowchart which shows the feature-value extraction process in this embodiment. It is a flowchart which shows the tendency model generation process of the feature-value of the recognition target by this embodiment. It is a flowchart which shows the difference presentation process between two points by this embodiment. It is a figure which shows the example of a screen display of the visualization information by this embodiment. It is a figure explaining the modification which displays the captured image when another user meets the other party on the screen shown in FIG. It is a figure which shows the example of visualization information which emphasized the difference part. It is a flowchart which shows the difference presentation process with the tendency by this embodiment. It is a figure which shows the example of a screen display of the visualization information by this embodiment. It is a flowchart which shows the difference presentation process according to the score by this embodiment. It is a figure explaining the presentation aspect in case the relationship between a user and the other party is not a close relationship. It is a figure explaining the presentation aspect in case the relationship between a user and the other party is a close relationship.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. 1. Overview of information processing system according to an embodiment of the present disclosure 2. Server configuration Operation processing 3-1. Data collection 3-2.2 Differences between point elements 3-3. Presentation of differences from trends 3-4. 3. Presentation of differences according to the score Summary

<< 1. Overview of Information Processing System According to One Embodiment of Present Disclosure >>
First, an overview of an information processing system according to an embodiment of the present disclosure will be described with reference to FIG. As shown in FIG. 1, in the information processing system according to the present embodiment, a server 1 that collects and stores a user's life log and a user terminal 2 that presents a difference as a topic offer are connected via a network 3. ing.

The life log is obtained by digitizing the user's life, behavior, experience, etc. into video, audio, position information, movement information, and the like. For example, the life log includes various wearable devices (smart watches, smart bands, Smart eyeglass etc.) continuously acquired. The server 1 accumulates and analyzes such a user's life log, so that a user who has met in the past, places that have been in the past, dishes that have been eaten in the past, etc. When re-recognized at a later date, the topic can be provided by presenting a difference from the past. The difference presentation can be performed by the user terminal 2, for example. The user terminal 2 is not limited to the smartphone shown in FIG. 1, but is, for example, a tablet terminal, a mobile phone terminal, a PDA (Personal Digital Assistant), a PC (Personal Computer), a portable music player, a portable game machine, or a wearable terminal ( HMD, smart eye glass, smart watch, smart band, etc.).

Also, the server 1 according to the present embodiment can predict future changes from past trends and present a difference between the present and the future and provide a topic.

The overview of the information processing system according to an embodiment of the present disclosure has been described above. Next, the configuration of the server 1 included in the information processing system of this embodiment will be described.

<< 2. Server configuration >>
FIG. 2 is a block diagram illustrating an example of the configuration of the server 1 according to the present embodiment. As shown in FIG. 2, the server 1 includes a communication unit 11, a raw data DB (database) 12, a feature amount extraction unit 13, a feature amount DB 14, a model generation unit 15, a model DB 16, a comparison unit 17, and a visualization information generation unit 18. And a prediction unit 19.

(Communication Department)
The communication unit 11 transmits / receives data by connecting to an external device by wireless / wired. For example, the communication unit 11 receives sensor information related to a recognition target and environmental sensor information from the information processing terminal 4 owned by the user. In addition, the communication unit 11 acquires predetermined information by connecting to an external server such as the multi-user model DB server 34, the general preference DB server 35, the product meta DB server 36, or the celebrity DB server 37. It is also possible. The communication unit 11 transmits the visualization information generated by the visualization information generation unit 18 to the user terminal 2.

(Raw data DB)
The raw data DB 12 is a storage unit that stores information about the recognition target and the environment when recognized, received from the information processing terminal 4 via the communication unit 11. Here, various log sensors included in the information processing terminal 4 will be specifically described.

-Information processing terminal-
The information processing terminal 4 acquires sensor information related to the recognition target and sensor information related to the surrounding environment when the target is recognized. The information processing terminal 4 includes, for example, a taste sensor 401, an odor sensor 402, a camera 403, a wireless communication unit 404, a position measurement unit 405, a barometer 406, a temperature / humidity meter 407, a clock unit 408, or an external reference information acquisition unit 409. Have Note that these sensors do not have to be provided in the same body. For example, the taste sensor 401 is provided in a tool used when eating such as chopsticks, spoons, and forks, and the odor sensor 402, camera 403, The wireless communication unit 404, the position measurement unit 405, the barometer 406, the temperature / humidity meter 407, the clock unit 408, and the external reference information acquisition unit 409 are provided in a wearable device, a smartphone, a mobile phone terminal, or the like worn by the user. Also good. The information processing terminal 4 transmits the acquired sensor information to the server 1.

Further, the information processing terminal 4 can acquire sensor information suitable for the recognition target. More specifically, for example, when the recognition target is cooking, the taste sensor 401 detects taste information (sweet, sour, salty, bitter, umami), the smell sensor 402 detects the smell of the dish, and the camera In 403, a captured image of the dish is acquired. When the recognition target is a person, the other person's odor is detected by the odor sensor 402, the captured image of the other party is acquired by the camera 403, and the other party's ID is received by the wireless communication unit 404. The wireless communication unit 404 is realized by, for example, Bluetooth (registered trademark), Wi-Fi (registered trademark), infrared communication, short-range wireless communication, and the like, and connected to a partner information processing terminal (wearable device, smartphone, etc.) An ID indicating who the other party is can be acquired. When the recognition target is a landscape, a captured image of the landscape is acquired by the camera 403.

Regardless of what the recognition target is, positional information is acquired by the position measurement unit 405 as information on the environment when the target is recognized, the atmospheric pressure is detected by the barometer 406, and the temperature / hygrometer 407 detects the temperature. Humidity is detected, and the date and time is acquired by the clock unit 408. The position measurement unit 405 is realized by, for example, a GPS (Global Positioning System) positioning unit, receives radio waves from GPS satellites, and detects the current position. In addition to the GPS, the position measurement unit 405 may detect the position by, for example, transmission / reception with Wi-Fi (registered trademark), a mobile phone / PHS / smartphone, or short-distance communication.

Also, information regarding the environment can be acquired by accessing various external servers by the external reference information acquisition unit 409. For example, when the recognition target is a dish, the external reference information acquisition unit 409 can acquire information such as a dish name and calories by accessing the server 31 storing the dish meta DB and collating the dish metadata. . In addition, the external reference information acquisition unit 409 can access the server 32 storing the weather information DB, and acquire the weather, temperature, humidity, and the like at that time from the date and time when the object is recognized. Further, the external reference information acquisition unit 409 can access the server 33 storing the place meta DB, and acquire detailed information of the place and facility from the position information at the time of object recognition.

The various log sensors provided in the information processing terminal 4 have been specifically described above. The information processing terminal 4 transmits the acquired sensor information to the server 1 at a predetermined timing. The specific example of the sensor shown in FIG. 2 is an example, and the present embodiment is not limited to this. For example, the information processing terminal 4 may further include a sound collection unit, an acceleration sensor, a geomagnetic sensor, a vibration sensor, and the like. Moreover, the information processing terminal 4 is implement | achieved by the smart phone, a smart band, smart eyeglass, HMD, a mobile telephone terminal etc., for example. Furthermore, the user terminal 2 may also function as the information processing terminal 4.

(Feature extraction unit)
The feature quantity extraction unit 13 extracts the feature quantity of the recognition target based on the sensor information related to the recognition target stored in the raw data DB 12. For example, when the recognition target is cooking, feature quantities that can be quantified from taste sensor information and odor sensor information, and calories that can be specified from cooking images are extracted. Further, when the recognition target is a person, a part that affects the impression of the person is extracted as a feature amount. For example, hairstyle classification, hair length, face roundness, estimated height, estimated waist, estimated weight, presence or absence of glasses, eyeglass color / shape / brand, presence / absence of accessories, accessory color / shape / brand, wear The color / shape / brand of the clothes being worn, the presence / absence of makeup, the characteristics of the color of the makeup (color of lipstick, presence / absence of gloss, color of foundation, etc.) Characterize product numbers, brands of shampoo, classification of behavior traps, etc. When the recognition target is a landscape, the feature amount extraction unit 13 characterizes the arrangement of buildings, signboards, mountains, seas, roads, and the like included in the landscape from the captured image of the landscape.

(Feature DB)
The feature amount DB 14 stores the feature amount of the recognition target extracted by the feature amount extraction unit 13.

(Model generator)
The model generation unit 15 generates a trend model of the feature quantity of the recognition target based on the past feature quantity of the recognition target. Thereby, it is possible to output a difference between a past tendency (for example, a color of clothes often worn) and the present, as well as a difference between two points such as the present and the past.

For example, a histogram is used as a model indicating past trends. When the recognition target is a person, the model generation unit 15 records the color of the clothes and the brand of the clothes with frequency and generates a histogram. Thereby, in the comparison part 17 mentioned later, it is output from the distribution of the said histogram as a difference whether the color and brand of the clothes currently worn by the target person are usually worn or not worn much. Can be done. In addition to clothes color and brand histograms, the trend model when the recognition target is a person can also generate histograms such as the presence of glasses, clothing types (skirts, pants, etc.), ornaments, and perfume brands. . The trend model when the recognition target is a dish includes a taste histogram based on accumulation of taste information acquired when the same dish was eaten in the past. Moreover, the model generation part 15 may calculate the model which shows the past tendency with a deviation value. In this case, the comparison unit 17 described later can determine whether the current feature amount is the same (general) or different (unusual) from the past tendency depending on whether or not the current feature amount is equal to or less than the threshold value.

Furthermore, the model generation unit 15 is not limited to generating a trend model based on past feature values when one user recognizes a recognition target, and refers to the server 34 that stores the multi-user model DB. It is also possible to generate a trend model by referring to past feature values when one or more other users recognize the target. That is, for example, when generating a trend model of the feature amount of the person A, the model generation unit 15 uses the feature amount extracted by the feature amount extraction unit 13 or the past feature amount accumulated in the feature amount DB 14 as a user. A trend model is generated using not only the feature amount based on the sensor information acquired when meeting A but also the feature amount based on the sensor information acquired from the server 34 when another user meets the person A. Is possible.

(Model DB)
The model DB 16 stores the recognition target trend model generated by the model generation unit 15.

(Comparison part)
The comparison unit 17 compares the current feature quantity of the recognition target with the past feature quantity (especially the latest past feature quantity), the past feature quantity trend, or the future predicted feature quantity, and determines the difference. Output. Before performing the comparison, it is necessary to specify whether the current recognition target is the same as the target recognized by the user in the past. When the recognition target is a person, the same person can be specified from the facial feature amount based on the face image. Further, when the other party's ID can be acquired, the same person can be identified from the matching ID. Further, when the recognition target is a dish, it can be specified whether the dish is the same menu from the store location, the place meta (store name), the dish image, and the like. Further, when the recognition target is a landscape, the same landscape can be specified from position information, orientation information, image feature amount, and the like. When the comparison unit 17 can identify the same recognition target, the current recognition target feature amount, the past feature amount (particularly the latest past feature amount), and the past feature amount trend (past model) Or, if the difference between the feature amounts is equal to or greater than a predetermined value by comparing with the predicted feature amount in the future, the difference is output as a comparison result.

(Visualization information generator)
The visualization information generation unit 18 generates difference visualization information to be presented to the user based on the difference output from the comparison unit 17. For example, the visualization information generation unit 18 may acquire a past image and a current image of the difference part from the raw data DB 12, and may generate a display image in which these are arranged and an image that emphasizes the difference part as the visualization information (FIG. 9). In addition, the visualization information generation unit 18 may highlight the portion indicating the current feature in the histogram indicating the past tendency of the difference portion (see FIG. 11). Moreover, the visualization information generation part 18 may display the date and time at that time with the past image of a difference part. In addition, the visualization information generation unit 18 uses a slider on the time axis to change the image recognition target gradually using the past captured image and the current captured image of the recognition target stored in the raw data DB 12. A screen that can be viewed while switching may be generated (see FIG. 7).

Further, the visualization information generation unit 18 may generate not only a screen for pointing out changes in the difference portion but also a screen including detailed information of the difference portion. For example, when the other party (recognition target) wears a product (glasses, hat, shoes, bag, accessory, etc.) different from the previous meeting, the product is output as a difference by the comparison unit 17. At this time, in order to identify what the product is, the visualization information generation unit 18 accesses the server 6 storing the product meta DB, refers to the product metadata, and details the product (product name, Brand, price range, release date, etc.). In addition, the visualization information generation unit 18 can refer to the product metadata to link to a purchase site for the product or present related products.

Further, the visualization information generation unit 18 can also generate a screen that points out the difference between the difference portion and “general preference” as advice. The general preference data can be acquired from the server 35 that stores the general preference information DB. The visualization information generation unit 18 can provide a topic on how to change the recognition target by pointing out the difference from “general preference” with respect to the change of the recognition target. General preferences include information such as recent trends and popular items obtained from fashion magazines, color schemes that match the color palette, and the ratio of base and accent colors. In addition, information obtained by generalizing the multi-user model and general common sense (ceremonial manners) are also included. Further, regarding the interior of the room, information on the color arrangement and arrangement such as unifying white to make the room look wider, arranging small furniture to make the room look wider, and using warm-colored lighting in the dining room is included. In addition, the hair style includes information such as how to bundle the hair according to the length of the hair, how to use the hair clip, the hair style that matches the shape of the face, and the hair style that matches the age.

Further, the visualization information generation unit 18 may display a calorie display estimated from an image, weight, age, height, and the like as additional information of the difference portion. Moreover, the visualization information generation part 18 may display the celebrity when the characteristic after a change is similar to a celebrity. The celebrity information can be obtained by accessing the server 37 storing the celebrity information DB.

As described above, the visualization information generation unit 18 not only points out the difference portion but also includes detailed information on the difference portion, advice based on the difference from “general preference”, and additional information on the difference portion. The topic provided to the user can be enriched by generating.

The visualization information generation unit 18 transmits the generated visualization information (an example of information related to the difference of recognition targets based on the comparison result of the comparison unit 17) to the user terminal 2 via the communication unit 11 and controls to notify the user. It also functions as a notification control unit.

(Prediction unit)
The prediction unit 19 predicts a future feature amount to be recognized, and outputs a prediction result to the comparison unit 17 or the visualization information generation unit 18. Thereby, not only the difference with the past but the difference with the future is performed in the comparison part 17, and the screen containing the future feature-value can be produced | generated in the visualization information generation part 18. FIG. As a first prediction method by the prediction unit 19, it is possible to predict future feature values by extrapolation based on the past and present feature values of two or more recognition targets. For example, based on the amount of hair two years ago, one year ago, and the current hair, the amount of hair one year later is predicted by a linear model. Further, as a second prediction method by the prediction unit 19, it is possible to predict a future feature value by using a trend model of a past feature value to be recognized. For example, the amount of wrinkles that can be generated when aging is further predicted based on a trend model indicating how to increase the number of wrinkles on the face to be recognized. In this case, the visualization information generation unit 18 can generate a face image based on the prediction result of the amount of wrinkles that can be generated when aging and include it in the presentation screen.

The configuration of the server 1 according to the present embodiment has been specifically described above. The server 1 is equipped with a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and a nonvolatile memory, and controls each component of the server 1.

<< 3. Action processing >>
Subsequently, the operation processing of the information processing system according to the present embodiment will be specifically described with reference to FIGS.

<3-1. Data collection>
First, various data collection according to the present embodiment will be described with reference to FIGS. FIG. 3 is a flowchart showing log collection processing according to the present embodiment. The processing shown in FIG. 3 is executed in the information processing terminal 4 (wearable device or the like) provided with various log sensors. Specifically, the information processing terminal starts log collection periodically or when a trigger is generated by an external event (S103), and identifies a recognition target (S106).

Next, the information processing terminal collects sensor information suitable for the recognition target (S109). For example, if the recognition target is cooking, information is acquired by the taste sensor 401, the odor sensor 402, and the camera 403. If the recognition target is a person, the odor sensor 402, the camera 403, and the wireless communication unit 404 acquire information.

Next, the information processing terminal collects environmental sensor information at the time of recognition (S112). The environmental sensor information is information that is not based on the recognition target. For example, current position information acquired by the position measuring unit 405, current temperature / humidity acquired by the temperature / humidity meter 407, and acquired by the clock unit 408. This corresponds to the current date and time.

Next, the information processing terminal attempts to expand log data (S115). That is, the external reference information acquisition unit 409 accesses various external servers (such as the cooking meta DB server 31, the weather information DB server 32, or the location meta DB server 33) to recognize information to be recognized and the environment at the time of recognition. Get more information.

Then, the information processing terminal transmits all the acquired information to the server 1 and stores it in the raw data DB 12.

Subsequently, the feature amount formation in the server 1 will be described with reference to FIG. FIG. 4 is a flowchart showing the feature amount extraction processing in the present embodiment. As shown in FIG. 4, first, the feature amount extraction unit 13 of the server 1 reads sensor information to be recognized from the raw data DB 12 (S123).

Next, the feature quantity extraction unit 13 extracts the feature quantity of the recognition target based on the sensor information of the recognition target (S126).

Then, the feature quantity extraction unit 13 stores the extracted feature quantity in the feature quantity DB 14 (S129). The feature amount extraction described above is performed in real time when the user recognizes the target.

Subsequently, generation of a trend model in the server 1 will be described with reference to FIG. FIG. 5 is a flowchart showing the trend model generation process of the feature quantity of the recognition target according to the present embodiment. As shown in FIG. 5, first, the model generation unit 15 of the server 1 checks whether or not a trend model having a predetermined feature amount to be recognized has already been generated (S133).

Next, when already generated (“No” in S133), the model generation unit 15 reads a trend model (past model) that has already been generated from the model DB 16 (S136).

On the other hand, if it has not been generated yet (“Yes” in S133), the model generation unit 15 initializes a trend model of a predetermined feature quantity.

Next, the model generation unit 15 reads a predetermined feature amount to be recognized from the feature amount DB 14 (S142). Here, feature quantities that have not yet been reflected in the trend model (for example, feature quantities newly acquired from the recognition target at present) are read.

Next, the model generation unit 15 updates the trend model of the feature amount based on the read feature amount (S145).

Next, the model generation unit 15 writes the updated tendency model in the model DB 16 (S148).

Next, if necessary, the model generation unit 15 synchronizes the user's model DB in the server 34 storing the multi-user model DB with the updated tendency model. The server 34 stores a trend model to be recognized and a feature amount DB for each user.

The generation of the trend model described above is generated for each of a plurality of feature amounts extracted from the recognition target. For example, trend models such as a feature of a hairstyle to be recognized, a feature of clothes, a feature of belongings, and a feature of body shape can be generated.

Subsequently, the difference presentation operation process and visualization information examples according to the present embodiment will be described with reference to FIGS. Each operation process of difference presentation described below can be performed in real time when the user recognizes the target.

<3-2.2 Presentation of differences between point elements>
First, difference presentation between two point elements (that is, the present and the latest past or future) according to the present embodiment will be specifically described with reference to FIGS.

FIG. 6 is a flowchart showing a difference presentation process between two points according to the present embodiment. As shown in FIG. 6, first, the comparison unit 17 reads the current feature amount to be recognized and the most recent feature amount from the feature amount DB 14 (S203).

Next, the comparison unit 17 compares feature amounts and calculates a difference (S206).

Next, the comparison unit 17 outputs the comparison result (difference part) to the visualization information generation unit 18 (S209).

Next, the visualization information generation unit 18 acquires information on the difference part from the raw data DB 12 (S212).

Next, the visualization information generation unit 18 generates a comment that points out the difference (S215). At this time, the visualization information generation unit 18 may acquire detailed information and related information of the difference part from the external server, and generate a comment including these.

Subsequently, the visualization information generation unit 18 generates visualization information based on the acquired information on the difference part and the generated comment, presents the generated visualization information to the user terminal 2 via the communication unit 11, and performs the difference. The location is notified (S218).

Here, a specific example of the visualization information presented to the user by the visualization information generation unit 18 will be described with reference to FIG. FIG. 7 is a diagram showing a screen display example of visualization information according to the present embodiment. As shown in FIG. 7, for example, the visualization information can be viewed using the past captured image and the current captured image of the recognition target while switching the image recognition target gradually with a slider on the time axis. Realized by screen. Screens 22-1 to 22-4 in FIG. 7 are screens that can be switched by moving a slider on the time axis.

Specifically, for example, when the most recent information (such as a captured image of the partner when the user first meets) is 1972 when the user recognizes the target, the most recent time axis on the 1972 is 1972 as shown on the screen 22-1. Set to the year. When the slider is moved to 1972, information such as the date, place, weather, etc. at the time of recognition is displayed together with the captured image (face image) to be recognized acquired in 1972. Thereby, the user can talk about the first meeting with the other party.

If the next time the user met the same partner in 2014, information on the partner between 1972 and 2014 is not accumulated, but as shown in screen 22-2, the slider is moved to 1972. When moving between the year and 2014, a composite image generated based on the past image may be displayed. On the screen 22-2, a composite image in which the past image of 1972 and the past image of 2014 are superimposed is displayed.

Next, as shown in the screen 22-3, when the slider on the time axis is moved to the current 2014, the recognition target date and time, place, weather, etc., along with the captured image (face image) to be recognized acquired in 2014, etc. Information is also displayed. Here, when the user meets the same partner as of 2014, the comparison unit 17 compares the feature amount of the latest past partner with the feature amount of the current partner, and visualizes the difference based on the comparison result. A screen 22-3 is displayed. For example, if the most recent past meeting with the same partner is a week ago, the screen 22-3 contains information such as the face image of the partner one week ago, the date and place of meeting, and another week before And the difference in the feature amount (such as a change in hairstyle). Thereby, the user can talk about the fact that the partner's hairstyle has changed since the last time they met. When there is no difference, it may be indicated that there is no change.

Subsequently, as shown in the screen 22-4, when the slider on the time axis is moved to the future such as 2020, information based on the future feature amount predicted by the prediction unit 19 is displayed. For example, the face image generated by the visualization information generation unit 18 based on the number of wrinkles, the amount of hair, the color of hair, the change in sagging face, and the like indicated by the future prediction result is displayed.

In the foregoing, an example of visualization information that displays past images to be recognized according to a time axis has been described. In addition, on the screen 22-2 shown in FIG. 7, the face image of the partner while the user is not meeting the partner is displayed based on the past image, but this embodiment is not limited to this. The visualization information generation unit 18 may acquire and display a captured image of a partner when another user meets the partner from the server 34 that stores the multi-user model DB. Hereinafter, a description will be given with reference to FIG.

FIG. 8 is a diagram for explaining a modified example of displaying a captured image when another user meets the other person on the screen 22-2 shown in FIG. As shown in FIG. 8, on the screen 22-2 ', the slider on the time axis is moved between 1972 and 2014 (for example, around 2000). Since the user does not meet the other party at the time indicated by the slider, the past image (captured image) of the other party is not stored in the raw data DB 12, but another user (for example, a common friend XX) meets the other party. In this case, the visualization information generation unit 18 acquires the partner image captured when another user meets from the server 34 and displays it on the screen 22-2 ′. In addition, the visualization information generation unit 18 displays a past image of the other party and a comment (annotation) such as “I have not met around 2000. An image when a common friend XX meets” is displayed. By attaching, it notifies the user that it is information when other users meet.

Next, another example of visualization information will be described with reference to FIG. FIG. 9 is a diagram illustrating an example of visualization information in which a different portion is emphasized. For example, when the color of the frame of the other party's glasses that the user has met has changed since yesterday, which is the most recent past, the glasses frame that is the difference is emphasized and notified to the user. Specifically, for example, as shown in FIG. 9, the current captured image of the partner and the latest captured image are displayed side by side on the screen 22-6, and further, an enlarged image of the difference portion and a comment regarding the difference portion Is displayed. Thereby, the user can talk about the fact that the color of the frame of the other person's glasses has changed since the last meeting.

<3-3. Presenting differences with trends>
Next, presentation of differences from the trend according to the present embodiment will be specifically described with reference to FIGS. In the above-described embodiment, the difference between two point elements on the time series such as the present and the latest past or the future is presented. However, the present embodiment is not limited to this, and the recognition target generated by the model generation unit 15 It is also possible to present the difference between the past feature quantity trend and the present one.

FIG. 10 is a flowchart showing a difference presentation process with the trend according to the present embodiment. As shown in FIG. 10, first, the comparison unit 17 reads the current feature quantity to be recognized from the feature quantity DB 14 (S233).

Next, the comparison unit 17 reads the past tendency to be recognized from the model DB 16 (S236).

Next, the comparison unit 17 compares the current feature amount with the past trend and calculates a difference (S239).

Next, the comparison unit 17 outputs the comparison result (difference part) to the visualization information generation unit 18 (S242).

Next, the visualization information generation unit 18 acquires information on the difference part from the raw data DB 12 (S245).

Next, the visualization information generation unit 18 generates a comment that points out the difference (S248). At this time, the visualization information generation unit 18 may acquire detailed information and related information of the difference part from the external server, and generate a comment including these.

Subsequently, the visualization information generation unit 18 generates visualization information based on the acquired information on the difference part and the generated comment, presents the generated visualization information to the user terminal 2 via the communication unit 11, and performs the difference. The location is notified (S251).

Here, a specific example of the visualization information presented to the user by the visualization information generation unit 18 will be described with reference to FIG. FIG. 11 is a diagram illustrating a screen display example of visualization information according to the present embodiment. As illustrated in FIG. 11, for example, the visualization information may be realized by a screen that highlights a portion indicating the current feature in a histogram indicating the past tendency of the difference portion. Specifically, as shown in the screen 22-7 in FIG. 11, for example, a histogram indicating the past tendency (wear frequency for each color) of the color of the clothes to be recognized is a representative past when classified for each color. It is pointed out that the color of the currently worn clothes is less worn than the past tendency. At this time, the visualization information generation unit 18 may display the image when the corresponding color was previously worn and the image of the currently worn clothing side by side.

<3-4. Difference presentation according to the score>
Next, difference presentation according to the score according to the present embodiment will be specifically described with reference to FIGS. In each embodiment described above, the difference portion calculated by the comparison unit 17 is presented to the user as it is. However, the presentation method according to the present embodiment is not limited to this, and for example, the size of the difference and the content attribute of the difference (negative / positive) ) Can be presented more appropriately by changing the presentation mode (display mode).

FIG. 12 is a flowchart showing the difference presenting process according to the score according to the present embodiment. As shown in FIG. 12, first, the comparison unit 17 reads the current feature quantity to be recognized and the feature quantity of the past past from the feature quantity DB 14 (S303).

Next, the comparison unit 17 compares feature amounts and calculates a difference (S306).

Next, the comparison unit 17 outputs the comparison result (difference part) and the difference score to the visualization information generation unit 18 (S209). The difference score is a score indicating the magnitude of the difference from the current feature amount. When there are a plurality of differences, a difference score is calculated for each difference and output to the visualization information generation unit 18.

Next, the visualization information generation unit 18 acquires information on the difference part from the raw data DB 12 (S312).

Next, the visualization information generating unit 18 calculates a positive-negative score of the difference (S315). The positive-negative score is a continuous value that indicates whether the difference (change) output by the comparison unit 17 is positive or negative. For example, “score: −100” is very negative. The difference “score: 50” means a relatively positive difference.

Next, the visualization information generation unit 18 determines a difference presentation mode (highlighting degree, arrangement, etc. on the display screen) based on the relationship between the user and the other party, the difference score, and the positive-negative score (S318). . For example, the visualization information generation unit 18 highlights and displays a difference with a large difference score or a difference with a high degree of positiveness. If the difference score is large but negative, the relationship with the partner (familiarity, gender, age) Decide to present with the degree of emphasis and placement according to proximity. In addition, when there are a plurality of differences and the relationship with the partner is intimate, the visualization information generation unit 18 increases the difference score in the descending order even if the positive-negative score is negative (eg, score: −50 or less). May be presented. On the other hand, if the relationship with the partner is not intimate, the visualization information generation unit 18 may preferentially display (arrange) the positive-negative score of 20 or more in the order of the difference score.

Next, the visualization information generation unit 18 generates a comment that points out the difference (S321). At this time, the visualization information generation unit 18 may acquire detailed information and related information of the difference part from the external server, and generate a comment including these.

Subsequently, the visualization information generation unit 18 generates visualization information based on the acquired information on the difference part and the generated comment, presents the generated visualization information to the user terminal 2 via the communication unit 11, and performs the difference. The location is notified (S324).

Here, a specific example of the visualization information presented to the user by the visualization information generation unit 18 will be described with reference to FIGS. 13 and FIG. 14, as an example, a description will be given of a difference in presentation mode according to the relationship with the partner when the comparison unit 17 outputs a change in hair color and a change in weight as differences. The change in hair color is, for example, a change from brown to black, and the difference score 20 and the positive-negative score indicate neutral changes. The change in body weight is an increase of, for example, an estimated 20 kg, and the difference score 30 and the positive-negative score indicate a negative change.

FIG. 13 is a diagram for explaining a presentation mode when the relationship between the user and the other party is not a close relationship. When the relationship between the user and the partner is not a close relationship, the visualization information generation unit 18 indicates a change in hair color that is a neutral change even if the difference score is low, as shown in a screen 22-8 in FIG. 50 is highlighted with large characters so as to stand out more, and the display order is also displayed at the top. On the other hand, the comment 51 pointing out a change in weight, which is a negative change even if the difference score is high, is displayed in small letters so as to be inconspicuous and the display order is displayed in the lower order. Thus, the user can talk about a neutral change such as a change in hair color.

FIG. 14 is a diagram for explaining a presentation mode when the relationship between the user and the partner is a close relationship. When the relationship between the user and the partner is a close relationship, the visualization information generation unit 18 points out a change in body weight with a high difference score even if it is a negative change, as shown in a screen 22-9 in FIG. The comment 52 is highlighted with large characters so as to stand out more, and the display order is also displayed at the top. On the other hand, the comment 53 indicating the change in hair color, which is a change with a low difference score, is displayed in small letters and in the lower order. Thereby, even if it is a negative change, such as a change in weight, the user can focus on a difference with a large change.

This completes the explanation of the difference presentation according to the score. In the flow shown in FIG. 12, the comparison between two point elements is performed. However, the present embodiment is not limited to this, and the comparison result between the current and past trends is similarly scored (difference score, positive-negative score). ), The presentation mode may be controlled.

<< 4. Summary >>
As described above, in the information processing system according to the embodiment of the present disclosure, it is possible to provide a temporal difference of a recognition target as a topic. Thereby, the user can notice the change of recognition object. When the recognition target is a person, the change can be used as a topic when meeting for a long time. In addition, when the recognition target is a landscape, a new discovery is born by seeing a time-series change in the landscape.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present technology is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

For example, a computer program for causing the functions of the server 1 and the user terminal 2 to be performed on hardware such as the CPU, ROM, and RAM incorporated in the server 1 and the user terminal 2 described above can be created. A computer-readable storage medium storing the computer program is also provided.

Further, the notification of the information related to the difference of the recognition target based on the comparison result of the comparison unit 17 is not limited to the display on the user terminal 2 of the visualization information generated by the visualization information generation unit 18 as described above. The server 1 may have a function of a notification control unit that controls the user terminal 2 to notify the user of information related to the recognition target difference based on the comparison result of the comparison unit 17 by voice or the like.

Further, the user terminal 2 may have at least a part of the configuration of the server 1 according to the present embodiment.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

In addition, this technique can also take the following structures.
(1)
A comparison unit that compares the current feature value of the recognition target with the feature value at another point in time series,
A generation unit that generates visualization information that visualizes the difference between the recognition targets based on the comparison result of the comparison unit;
An information processing apparatus comprising:
(2)
The information processing apparatus according to (1), wherein the feature quantity of the recognition target is extracted from sensor information detected when a user faces the recognition target.
(3)
The information processing apparatus according to (1) or (2), wherein the comparison unit compares a current feature amount of the recognition target with a past feature amount.
(4)
The information processing apparatus according to (3), wherein the comparison unit compares a current feature amount of the recognition target with a feature amount in the past.
(5)
The information processing apparatus according to (3), wherein the comparison unit compares a current feature amount of the recognition target with a model indicating a tendency of the past feature amount of the recognition target.
(6)
The information processing apparatus according to (5), wherein the past feature amount model is generated based on the past feature amount of the recognition target.
(7)
The model of the past feature amount is generated based on the past feature amount of the recognition target extracted from sensor information detected when one or more other users face the recognition target. ).
(8)
The information processing apparatus according to (1) or (2), wherein the comparison unit compares a current feature amount of the recognition target with a future feature amount of the recognition target.
(9)
The information processing apparatus according to (8), wherein the future feature amount is predicted based on a past feature amount model of the recognition target and a current feature amount.
(10)
The information processing apparatus according to any one of (1) to (9), wherein the feature amount is extracted from sensor information suitable for the recognition target.
(11)
The information processing apparatus according to (10), wherein when the recognition target is a person, the feature amount is extracted from a captured image or odor information of the person detected by a camera sensor or an odor sensor.
(12)
The feature amount is extracted from the taste information, odor information, or captured image of the dish detected by a taste sensor, an odor sensor, or a camera sensor when the recognition target is a dish. Information processing device.
(13)
The information processing apparatus according to (10), wherein the feature amount is extracted from a captured image of the scenery detected by a camera sensor when the recognition target is a scenery.
(14)
The information processing apparatus according to any one of (1) to (13), wherein the generation unit generates, as visualization information, a display screen that emphasizes the difference portion to be recognized.
(15)
The generation unit determines a presentation mode of the difference according to the magnitude of the difference of the recognition target, the positive-negative attribute of the difference, and the relationship between the recognition target and the user, (1) to (14 The information processing apparatus according to any one of the above.
(16)
The information processing apparatus according to any one of (1) to (15), further including a notification control unit that controls to notify a user of information related to the recognition target difference based on a comparison result of the comparison unit. The information processing apparatus described.
(17)
Comparing the current feature quantity of the recognition target with the feature quantity at another point in time,
Generating visualization information that visualizes the difference of the recognition target based on the comparison result;
Including a control method.
(18)
Computer
A comparison unit that compares the current feature value of the recognition target with the feature value at another point in time series,
A generation unit that generates visualization information that visualizes the difference between the recognition targets based on the comparison result of the comparison unit;
Program to function as

1 server 11 communication unit 12 raw data DB
13 Feature Extraction Unit 14 Feature DB
15 Model generator 16 Model DB
17 Comparison unit 18 Visualization information generation unit 19 Prediction unit 2 User terminal 3 Network 31 Cooking meta DB server 32 Weather information DB server 33 Location meta DB server 34 Multi-user model DB server 35 General preference DB server 36 Product meta DB Server 37 Celebrity DB server

Claims

A comparison unit that compares the current feature value of the recognition target with the feature value at another point in time series,
A generation unit that generates visualization information that visualizes the difference between the recognition targets based on the comparison result of the comparison unit;
An information processing apparatus comprising:
The information processing apparatus according to claim 1, wherein the feature quantity of the recognition target is extracted from sensor information detected when a user faces the recognition target.
The information processing apparatus according to claim 1, wherein the comparison unit compares a current feature amount of the recognition target with a past feature amount.
The information processing apparatus according to claim 3, wherein the comparison unit compares the current feature value of the recognition target with a feature value in the past.
The information processing apparatus according to claim 3, wherein the comparison unit compares a current feature amount of the recognition target with a model indicating a tendency of a past feature amount of the recognition target.
The information processing apparatus according to claim 5, wherein the model of the past feature quantity is generated based on the past feature quantity of the recognition target.
The past feature amount model is generated based on past feature amounts of the recognition target extracted from sensor information detected when one or more other users face the recognition target. The information processing apparatus described in 1.
The information processing apparatus according to claim 1, wherein the comparison unit compares a current feature amount of the recognition target with a future feature amount of the recognition target.
The information processing apparatus according to claim 8, wherein the future feature amount is predicted based on a past feature amount model of the recognition target and a current feature amount.
The information processing apparatus according to claim 1, wherein the feature amount is extracted from sensor information suitable for the recognition target.
The information processing apparatus according to claim 10, wherein when the recognition target is a person, the feature amount is extracted from a captured image or odor information of the person detected by a camera sensor or an odor sensor.
The feature amount is extracted from taste information, odor information, or a captured image of the dish detected by a taste sensor, an odor sensor, or a camera sensor when the recognition target is a dish. Information processing device.
The information processing apparatus according to claim 10, wherein the feature amount is extracted from a captured image of the scenery detected by a camera sensor when the recognition target is a scenery.
The information processing apparatus according to claim 1, wherein the generation unit generates a display screen that highlights the difference portion to be recognized as visualization information.
2. The information according to claim 1, wherein the generation unit determines a presentation mode of the difference according to a difference size of the recognition target, a positive-negative attribute of the difference, and a relationship between the recognition target and the user. Processing equipment.
The information processing apparatus according to claim 1, further comprising a notification control unit that controls to notify a user of information related to the recognition target difference based on a comparison result of the comparison unit.
Comparing the current feature quantity of the recognition target with the feature quantity at another point in time,
Generating visualization information that visualizes the difference of the recognition target based on the comparison result;
Including a control method.
Computer
A comparison unit that compares the current feature value of the recognition target with the feature value at another point in time series,
A generation unit that generates visualization information that visualizes the difference between the recognition targets based on the comparison result of the comparison unit;
Program to function as