JP7291177B2 - Terminal device, information processing method and information processing program - Google Patents

Description

The present invention relates to a terminal device, an information processing method and an information processing program.

A technique is disclosed for sending teacher data to a server device for causing the server device to perform machine learning.

Japanese Patent No. 6880290

Machine learning is generally performed by a server device, as in the conventional technology described above. However, if the data contains personal information, it may not be desirable to send the data to an external server device such as a company. In addition, since there are various restrictions on the transmission and reception of data including personal information, it may not be easy to freely transmit and receive data.

Therefore, in recent years, attention has been focused on on-device machine learning, such as federated learning, in which data is distributed to individual terminal devices without being aggregated in a server device. . However, when performing machine learning on individual terminal devices using sensor data that is affected by the shape and performance of the terminal device, problems arise when changing models or when transferring data between multiple devices. In addition, since the data includes personal information, there is also the problem that the data cannot be transmitted to an external server device such as a company for processing.

The present application has been made in view of the above, and an object of the present application is to seamlessly transfer data between individual terminal devices that perform machine learning.

A terminal device according to the present application is a terminal device owned by a user, and includes an acquisition unit for collecting context information of the user by a function of the terminal device as first context information, and A receiving unit that receives the user's context information collected by a function of the other terminal device from another terminal device owned by the user, and a correspondence relationship between the first context information and the second context information. and an estimating unit that learns to estimate a conversion rule between the first context information and the second context information; a conversion unit that converts the second context information into the first context information according to the conversion rule; and a learning unit that performs machine learning using the context information of the user as learning data.

According to one aspect of the embodiments, data can be seamlessly handed over between individual terminal devices that perform machine learning.

FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 4 is a sequence diagram illustrating a processing procedure according to the embodiment; FIG. 5 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing a terminal device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. In addition, the terminal device, the information processing method, and the information processing program according to the present application are not limited by this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Outline of information processing method]
First, an outline of an information processing method performed by an information processing apparatus according to an embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. Note that in FIG. 1, a case in which data is seamlessly handed over between individual terminal devices that perform machine learning will be described as an example.

As shown in FIG. 1 , the information processing system 1 includes a terminal device 10 and an information providing device 100 . The terminal device 10 and the information providing device 100 are communicably connected to each other by wire or wirelessly via a network N (see FIG. 2).

The terminal device 10 is a smart device such as a smartphone or tablet used by a user U (user), and communicates with an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). It is a portable terminal device capable of performing The terminal device 10 has a screen such as a liquid crystal display and has a touch panel function, and the user U can perform a tap operation, a slide operation, a scroll operation, or the like with a finger, a stylus, or the like, and display data such as contents. Accepts various operations for . An operation performed on an area where content is displayed on the screen may be an operation on the content. In addition, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC as well as a smart device.

The information providing device 100 cooperates with the terminal device 10 of each user U, and provides the terminal device 10 of each user U with API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various It is an information processing device that provides data, and is realized by a server device, a cloud system, or the like.

Also, the information providing device 100 may be an information processing device that provides some web service online to the terminal device 10 of each user U. FIG. For example, the information providing apparatus 100 provides Web services such as Internet connection, search service, SNS (Social Networking Service), electronic commerce, electronic payment, online game, online banking, online trading, accommodation/ticket reservation, video/music distribution, Services such as news, maps, route search, route guidance, route information, operation information, and weather forecast may be provided. In practice, the information providing apparatus 100 may cooperate with various servers that provide web services as described above, mediate web services, or take charge of web service processing.

As shown in FIG. 1, in the present embodiment, the information providing device 100 (server side) employs federated learning, and the terminal device 10 (terminal side) of each user U A shared model (core model) is provided (step S1).

Based on the shared model, the terminal device 10 performs on-device machine learning using local data (context information of the user U) collected using a sensor or the like mounted on the terminal device 10 as learning data to determine patterns. Inference is performed from the input data using the pattern (step S2).

Subsequently, the terminal device 10 transmits the inference result obtained as the inference result and the difference parameter between the shared model obtained as the learning result to the information providing device 100 (server side) (step S3). At this time, the terminal device 10 may transmit the inference result in the form of vector data converted (vectorized) into a multidimensional vector. Vector data is irreversible information, and the original data cannot be restored from the vector values of the vector data. For example, the terminal device 10 may perform so-called embedding to generate an embedding vector based on the inference result. Encrypting and transmitting vector data, which is transmission data, is possible within the scope of common technical knowledge. Alternatively, tunneling may be used to establish a closed virtual direct line between the terminal side and the server side. For example, a VPN (Virtual Private Network) may be constructed between the terminal side and the server side.

The information providing device 100 performs an aggregation process of the inference results received from the terminal device 10 of each user U (step S4). Aggregation processing includes arithmetic processing such as statistics and analysis.

Subsequently, the information providing device 100 corrects the sharing model based on the difference parameters received from the terminal device 10 of each user U (step S5). Then, the information providing device 100 provides the modified sharing model (or the difference parameter from the previous sharing model) to the terminal device 10 of each user U (returns to step S1). That is, the information providing device 100 updates and upgrades the sharing model provided to the terminal device 10 of each user U. FIG.

Note that if the information providing apparatus 100 does not need to update the sharing model, the process of step S5 may not be performed. In this case, in step S3, the terminal device 10 should transmit only the inference result.

As described above, when the terminal device 10 (terminal side), not the information providing device 100 (server side), performs machine learning using the context information of the user U as learning data, when changing the model of the terminal, Also, model/data handover is required between multiple devices.

If there is no handover of models/data when changing models or between multiple devices, the legacy of the past will be ruined, and there will be the problem of having to build a model from scratch with new data. Therefore, new devices and other devices are required to appropriately share models/data of previous (past) devices.

Here, since the local data is a mass of personal information, there are users U who are reluctant to pass it to the information providing apparatus 100 (server side). The reason why the terminal device 10 (terminal side) rather than the information providing device 100 (server side) performs on-device machine learning is that the user U does not want the company side to know the data. I don't want to send data to Therefore, when data is transferred between devices owned by user U, it is necessary to devise a way to never transfer data to the information providing apparatus 100 (on the server side).

In this embodiment, among the terminal devices 10 of the user U, the terminal device 10A, which is the old terminal (old device), transfers the model/data directly to the terminal device 10B, which is the new terminal (new device) (step S6 ). At this time, authentication may be performed between the old terminal (old device) and the new terminal (new device). The authentication may be FIDO (Fast Identity Online) authentication. Then, only when the authentication is successful, the terminal device 10A, which is the old terminal (old device), transfers the model/data directly to the terminal device 10B, which is the new terminal (new device). It should be noted that it is possible within the scope of common technical knowledge to encrypt and transmit the model/data to be transmitted.

In this embodiment, in principle, there is no exchange of local data between the terminal device 10 (terminal side) and the information providing device 100 (server side). Methods for sharing data between devices owned by the user U include the following methods.

For example, the user U passes data from an old terminal (old device) to a new terminal (new device) by P2P (Peer to Peer). At this time, the user U may transmit data from the old terminal to the new terminal by short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network).

Alternatively, the user U may migrate data from the old terminal to the new terminal via the home LAN. For example, the user U transfers from the old terminal to the new terminal, a hub like a smart home, a home router (a local relay device managed by the user U), a personal PC or a home server (home server), or the like. You can transfer data via At this time, the user U can back up the data from the old terminal to a hub such as a smart home, a home router (a local relay device managed by the user U), a personal PC, a home server, or the like. good. Note that the backup of data from the old terminal may be performed in real time when the data is acquired (real-time support). The new terminal may then acquire the backed up data.

Alternatively, user U compresses the data on the old terminal into a file in a format that can be decompressed only by the user (encryption that cannot be deciphered by the company), and downloads and decompresses the compressed file on the new terminal side. good too. For example, the old terminal may encrypt data with a private key and only provide the public key with the encrypted data to the new terminal. In practice, the user U may transfer data from the old terminal to the new terminal via a closed network.

Thus, the method of transferring/sharing data between devices owned by user U may be P2P, home LAN (home hub format), or file compression.

However, the new device has some differences from the previous device. For example, new sensors are added/sensors are missing. Also, the performance of the sensor may be improved/decreased. Furthermore, even if the sensors are the same, the values obtained may differ due to differences in the shape of the terminals.

That is, the new device and the previous device may differ in the type and performance of the sensors installed in each device. Moreover, since the shape and size of the device itself change, the measurement results of the sensor may also differ. Therefore, when the model is passed from the previous device to the new device, the estimation accuracy of the model as it is decreases. Therefore, when the specs of the device increase, the problem of what to do with learning arises. It also raises the issue of how to apply the original, low-accuracy model to new devices.

In this way, for example, when the user U changes the model of the mobile terminal, if the new terminal appropriately uses the user U-dedicated model that was used in the old terminal, the new terminal device can be used for the type of sensor, for example. As the number of devices increases, the types of data that can be acquired are added, and the accuracy of data increases due to improvements in sensor performance (and vice versa). When using it, it is necessary to control the data input to the model or modify the model.

In the present embodiment, among the terminal devices 10 of the user U, the terminal device 10B, which is the new terminal (old device), stores the local data of the terminal device 10A, which is the old terminal (old device), and the new terminal (new device). and the local data of the terminal device 10B, and guesses the conversion rule (step S7). For example, the terminal device 10B learns the correspondence relationship between the local data received from the terminal device 10A and the local data collected by itself, and associates the local data received from the terminal device 10A with the local data collected by itself. Infer the mapping (transformation rule).

Subsequently, among the terminal devices 10 of the user U, the terminal device 10B, which is the new terminal (new device), converts the local data of the terminal device 10A, which is the old terminal (old device), into the new data according to the estimated conversion rule. It is converted into local data of the terminal device 10B, which is a terminal (new device) (step S8). That is, the terminal device 10B converts the local data received from the terminal device 10A into data of the same format as the local data collected by itself.

Here, as a method of converting the local data of the old device into data of the same format as the local data of the new device, there are two methods, a "data-side approach" and a "model-side approach", as described below.

(data-side approach)
(1) Dictionary lookup Prepare a correspondence table of data of the new device and the old device in advance, and convert the data of the old device into the data of the new device according to the correspondence table when exchanging data or sharing a model. For example, when sharing old device data including device acceleration with a new device, the original device acceleration value is multiplied by 0.8 and learned together with the new data. That is, if the context of the old device is "XX" and the context of the new device is "YY", a conversion formula of "XX→YY" is created to convert the data.

(2) Judgment based on user performance If there is a change in the sensor data of the new device and the old device, but there is no change in the user's location information or action history, the user's behavior is always the same, and it can be obtained It is assumed that only the sensor data are different. Therefore, tuning is performed so that the statistical data of the new device and the old device are consistent. Alternatively, only the normalized data values are treated, not the original raw data values. (variations, per context)
For example, if the average value of the acceleration sensor obtained during the morning commute is "120", the average value of the acceleration sensor of the new terminal is calculated to match it.

(3) Corresponding to increases and decreases in the number of sensors If the number of sensors increases in the new device, if the new device does not use the new sensor and learns in the same way as the old device as an extension of the conventional method, the data of the old device can be used as is. is also possible. However, in learning based on the user's context, it would be a waste not to use data from new sensors.

Therefore, based on the data newly obtained by the new device, the data is supplemented by assuming that the old device was equipped with the new sensor. At this time, the false sensor of the mapping is learned from the model. For example, in the data of the new device and the old device, it can be inferred that the same sensor data can be obtained at the place of the same position information. Also, if the same sensor data is obtained from the sensors installed in both the new device and the old device, if the old device was equipped with the new sensor, the new sensor will be the same as the new device. It can be inferred that sensor data is available.

If the old device cannot use the same sensor as the new device, when the data of the old device is taken over, the data is converted assuming that the old device is equipped with a new sensor. At this time, data conversion is performed taking into consideration the type of sensor of the new device. Since the old device is usually different in shape from the new device, even if the same sensor is used, the sensor response may be different.

(Approach on the model side)
A method of using only abstracted/categorized data (data with attributes that can be transferred between models) is also conceivable. However, if the measurement performance (frequency, etc.) of the device sensor is improved, it will be possible to make more detailed/accurate estimations, so the abstract data input to the model will change. In terms of on-device machine learning, past input data is re-learned with a lower weight. This is close to moving data.

In addition, in the information providing apparatus 100 (server side), along with federated learning, each user's device information and the transition of the model inherited from each user's old device to the new device (connection coefficient, etc.) ) and provide this learning result to the user's new device. For example, the information providing apparatus 100 (on the server side) learns the transition of the model of each user who changed the model from model X to model Y. FIG. The user's new device converts the model inherited from the old device based on this learning result.

Next, among the terminal devices 10 of the user U, the terminal device 10B, which is the new terminal (new device), converts the local data of the terminal device 10A, which is the converted old terminal (old device), to the new terminal (new device ) and the local data of the terminal device 10B as learning data, on-device machine learning is performed, and an inference is made based on the learning result (step S9).

As a result, the terminal device 10 of the user U can realize seamless handover of data between individual terminal devices that perform machine learning. It is also possible to change the way of handover between data and models.

In the above description, the terminal device 10A that is the old terminal (old device) and the terminal device 10B that is the new terminal (new device) are not necessarily limited to the old and new terminals when changing models (updating terminals). It may be a terminal before and after switching when switching the terminal to be used among a plurality of devices owned by the person.

For example, if the user U has both a smartphone and a tablet terminal, it is possible to dynamically transfer data and models between the smartphone and the tablet terminal at the timing of using each device. good. When a user U who is using a smartphone uses a tablet terminal, the data and model are handed over from the smartphone to the tablet terminal, and when the user uses the smartphone again, the data and model are handed over from the tablet terminal to the smartphone.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. As shown in FIG. In FIG. 2, as the terminal devices 10, a terminal device 10A that is an old terminal (old device) and a terminal device 10B that is a new terminal (new device) are shown. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Also, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to the illustrated one. For example, in FIG. 2, for simplification of illustration, as the terminal devices 10, one terminal device 10A as the old terminal (old device) and one terminal device 10B as the new terminal (new device) are shown. , this is merely an example and is not limited, and each may be two or more. For example, a terminal device 10A that is an old terminal (old device) and a terminal device 10B that is a new terminal (new device) may exist for each user.

The terminal device 10 is an information processing device used by the user U. FIG. For example, the terminal device 10 includes smart devices such as smartphones and tablet terminals, feature phones, PCs (Personal Computers), PDAs (Personal Digital Assistants), game machines and AV equipment with communication functions, car navigation systems, smart watches, These include wearable devices such as head-mounted displays, smart glasses, smart speakers, and cameras. However, in practice, it is not limited to these examples.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local It is possible to communicate with the information providing apparatus 100 by connecting to the network N via short-range wireless communication such as Area Network).

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, or the like. Note that the information providing apparatus 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described using FIG. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40. Prepare.

(Communication unit 11)
The communication unit 11 is connected to the network N (see FIG. 2) by wire or wirelessly, and transmits and receives information to and from the information providing apparatus 100 via the network N. FIG. For example, the communication unit 11 is implemented by a NIC (Network Interface Card), an antenna, or the like.

(Display unit 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescent Display). Also, the display unit 12 is a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons and the like for inputting characters, numbers, and the like. The input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, or the like. Moreover, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13 . Also, the input unit 13 may be a microphone or the like that receives voice input from the user U. FIG. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, position information (for example, latitude and longitude). That is, the positioning unit 14 positions the position of the terminal device 10 . GPS is merely an example of GNSS (Global Navigation Satellite System).

Also, the positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position using various communication functions of the terminal device 10 as described below as auxiliary positioning means for position correction and the like.

(Wi-Fi positioning)
For example, the positioning unit 14 measures the position of the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the position of the terminal device 10 by measuring the distance to a nearby base station or access point.

(beacon positioning)
The positioning unit 14 may also use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 positions the position of the terminal device 10 by connecting with a beacon transmitter connected by the Bluetooth (registered trademark) function.

(geomagnetic positioning)
Further, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetism pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, shops, etc., or has a function of reading an RFID tag In this case, the location used is recorded together with the information that the payment was made by the terminal device 10 . The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Also, the position may be measured by an optical sensor provided in the terminal device 10, an infrared sensor, or the like.

The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the positioning means described above, if necessary.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10 . The connection may be wired connection or wireless connection. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, a magnetic sensor 27, and an image sensor ( camera) 28.

The sensors 21 to 28 described above are only examples and are not limited. That is, the sensor unit 20 may be configured to include a part of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28. .

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as movement direction, speed, and acceleration of the terminal device 10 . The gyro sensor 22 detects physical movements of the terminal device 10 such as inclination in three axial directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 includes the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., techniques such as pedestrian dead-reckoning (PDR: Pedestrian Dead-Reckoning) using these sensors 21 to 23, etc. , the position of the terminal device 10 can be determined. This makes it possible to acquire indoor position information that is difficult to acquire with a positioning system such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Further, by using the gyro sensor 22, it is possible to know the traveling direction, the direction of the line of sight, and the inclination of the body of the user U. Also, from the atmospheric pressure detected by the atmospheric pressure sensor 23, the altitude at which the terminal device 10 of the user U is present and the number of floors can be known.

The temperature sensor 24 detects the temperature around the terminal device 10, for example. The sound sensor 25 detects sounds around the terminal device 10, for example. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, geomagnetism around the terminal device 10 . The image sensor 28 captures an image around the terminal device 10 .

The atmospheric pressure sensor 23, the temperature sensor 24, the sound sensor 25, the optical sensor 26, and the image sensor 28 described above detect the atmospheric pressure, temperature, sound, and illuminance, respectively, or capture an image of the surroundings to detect the terminal device 10. It is possible to detect the surrounding environment and situations. In addition, it is possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and situation of the terminal device 10 .

(control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, and various circuits. Also, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31 , a reception unit 32 , a processing unit 33 , a learning unit 34 , an acquisition unit 35 , an estimation unit 36 and a conversion unit 37 .

(Sending unit 31)
The transmission unit 31 transmits, via the communication unit 11, an inference result obtained as a result of inference based on on-device machine learning and a difference parameter between the shared model obtained as a result of the learning and the information providing apparatus 100 (server side).

Further, the transmission unit 31 transmits the local data collected by itself to the other terminal device 10 owned by the user U via the communication unit 11 or by short-range wireless communication. For example, when the terminal device 10 is the terminal device 10A, which is an old terminal (old device), the transmission unit 31 transmits the local data collected by itself to the new terminal device 10 via the communication unit 11 or by short-range wireless communication. (new device) to the terminal device 10B.

That is, the transmission unit 31 transmits the user U's context information collected by the function of the terminal device 10 to the other terminal device 10 owned by the user U via the communication unit 11 . In addition, the transmission unit 31 collects old context information via the communication unit 11 to another terminal device 10, which is an old terminal used by the user U at present or in the past, by the function of the terminal device 10. Send the context information of the user U who has been sent. In addition, the transmission unit 31 transmits, via the communication unit 11, as second context information to another terminal device 10 having a different number or performance of sensors from that of the terminal device 10 by sensors mounted on the terminal device 10. Send collected sensor data. Further, the transmission unit 31 may transmit a learned model using the user U's context information as learning data to another terminal device 10 owned by the user U via the communication unit 11 .

In addition, the transmission unit 31 transmits data to another terminal device 10 owned by the user U via the communication unit 11 using P2P, a home LAN, or a closed network, without leaking the terminal device to the outside. Send the user U's context information collected by function 10.

(Receiver 32)
The receiving unit 32 receives the sharing model provided from the information providing device 100 via the communication unit 11 . The receiving unit 32 also receives the modified sharing model (or the difference parameter from the previous sharing model) from the information providing device 100 via the communication unit 11 .

The receiving unit 32 also receives local data from another terminal device 10 owned by the user U via the communication unit 11 or by short-range wireless communication. For example, when the terminal device 10 is the terminal device 10B that is the new terminal (new device), the receiving unit 32 receives the terminal device that is the old terminal (old device) via the communication unit 11 or by short-range wireless communication. Receive local data for device 10A.

That is, the receiving unit 32 receives the user U's context information collected by the function of the other terminal device 10 from the other terminal device 10 owned by the user U via the communication unit 11 . In addition, the receiving unit 32 receives, via the communication unit 11, the old context information from the other terminal device 10, which is the old terminal currently or previously used by the user U, by the function of the other terminal device 10. Receive the collected user U context information. In addition, the receiving unit 32, via the communication unit 11, receives from another terminal device 10 having a different number or performance of sensors from the terminal device 10 as the second context information, sensors mounted on the other terminal device 10 receive sensor data collected by Further, the receiving unit 32 may receive a model learned by using user U's context information as learning data from another terminal device 10 owned by the user U via the communication unit 11 .

In addition, the receiving unit 32 receives data from another terminal device 10 owned by the user U via the communication unit 11 using P2P, a home LAN, or a closed network, without leaking to the outside. User U contextual information collected by the functions of the device 10 is received.

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing apparatus 100 to the display unit 12 for display.

Also, the processing unit 33 updates and upgrades the current sharing model based on the modified sharing model (or the difference parameter from the previous sharing model).

(Learning unit 34)
Based on the shared model, the learning unit 34 performs on-device machine learning using the local data (context information of the user U) of the terminal device 10 as learning data. Thereby, the learning unit 34 constructs a unique inference model based on the shared model, and generates a difference parameter between the inference model and the shared model. That is, the learning unit 34 performs machine learning using the context information of the user U as learning data.

(Acquisition unit 35)
The acquisition unit 35 collects local data (user U's context information) using the function of the terminal device 10 . For example, the acquisition unit 35 obtains various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10, and The location information of the terminal device 10 that has been positioned is collected as local data (user U's context information).

That is, the acquisition unit 35 collects the context information of the user U using the functions of the terminal device 10 . For example, the acquisition unit 35 collects the context information of the user U by the function of the terminal device 10, which is the new terminal, as the new context information. In addition, the acquisition unit 35 collects sensor data as the context information of the user U from the sensors installed in the terminal device 10 .

(Estimation unit 36)
The estimation unit 36 learns the correspondence relationship between the local data of the other terminal device 10 owned by the user U and the local data of the current terminal device 10, and estimates the conversion rule. The conversion rule may be in a table format such as a conversion table, or may be a conversion model obtained by machine learning. For example, the estimation unit 36 may build a conversion model by learning local data of other terminal devices 10 and local data of the current terminal device 10 as learning data.

That is, the estimating unit 36 combines context information (first context information) of the user U collected by the function of the terminal device 10 and context information (second context information) of the user U received from other terminal devices 10 ) to estimate a conversion rule between the first context information and the second context information. For example, the estimating unit 36 learns the correspondence relationship between the new context information collected by the function of the terminal device 10 that is the new terminal and the old context information received from the other terminal device 10 that is the old terminal, A conversion rule between the new context information and the old context information is deduced.

At this time, the estimation unit 36 may learn the correspondence relationship between the first context information and the second context information having the same position information, and estimate the conversion rule between the first context information and the second context information. good. Alternatively, the estimation unit 36 learns the correspondence relationship between the first context information and the second context information, in which at least a part of the sensor data is common, and estimates the conversion rule between the first context information and the second context information. You may

Further, the estimation unit 36 assumes that the sensor mounted on the terminal device 10 is also mounted on another terminal device 10, and converts the sensor data included in the first context information into the second context information. A conversion rule between the first context information and the second context information may be additionally estimated.

In addition, the estimating unit 36 estimates a model conversion rule based on the learning result regarding the transition of the model of another user U who changed the model from the same model as the other terminal device 10 to the same model as the terminal device 10. may At this time, the information providing device 100 performs federated learning as well as machine learning regarding changes in the model of the other user U who changed the model from the same model as the other terminal device 10 to the same model as the terminal device 10, The learning result is transmitted to the terminal device 10 of each user U who changed the model from the same model as the other terminal device 10 to the same model as the terminal device 10 concerned. The reception unit 32 of the terminal device 10 of each user U is related to the model transition of the other user U who changed the model from the information providing device 100 from the same model as the other terminal device 10 to the same model as the terminal device 10. Receive learning results.

(Converter 37)
The conversion unit 37 converts the local data of the other terminal device 10 owned by the user U into the local data of the current terminal device 10 according to the estimated conversion rule. That is, the local data of the other terminal device 10 owned by the user U is converted into data of the same format as the local data of the current terminal device 10 . For example, the conversion unit 37 may input the local data of another terminal device 10 into the conversion model constructed by the estimation unit 36, convert it into the format of the local data of the current terminal device 10, and output it.

That is, the conversion unit 37 converts the second context information into the first context information according to the conversion rule. For example, the conversion unit 37 converts old context information into new context information according to conversion rules. Further, the conversion unit 37 may convert the model according to the conversion rule of the model.

(storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disk. be. Various programs, various data, and the like are stored in the storage unit 40 .

In this embodiment, the storage unit 40 stores a model 41 , local data 42 , handover data 43 and conversion rules 44 .

The model 41 performs on-device machine learning using a shared model provided by the information providing device 100, a model inherited from another terminal device 10 owned by the user U, and local data 42 of the terminal device 10 as learning data. It is an inference model etc. obtained as a result of doing.

Also, the local data 42 is user U's context information collected by the terminal device 10 . For example, the local data 42 includes various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10, and data obtained by the positioning unit 14. It is the location information of the terminal device 10 that has been positioned, and the like. Also, the local data 42 may include user U attribute information and various types of history information (log data) indicating user U's behavior. In addition, the local data 42 may include various types of information regarding the user U's terminal device 10 .

The handover data 43 is local data handed over from another terminal device 10 owned by the user U. That is, the handover data 43 is user U's context information collected by another terminal device 10 owned by the user U. FIG.

Also, the conversion rule 44 is data indicating the correspondence relationship between the local data 42 and local data inherited from another terminal device 10 . The conversion rule 44 may be a conversion table or a conversion model obtained by machine learning.

Note that the storage unit 40 may store various types of information, not limited to the above, depending on the purpose. For example, the storage unit 40 stores user U's demographic (demographic attribute), psychographic (psychological attribute), geographic (geographical attribute), behavioral (behavioral attribute), and other attributes. Information may be stored. For example, the storage unit 40 stores name, family structure, hometown (local), occupation, position, income, qualification, residence type (detached house, condominium, etc.), car availability, school/commuting time, school/commuting route, Commuter pass sections (stations, lines, etc.), frequently used stations (other than the nearest station to your home or place of work), lessons (places, time zones, etc.), hobbies, interests, lifestyle, etc. good. The storage unit 40 also stores a location history that is a history of the location and date and time of the user U, a search history that is a history of search queries input by the user U, a browsing history that is a history of content browsed by the user U, Purchase history (payment history), which is the history of user U's product purchases and payment processing; exhibition history and sales history, which is the history of user U's listings on the marketplace; posting history, which is the history of user U's postings etc. may be stored. In addition, the storage unit 40 may store a usage history of various applications installed in the terminal device 10 by the user U, a payment history of payment (electronic payment) using the terminal device 10 of the user U, and the like. . These pieces of information are available as local data 42 .

[4. Processing procedure]
Next, a processing procedure performed by the terminal device 10 according to the embodiment will be described with reference to FIG. FIG. 4 is a sequence diagram illustrating a processing procedure according to the embodiment; Note that the processing procedure described below is repeatedly executed by the control unit 30 of the terminal device 10 .

As shown in FIG. 4, the acquiring unit 35 of the terminal device 10A, which is an old terminal (old device), collects local data (user U's context information) using the function of the terminal device 10A (step S101). For example, the acquisition unit 35 receives various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10A, the positioning unit 14 The location information and the like of the terminal device 10A positioned by is collected as local data (context information of the user U).

Next, the learning unit 34 of the terminal device 10A performs on-device machine learning using the shared model as a base and the local data (user U's context information) of the terminal device 10A as learning data (step S102). Note that the receiving unit 32 of the terminal device 10A may receive the sharing model provided from the information providing device 100 (server side) via the communication unit 11 . In addition, the transmission unit 31 of the terminal device 10A may transmit, via the communication unit 11, the difference parameter from the shared model as the learning result and the inference result to the information providing device 100 (server side).

Subsequently, the transmission unit 31 of the terminal device 10A transmits the learned model and local data (user U's context information) of the terminal device 10A to the terminal device 10B, which is a new terminal (new device), via the communication unit 11. (step S103). At this time, the receiving unit 32 of the terminal device 10B receives the learned model and local data of the terminal device 10A via the communication unit 11 . That is, the learned model and local data are handed over between the terminal device 10A and the terminal device 10B. Note that the processing unit 33 of the terminal device 10A may encrypt the learned model and local data. In this case, the transmission unit 31 may transmit a decryption key (public key or the like) to the terminal device 10B together with the encrypted learned model and local data. Also, the method of transferring models/data between devices may be P2P, home LAN (home hub format), or file compression.

Subsequently, the acquisition unit 35 of the terminal device 10B collects local data (context information of the user U) using the function of the terminal device 10B (step S104).

Subsequently, the estimation unit 36 of the terminal device 10B learns the correspondence relationship between the local data of the other terminal device 10A and the local data of the terminal device 10B, and estimates the conversion rule (step S105). For example, the estimation unit 36 of the terminal device 10B learns the correspondence relationship between the local data received from the other terminal device 10A and the local data collected by the function of the terminal device 10B, and receives from the other terminal device 10A. A mapping (conversion rule) for making the obtained local data correspond to the local data collected by the function of the terminal device 10B is estimated. Furthermore, the estimating unit 36 estimates the model conversion rule based on the learning result regarding the transition of the model of the other user U who changed the model from the same model as the other terminal device 10A to the same model as the terminal device 10B. may

Subsequently, the conversion unit 37 of the terminal device 10B converts the local data of the other terminal device 10A and the local data of the terminal device 10B according to the estimated conversion rule (step S106). That is, the local data of the other terminal device 10A is converted into data of the same format as the local data of the terminal device 10B. Furthermore, the conversion unit 37 may convert the learned model inherited from the other terminal device 10A according to the model conversion rule.

Subsequently, the learning unit 34 of the terminal device 10B learns the local data (context information of the user U) of the terminal device 10B including the converted data based on the learned model inherited from the other terminal device 10A. On-device machine learning is performed as data (step S107).

[5. Modification]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. So, below, the modification of embodiment is demonstrated.

In the above embodiment, part or all of the processing executed by the information providing device 100 may actually be executed by the terminal device 10 . For example, the processing may be completed stand-alone (by the terminal device 10 alone). In this case, it is assumed that the terminal device 10 has the functions of the information providing device 100 in the above embodiment. In addition, in the above-described embodiment, the terminal device 10 cooperates with the information providing device 100, so from the user U's point of view, it appears that the terminal device 10 is executing the processing of the information providing device 100 as well. That is, from another point of view, it can be said that the terminal device 10 includes the information providing device 100 .

Further, in the above-described embodiment, the terminal device 10 may be a smart phone or the like that realizes advanced camera functions by on-device AI (Artificial Intelligence). Also, the terminal device 10 may be a robot, self-driving car, drone, or the like equipped with an on-device AI.

Moreover, in the above embodiment, it is possible not only to transfer models/data from an old terminal to a new terminal, but also to transfer models/data from a new terminal to an old terminal. That is, the terminal device 10B, which is a new terminal (new device), can migrate the model/data to the terminal device 10A, which is an old terminal (old device). For example, if the terminal device 10B, which is the new terminal (new device), has a problem or needs repair, it is possible to temporarily save the model/data to the terminal device 10A, which is the old terminal (old device). be. In this way, model/data migration between terminals (devices) can be bi-directional rather than uni-directional.

[6. effect〕
As described above, the terminal device 10 (10B) according to the present application is the terminal device 10 (10B) owned by the user U, and is used as the first context information by the function of the terminal device 10 (10B). The acquisition unit 35 that collects the context information of the person U, and the second context information that is collected from the other terminal device 10 (10A) owned by the user U by the function of the other terminal device 10 (10A). A receiving unit 32 for receiving context information of a person U, and an estimating unit 36 for learning a correspondence relationship between the first context information and the second context information and estimating a conversion rule between the first context information and the second context information. , a conversion unit 37 that converts the second context information into the first context information according to a conversion rule, and a learning unit 34 that performs machine learning using the context information of the user U as learning data.

The acquisition unit 35 also collects context information of the user U as new context information using the function of the terminal device 10 (10B), which is a new terminal. The receiving unit 32 collects old context information from the other terminal device 10 (10A), which is the old terminal currently or previously used by the user U, by the function of the other terminal device 10 (10A). Receive user U's context information. The estimation unit 36 learns the correspondence relationship between the new context information and the old context information, and estimates the conversion rule between the new context information and the old context information. The conversion unit 37 converts the old context information into the new context information according to the conversion rule. The learning unit 34 performs machine learning using the context information of the user U as learning data.

In addition, the acquisition unit 35 collects sensor data as the first context information from the sensor mounted on the terminal device 10 (10B). The receiving unit 32 receives, as second context information, from another terminal device 10 (10A) having a different number or performance of sensors from the terminal device 10 (10B), sensors mounted on the other terminal device 10 (10A) receive sensor data collected by The estimation unit 36 learns the correspondence relationship between the first context information and the second context information, and estimates the conversion rule between the first context information and the second context information. The conversion unit 37 converts the second context information into the first context information according to the conversion rule. The learning unit 34 performs machine learning using the context information of the user U as learning data.

The estimation unit 36 also learns the correspondence relationship between the first context information and the second context information having the same position information, and estimates a conversion rule for the first context information and the second context information.

In addition, the estimation unit 36 learns the correspondence relationship between the first context information and the second context information in which at least a part of the sensor data is common, and estimates the conversion rule between the first context information and the second context information. do.

Further, the estimation unit 36 assumes that the sensor installed in the terminal device 10 (10B) is also installed in another terminal device 10 (10A), and the sensor data included in the first context information is added to the second context information to estimate the conversion rule between the first context information and the second context information.

Further, the receiving unit 32 receives a model learned using the second context information as learning data from the other terminal device 10 (10A) owned by the user U. The estimating unit 36 converts the model based on the learning result regarding the transition of the model of the other user U who changed the model from the same model as the other terminal device 10 (10A) to the same model as the terminal device 10 (10B). Infer the rules. The conversion unit 37 converts the model according to the conversion rule of the model.

In addition, the receiving unit 32 receives data from another terminal device 10 (10A) owned by the user U using P2P, a home LAN, or a closed network without leaking to the outside. ) receives user U's contextual information collected by the function of U.

By any one or combination of the above processes, the terminal device according to the present application can seamlessly transfer data between individual terminal devices that perform machine learning.

[7. Hardware configuration]
Also, the terminal device 10 and the information providing device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 5, for example. Hereinafter, the terminal device 10 will be described as an example. FIG. 5 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

Arithmetic device 1030 operates based on programs stored in primary storage device 1040 and secondary storage device 1050, programs read from input device 1020, and the like, and executes various types of processing. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the terminal device 10 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 30 by executing programs loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[8. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific forms of distribution and integration of each device are not limited to those illustrated, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing content.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the control unit can be read as control means or a control circuit.

1 information processing system 10 terminal device 11 communication unit 14 positioning unit 20 sensor unit 30 control unit 31 transmission unit 32 reception unit 33 processing unit 34 learning unit 35 acquisition unit 36 estimation unit 37 conversion unit 40 storage unit 100 information providing device

Claims (10)

A terminal device owned by a user,
an acquisition unit that collects the user's context information as the first context information by the function of the terminal device;
a receiving unit that receives, as second context information, context information of the user collected from another terminal device owned by the user by a function of the other terminal device;
an estimation unit that learns a correspondence relationship between the first context information and the second context information and estimates a conversion rule between the first context information and the second context information;
a conversion unit that converts the second context information into the first context information according to the conversion rule;
a learning unit that performs machine learning using the context information of the user as learning data;
A terminal device comprising: The acquisition unit collects context information of the user as new context information by a function of the terminal device which is a new terminal,
The receiving unit, as the old context information, the user's context information collected from the other terminal device, which is the old terminal currently or in the past used by the user, by the function of the other terminal device. receive the
the estimating unit learns a correspondence relationship between the new context information and the old context information, and estimates a conversion rule between the new context information and the old context information;
The conversion unit converts the old context information into the new context information according to the conversion rule,
The terminal device according to claim 1, wherein the learning unit performs machine learning using the user's context information as learning data. The acquisition unit collects sensor data from a sensor mounted on the terminal device as the first context information,
The receiving unit receives, as the second context information, sensor data collected by a sensor mounted on the other terminal device from the other terminal device having a different number or performance of sensors from the terminal device. ,
the estimation unit learns a correspondence relationship between the first context information and the second context information to estimate a conversion rule between the first context information and the second context information;
The conversion unit converts the second context information into the first context information according to the conversion rule;
The terminal device according to claim 1 or 2, wherein the learning unit performs machine learning using the user's context information as learning data. The estimation unit learns a correspondence relationship between the first context information and the second context information having common position information, and estimates a conversion rule between the first context information and the second context information. The terminal device according to any one of claims 1 to 3, characterized by: The estimation unit learns a correspondence relationship between the first context information and the second context information, in which at least a part of sensor data is common, and a conversion rule between the first context information and the second context information. The terminal device according to any one of claims 1 to 4, characterized by estimating The estimation unit adds sensor data included in the first context information to the second context information on the assumption that the sensor mounted on the terminal device is also mounted on the other terminal device. 6. The terminal device according to any one of claims 1 to 5, wherein a conversion rule between said first context information and said second context information is estimated in the form of: The receiving unit receives a model trained using the second context information as learning data from another terminal device owned by the user,
The estimating unit estimates a conversion rule of the model based on a learning result regarding the transition of the model of another user who changed the model from the same model as the other terminal device to the same model as the terminal device,
The terminal device according to any one of claims 1 to 6, wherein the conversion unit converts the model according to a conversion rule of the model. The receiving unit uses P2P, a home LAN, or a closed network from another terminal device owned by the user, without leaking to the outside, the usage information collected by the function of the other terminal device The terminal device according to any one of claims 1 to 7, characterized in that it receives context information of a user. An information processing method executed by a terminal device owned by a user,
an obtaining step of collecting the context information of the user by the function of the terminal device as the first context information;
a receiving step of receiving, as second context information, context information of the user collected from another terminal device owned by the user by a function of the other terminal device;
an estimation step of learning a correspondence relationship between the first context information and the second context information and estimating a conversion rule between the first context information and the second context information;
a conversion step of converting the second context information into the first context information according to the conversion rule;
a learning step of performing machine learning using the context information of the user as learning data;
An information processing method comprising: an acquisition procedure for collecting context information of the user as the first context information by a function of a terminal device owned by the user;
a receiving procedure for receiving, as second context information, context information of the user collected from another terminal device owned by the user by a function of the other terminal device;
an estimation step of learning a correspondence relationship between the first context information and the second context information and estimating a conversion rule between the first context information and the second context information;
a conversion procedure for converting the second context information into the first context information according to the conversion rule;
a learning procedure for performing machine learning using the context information of the user as learning data;
An information processing program for causing a terminal device to execute

Images