JP2023011145A - Terminal device, information processing method, and information processing program - Google Patents

Abstract

To share sensor data between terminal devices for machine learning, to learn with sensor data of the other terminal devices.SOLUTION: A terminal device includes: a receiving unit which receives, from another terminal device carried by another user, other-device sensor data collected by sensors of the other terminal devices; and a learning unit which executes machine learning using the other-device sensor data as learning data. The terminal device also includes: an acquisition unit which collects device sensor data through a sensor of the terminal device carried by a user; a comparison unit which compares the device sensor data with the other-device sensor data; and an adoption unit which adopts one of the device sensor data and the other-device sensor data, in accordance with a result of the comparison. The learning unit may execute machine learning using the adopted sensor data, as learning data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a terminal device, an information processing method and an information processing program.

A technique is disclosed for sending teacher data to a server device for causing the server device to perform machine learning.

Japanese Patent No. 6880290

Machine learning is generally performed by a server device, as in the conventional technology described above. However, if the data contains personal information, it may not be desirable to send the data to an external server device such as a company. In addition, since there are various restrictions on the transmission and reception of data including personal information, it may not be easy to freely transmit and receive data.

Therefore, in recent years, attention has been focused on on-device machine learning, such as federated learning, in which data is distributed to individual terminal devices without being aggregated in a server device. . However, when machine learning is performed on individual terminal devices using sensor data that is affected by the shape and performance of the terminal device, the model of the terminal device owned by each user, the type and performance of the installed sensor, etc. There is a problem that a difference occurs in the measurement result depending on the

This application is made in view of the above, and an object of this application is to share sensor data between individual terminal devices that perform machine learning, and to learn using sensor data of other terminal devices.

A terminal device according to the present application includes a receiving unit that receives other person's sensor data collected by a sensor of the other terminal device from another terminal device owned by another user; and a learning unit that performs machine learning as a

According to one aspect of the embodiment, it is possible to share sensor data among individual terminal devices that perform machine learning, and to perform learning using sensor data of other terminal devices.

FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 3 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 4 is a sequence diagram illustrating a processing procedure according to the embodiment; FIG. 5 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing a terminal device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. In addition, the terminal device, the information processing method, and the information processing program according to the present application are not limited to this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Outline of information processing method]
First, an outline of an information processing method performed by an information processing apparatus according to an embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. Note that in FIG. 1, an example will be described in which sensor data is shared between individual terminal devices that perform machine learning, and learning is performed using sensor data from other terminal devices.

As shown in FIG. 1 , the information processing system 1 includes a terminal device 10 and an information providing device 100 . The terminal device 10 and the information providing device 100 are communicably connected to each other by wire or wirelessly via a network N (see FIG. 2).

The terminal device 10 is a smart device such as a smartphone or tablet used by a user U (user), and communicates with an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). It is a portable terminal device capable of performing The terminal device 10 has a screen such as a liquid crystal display and has a touch panel function. Accepts various operations for . An operation performed on an area where content is displayed on the screen may be an operation on the content. In addition, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC as well as a smart device.

The information providing device 100 cooperates with the terminal device 10 of each user U, and provides the terminal device 10 of each user U with API (Application Programming Interface) services for various applications (hereinafter referred to as apps) and various It is an information processing device that provides data, and is realized by a server device, a cloud system, or the like.

Also, the information providing device 100 may be an information processing device that provides some web service online to the terminal device 10 of each user U. FIG. For example, the information providing apparatus 100 provides Web services such as Internet connection, search service, SNS (Social Networking Service), electronic commerce, electronic payment, online game, online banking, online trading, accommodation/ticket reservation, video/music distribution, Services such as news, maps, route search, route guidance, route information, operation information, and weather forecast may be provided. In practice, the information providing apparatus 100 may cooperate with various servers that provide web services as described above, mediate web services, or take charge of web service processing.

As shown in FIG. 1, in the present embodiment, the information providing device 100 (server side) employs federated learning, and the terminal device 10 (terminal side) of each user U A shared model (core model) is provided (step S1).

Based on the shared model, the terminal device 10 performs on-device machine learning using sensor data (user U's context information) collected using various sensors mounted (or connected) to the terminal device 10 as learning data. A pattern is created by this, and inference is performed from the input data using the pattern (step S2).

Subsequently, the terminal device 10 transmits the inference result obtained as the inference result and the difference parameter between the shared model obtained as the learning result to the information providing device 100 (server side) (step S3). At this time, the terminal device 10 may transmit the inference result in the form of vector data converted (vectorized) into a multidimensional vector. Vector data is irreversible information, and the original data cannot be restored from the vector values of the vector data. For example, the terminal device 10 may perform so-called embedding to generate an embedding vector based on the inference result. Encrypting and transmitting vector data, which is transmission data, is possible within the scope of common technical knowledge. Alternatively, tunneling may be used to establish a closed virtual direct line between the terminal side and the server side. For example, a VPN (Virtual Private Network) may be constructed between the terminal side and the server side.

The information providing device 100 performs an aggregation process of the inference results received from the terminal device 10 of each user U (step S4). Aggregation processing includes arithmetic processing such as statistics and analysis.

Subsequently, the information providing device 100 modifies the sharing model based on the difference parameters received from the terminal device 10 of each user U (step S5). Then, the information providing device 100 provides the modified sharing model (or the difference parameter from the previous sharing model) to the terminal device 10 of each user U (returns to step S1). That is, the information providing device 100 updates and upgrades the sharing model provided to the terminal device 10 of each user U. FIG.

Note that if the information providing apparatus 100 does not need to update the sharing model, the process of step S5 may not be performed. In this case, in step S3, the terminal device 10 should transmit only the inference result.

In this way, when the terminal device 10 (terminal side), not the information providing device 100 (server side), performs machine learning using sensor data as learning data, the model of the terminal device owned by each user, Depending on the type and performance of the installed sensors, there may be differences in the measured sensor data. For example, even if users are in the same place or are doing the same thing, the number and accuracy of sensor data measured depends on the model of the terminal device and the type and performance of the installed sensor. can vary greatly. As a result, learning outcomes may also be affected. Therefore, it is desirable to equalize/homogenize sensor data as much as possible for users who are in the same place or who are taking the same action.

However, since sensor data is also information that can estimate the position, behavior, and context of an individual, it may not be desirable to share it unconditionally among all users. Therefore, it is possible to share sensor data between users who have permitted/agreed to share sensor data with each other in advance (for example, between family members, between friends, etc.). It is desirable to share information and sensor data in situations where there is no problem. Hereinafter, terminal devices around the user U indicate terminal devices of other users who have permitted/agreed to share sensor data with the terminal device 10 of the user U. FIG.

In this embodiment, sensor data is automatically/periodically shared between the terminal devices 10 of each user U (step S6). As a result, the data collected from the terminal devices around the user U are regarded as "the data acquired by the terminal device 10 of the user U" and learning is performed. At this time, among the terminal devices 10 of each user U, the terminal device 10 (10A) of the first user U (U1) is sent to the terminal device 10 (10B) of the second user U (U2). Transmit data.

For example, the terminal device 10 (10B) of the second user U (U2) automatically/periodically requests sensor data by broadcast, and the terminal device 10 (10B) of the first user U (U1) 10A) sends sensor data in response. In this case, the terminal device 10 of the user U may inquire whether or not the surrounding terminal device has predetermined sensor data, and if the surrounding terminal device has the sensor data, request the sensor data. . In addition, the terminal device 10 of the user U declares the desired sensor data and the context of the user U, and receives the sensor data from the terminal device of another user whose context is similar to that of the user U. may

For example, if the terminal device 10 of the user U does not have a temperature sensor, it broadcasts a request such as "give me temperature sensor data" to surrounding terminal devices. In response to the request, the surrounding terminal devices transmit the temperature sensor data to the user U's terminal device 10 . Note that the temperature sensor is merely an example. Data from multiple sensors may also be requested. Alternatively, the data of all sensors may be requested unconditionally.

Further, when the terminal device 10 of the user U has the sensor data when the user U is walking but does not have the sensor data when the user U is running, the terminal device 10 reads "the user is running Broadcast a request such as "Give me the sensor data of the time". In response to the request, the surrounding terminal devices transmit sensor data to the terminal device 10 of the user U when the user of the terminal device is running.

In addition, when there is little sensor data when the user U is at a predetermined place (a store, a spot, etc.), the terminal device 10 of the user U may make up for the lack of learning data by Please broadcast the information "Give me the data when you are at", and receive the sensor data when the user of the terminal device is at the location from the surrounding terminal devices.

Note that the terminal device 10 (10A) of the first user U (U1) automatically/periodically communicates P2P (Peer to Peer) to pass the sensor data. Alternatively, sensor data may be passed between terminal devices of users running the same application (game, payment application, etc.) or users taking the same action. If both users are using the same application, the sensor data may be passed between the terminal devices of the users via the application.

In addition, the terminal device 10 (10A) of the first user U (U1) automatically/periodically connects the terminal device 10 (10A) to the terminal device 10 (10B) via Bluetooth (registered trademark) or wireless The sensor data may be transmitted by short-range wireless communication such as LAN (Local Area Network). The sensor data may be transmitted by so-called "passing communication". If the data is collected from the terminal devices of users who are close enough to be able to communicate by short-range communication, it is presumed that the sensor data at that time is common.

Alternatively, the terminal device 10 (10A) of the first user U (U1) automatically/periodically connects the home LAN to the terminal device 10 (10B) of the second user U (U2). Sensor data may be transmitted via. For example, the terminal device 10 (10A) of the first user U (U1) sends the terminal device 10 (10B) of the second user U (U2) to a hub such as a smart home or a home router (using The sensor data may be transmitted via a local relay device managed by person U), a personal PC, a home server (home server), or the like. At this time, the user U sends the sensor data from the terminal device 10 (10A) to a hub such as a smart home, a home router (a local relay device managed by the user U), a personal PC, a home server, or the like. can be backed up. The backup of sensor data from the terminal device 10 (10A) may be performed in real time when the sensor data is acquired (real-time support). Then, the terminal device 10 (10B) may acquire the backed up sensor data. If the data is collected from terminal devices such as family members in the same household, it is assumed that the sensor data in the household is common. The family is just one example. For example, it may be a colleague or a member of a group in the same facility, or an employee and a customer in the same facility.

Alternatively, the user U may compress the sensor data into a file in a format that can be decompressed only by the user U on the terminal device 10 (10A), and download and decompress the compressed file on the terminal device 10 (10B) side. good. For example, the terminal device 10 (10A) may encrypt sensor data with a private key and provide the public key together with the encrypted sensor data only to the terminal device 10 (10B). In practice, the terminal device 10 (10A) of the first user U (U1) connects to the terminal device 10 (10B) of the second user U (U2) via a closed network. ) to pass sensor data.

Thus, the method of transmitting/sharing sensor data between devices owned by each user U may be P2P, home LAN (home hub format), or file compression.

Here, when transmitting/sharing sensor data between devices owned by each user U, the terminal device 10 (10A) of the first user U (U1) and the terminal device 10 (10A) of the second user U (U2) Authentication may be performed with the terminal device 10 (10B). The authentication may be FIDO (Fast Identity Online) authentication. Then, only when the authentication is successful, the terminal device 10 (10A) of the first user U (U1) transmits the sensor data to the terminal device 10 (10B) of the second user U (U2). . It should be noted that it is possible within the scope of common technical knowledge to encrypt and transmit the transmitted sensor data.

Moreover, actually, the terminal device 10 (10B) of the second user U (U2) also transmits its own sensor data to the terminal device 10 (10A) of the first user U (U1). can be done. That is, the transmission of sensor data between terminals can be bidirectional rather than unidirectional. In this case, the terminal device 10 (10A) of the first user U (U1) also performs the same process as the terminal device 10 (10B) of the second user U (U2).

Furthermore, not only the sensor data but also the learned model may be shared between the terminal devices 10 of each user U. For example, the terminal device 10 (10A) of the first user U (U1) transmits not only the sensor data but also the sensor data to the terminal device 10 (10B) of the second user U (U2). A learned model may be transmitted as data.

In this embodiment, among the terminal devices 10 of each user U, the terminal device 10 (10B) of the second user U (U2) is the terminal device 10 (10A) of the first user U (U1). and the sensor data of the terminal device 10 (10B) of the second user U (U2) (step S7).

Here, the sensor data to be compared is desirably sensor data measured at the same (or close to) time and position (that is, sensor data having common position information and location history). Therefore, the terminal device 10 (10B) of the second user U (U2) uses the position information as a reference to store the sensor data of the terminal device 10 (10A) of the first user U (U1) and the second The sensor data of the terminal device 10 (10B) of the user U (U2) is compared. In this case, the position information and the sensor data are linked. Position information and sensor data may be associated with time information as a key. Further, in practice, the sensor data may include positional information obtained by positioning and data measured by other sensors.

However, when using real-time sensor data, the terminal device 10 of the user U may compare real-time sensor data of itself and others at the time of collection regardless of position information or time information.

Subsequently, the terminal device 10 (10B) of the second user U (U2) selects the terminal device 10 (10B) of the sensor data of the terminal device 10 (10A) of the first user U (U1). data not included in the sensor data (measurement data of a sensor not installed in the terminal device) or data with higher precision than the sensor data of the terminal device 10 (10B) (measurement data of a higher performance sensor ) is employed (step S8).

At this time, the terminal device 10 (10B) of the second user U (U2) receives the sensor data of the terminal device 10 (10A) of the first user U (U1) and the data of the terminal device 10 (10B). Data that is not included in the sensor data of the terminal device 10 (10B) or data that is more accurate than the sensor data of the terminal device 10 (10B) may be determined from the difference from the sensor data. For example, the terminal device 10 (10B) detects data missing from the sensor data of the terminal device 10 (10B) in the sensor data of the terminal device 10 (10A) of the first user U (U1), Data (detailed data) having a larger number of digits after the decimal point than the sensor data of the terminal device 10 (10B) may be used.

Alternatively, the terminal device 10 (10B) of the second user U (U2) sends information about the terminal device 10 (10A) together with the sensor data of the terminal device 10 (10A) of the first user U (U1). Based on the information about the terminal device 10 (10A) and the information about the terminal device 10 (10B), the data not included in the sensor data of the terminal device 10 (10B), the terminal device 10 ( 10B) data with higher precision than the sensor data may be determined. The information about the terminal device 10 is, for example, model information of the terminal device 10, information about sensors installed in the terminal device 10, and the like.

Subsequently, the terminal device 10 (10B) of the second user U (U2) receives the sensor data of the terminal device 10 (10A) of the first user U (U1) and the sensor data of the second user U (U2). ) and the sensor data of the terminal device 10 (10B) of ), on-device machine learning is performed using the adopted sensor data as learning data, and an inference is made based on the learning result (step S9).

As a result, the terminal device 10 of the user U can share sensor data among individual terminal devices that perform machine learning, and can learn using the sensor data of other terminal devices. In particular, learning can be performed using sensor data of other terminal devices that have sensors not present in the own terminal device. In addition, it is possible to learn by comparing the sensor data of its own and other terminal devices and using better sensor data.

In the above description, the sensor data of the terminal device 10 is compared with the sensor data of the terminal device 10, and the sensor data of the terminal device 10 is not included in the sensor data collected from the other terminal device 10. Although data with higher accuracy than data is used, in practice, machine learning may be performed by using all the sensor data of the self and others as they are as learning data without comparison.

Further, when the terminal device 10 of the user U cannot collect sensor data from other terminal devices 10, the terminal device 10 performs machine learning using only its own sensor data as learning data. Alternatively, the terminal device 10 of the user U does not perform machine learning when the sensor data cannot be collected from the other terminal device 10, and only when the sensor data can be collected from the other terminal device 10, the above procedure You may make it implement machine learning in .

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 2 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 2, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. As shown in FIG. In FIG. 2, as the terminal devices 10, the terminal device 10 (10A) of the first user U (U1) and the terminal device 10 (10B) of the second user U (U2) are shown. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Also, the number of devices included in the information processing system 1 shown in FIG. 2 is not limited to the illustrated one. For example, in FIG. 2, for simplification of illustration, the terminal devices 10 are the terminal device 10 (10A) of the first user U (U1) and the terminal device 10 (10A) of the second user U (U2). 10B) and 10B) are shown one by one, but this is only an example and is not limited, and each may be two or more. For example, the terminal device 10 (10A) of the first user U (U1) and the terminal device 10 (10B) of the second user U (U2) may exist for each user.

The terminal device 10 is an information processing device used by the user U. FIG. For example, the terminal device 10 includes smart devices such as smartphones and tablet terminals, feature phones, PCs (Personal Computers), PDAs (Personal Digital Assistants), game machines and AV equipment with communication functions, car navigation systems, smart watches, These include wearable devices such as head-mounted displays, smart glasses, smart speakers, and cameras. However, in practice, it is not limited to these examples.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local It is possible to communicate with the information providing apparatus 100 by connecting to the network N via short-range wireless communication such as Area Network).

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, or the like. Note that the information providing apparatus 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described using FIG. FIG. 3 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40. Prepare.

(Communication unit 11)
The communication unit 11 is connected to the network N (see FIG. 2) by wire or wirelessly, and transmits and receives information to and from the information providing apparatus 100 via the network N. FIG. For example, the communication unit 11 is implemented by a NIC (Network Interface Card), an antenna, or the like.

(Display unit 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescent Display). Also, the display unit 12 is a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons and the like for inputting characters, numbers, and the like. The input unit 13 may be an input/output port (I/O port), a USB (Universal Serial Bus) port, or the like. Moreover, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13 . Also, the input unit 13 may be a microphone or the like that receives voice input from the user U. FIG. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, position information (for example, latitude and longitude). That is, the positioning unit 14 positions the position of the terminal device 10 . GPS is merely an example of GNSS (Global Navigation Satellite System).

Also, the positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position using various communication functions of the terminal device 10 as described below as auxiliary positioning means for position correction and the like.

(Wi-Fi positioning)
For example, the positioning unit 14 measures the position of the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the position of the terminal device 10 by measuring the distance to a nearby base station or access point.

(beacon positioning)
The positioning unit 14 may also use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting with a beacon transmitter connected by the Bluetooth (registered trademark) function.

(geomagnetic positioning)
Further, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetism pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag In this case, the location used is recorded together with the information that the payment was made by the terminal device 10 . The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Also, the position may be measured by an optical sensor provided in the terminal device 10, an infrared sensor, or the like.

The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the positioning means described above, if necessary.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10 . The connection may be wired connection or wireless connection. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, a magnetic sensor 27, and an image sensor ( camera) 28.

The sensors 21 to 28 described above are only examples and are not limited. That is, the sensor unit 20 may be configured to include a part of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28. .

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as movement direction, speed, and acceleration of the terminal device 10 . The gyro sensor 22 detects physical movements of the terminal device 10 such as inclination in three axial directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 includes the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., techniques such as pedestrian dead-reckoning (PDR: Pedestrian Dead-Reckoning) using these sensors 21 to 23, etc. , the position of the terminal device 10 can be determined. This makes it possible to acquire indoor position information that is difficult to acquire with a positioning system such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. Further, by using the gyro sensor 22, it is possible to know the traveling direction, the direction of the line of sight, and the inclination of the body of the user U. Also, from the atmospheric pressure detected by the atmospheric pressure sensor 23, the altitude at which the terminal device 10 of the user U is present and the number of floors can be known.

The temperature sensor 24 detects the temperature around the terminal device 10, for example. The sound sensor 25 detects sounds around the terminal device 10, for example. The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, geomagnetism around the terminal device 10 . The image sensor 28 captures an image around the terminal device 10 .

The atmospheric pressure sensor 23, the temperature sensor 24, the sound sensor 25, the optical sensor 26, and the image sensor 28 described above detect the atmospheric pressure, temperature, sound, and illuminance, respectively, or capture an image of the surroundings to detect the terminal device 10. It is possible to detect the surrounding environment and situations. In addition, it is possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and situation of the terminal device 10 .

(control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, and various circuits. Also, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31 , a reception unit 32 , a processing unit 33 , a learning unit 34 , an acquisition unit 35 , a comparison unit 36 and an adoption unit 37 .

(Sending unit 31)
The transmission unit 31 transmits, via the communication unit 11, an inference result obtained as a result of inference based on on-device machine learning and a difference parameter between the shared model obtained as a result of the learning and the information providing apparatus 100 (server side).

Further, the transmission unit 31 transmits sensor data collected by itself to other terminal devices 10 owned by other users U via the communication unit 11 or by short-range wireless communication. For example, when the terminal device 10 is the terminal device 10 (10A) of the first user U (U1), the transmission unit 31 transmits the terminal device to its own terminal device via the communication unit 11 or by short-range wireless communication. 10 (10A) transmits the collected sensor data to the terminal device 10 (10B) of the second user U (U2).

That is, the transmission unit 31 transmits sensor data collected by the sensor mounted on the terminal device 10 to another terminal device 10 owned by another user U via the communication unit 11 . For example, the transmission unit 31 transmits sensor data collected by the sensors mounted on the terminal device 10 to another terminal device 10 having a different number or performance of sensors from the terminal device 10 via the communication unit 11. Send. Further, the transmission unit 31 may transmit a model learned using sensor data as learning data to another terminal device 10 owned by another user U via the communication unit 11 .

In addition, the transmission unit 31 transmits the information to another terminal device 10 owned by another user U via the communication unit 11 using P2P, a home LAN, or a closed network without leaking the information to the outside. Sensor data collected by a sensor mounted on the terminal device 10 is transmitted.

(Receiver 32)
The receiving unit 32 receives the sharing model provided from the information providing device 100 via the communication unit 11 . The receiving unit 32 also receives the modified sharing model (or the difference parameter from the previous sharing model) from the information providing device 100 via the communication unit 11 .

Further, the receiving unit 32 receives sensor data from another terminal device 10 owned by another user U via the communication unit 11 or by short-range wireless communication. For example, when the terminal device 10 is the terminal device 10 (10B) of the second user U (U2), the receiving unit 32 receives the first usage data via the communication unit 11 or by short-range wireless communication. The sensor data of the terminal device 10 (10A) of the person U (U1) is received.

That is, the receiving unit 32 receives sensor data collected by the sensors installed in the other terminal devices 10 from the other terminal devices 10 owned by the other users U via the communication unit 11 . In addition, the receiving unit 32 receives sensor data collected by sensors mounted on the other terminal device 10 from another terminal device 10 having a different number or performance of sensors from the terminal device 10 via the communication unit 11. to receive Further, the receiving unit 32 may receive a model learned using sensor data as learning data from another terminal device 10 owned by another user U via the communication unit 11 .

In addition, the receiving unit 32 uses P2P, a home LAN, or a closed network from another terminal device 10 owned by another user U via the communication unit 11, without leaking to the outside. receive the sensor data collected by the sensor mounted on the terminal device 10 of the .

In this way, the receiving unit 32 receives other person's sensor data collected by the sensor of the other terminal device 10 from the other terminal device 10 owned by the other user U. At this time, the receiving unit 32 may receive other person's sensor data collected by the sensor of the other terminal device 10 from the other terminal device 10 via short-range wireless communication or LAN in the same premises. Further, the receiving unit 32 may receive, from the other terminal device 10 , other-person sensor data collected in real time by the sensor of the other terminal device 10 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing apparatus 100 to the display unit 12 for display.

Also, the processing unit 33 updates and upgrades the current sharing model based on the modified sharing model (or the difference parameter from the previous sharing model).

(Learning unit 34)
Based on the shared model, the learning unit 34 performs on-device machine learning using the sensor data of the terminal device 10 as learning data. Thereby, the learning unit 34 constructs a unique inference model based on the shared model, and generates a difference parameter between the inference model and the shared model. That is, the learning unit 34 performs machine learning using sensor data as learning data.

In the present embodiment, the learning unit 34 performs machine learning using other person's sensor data as learning data. For example, the learning unit 34 performs machine learning using self sensor data and other sensor data as learning data. At this time, the learning unit 34 performs machine learning using the sensor data adopted from the self sensor data and the other person's sensor data as learning data.

(Acquisition unit 35)
The acquisition unit 35 collects sensor data from a sensor mounted on the terminal device 10 owned by the user U. FIG. That is, the acquisition unit 35 collects self-sensor data from the sensor of the terminal device 10 . For example, the acquisition unit 35 obtains various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10, and Positional information and the like of the terminal device 10 that has been positioned is collected as sensor data.

(Comparator 36)
The comparison unit 36 compares the self sensor data and the other person's sensor data. At this time, the comparison unit 36 may compare the self sensor data and the other person's sensor data on the basis of the position information.

(Recruitment department 37)
The adopting unit 37 adopts either one of the self sensor data and the other person's sensor data (preferred sensor data) according to the comparison result. At this time, the adopting unit 37 adopts the other person's sensor data having data not included in the self sensor data. For example, the adopting unit 37 adopts other person's sensor data collected by a sensor installed in another terminal device 10 but not installed in the terminal device 10 concerned. Also, the hiring unit 37 employs other person's sensor data that is more accurate than the self sensor data.

(storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disk. be. Various programs, various data, and the like are stored in the storage unit 40 .

In this embodiment, the storage unit 40 stores a model 41 , self sensor data 42 , other sensor data 43 , and employment rules 44 .

The model 41 is a shared model provided by the information providing apparatus 100, an inference model obtained as a result of performing on-device machine learning using sensor data as learning data, or the like.

Self-sensor data 42 is sensor data collected by the terminal device 10 . For example, the self sensor data 42 includes various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10, the positioning unit 14 It is the position information of the terminal device 10 positioned by. In addition, the self sensor data 42 may include attribute information of the user U and various types of history information (log data) indicating the behavior of the user U. In addition, the self sensor data 42 may include various information regarding the user U's terminal device 10 .

Further, the sensor data 43 of others is sensor data collected from other terminal devices 10 owned by other users U. FIG. That is, the other person's sensor data 43 is sensor data collected by another terminal device 10 owned by another user U. FIG.

Also, the adoption rule 44 indicates a rule for adopting either the self sensor data 42 or the other sensor data 43 . For example, among the other sensor data 43, data not included in the self sensor data 42 (measurement data of sensors not installed in the terminal device), or data with higher precision than the self sensor data 42 (higher performance data) It shows the rules such as adopting the measurement data of various sensors). Note that the adoption rule 44 may be a adoption model obtained by machine learning using the other sensor data 43 (and self sensor data 42) and the adopted sensor data as learning data. In this case, by inputting the other person's sensor data 43 (and self sensor data 42) as input data to the adoption model, the sensor data obtained as output may be adopted.

Note that the storage unit 40 may store various types of information, not limited to the above, depending on the purpose. For example, the storage unit 40 stores user U's demographic (demographic attribute), psychographic (psychological attribute), geographic (geographical attribute), behavioral (behavioral attribute), and other attributes. Information may be stored. For example, the storage unit 40 stores name, family structure, hometown (local), occupation, position, income, qualification, residence type (detached house, condominium, etc.), car availability, school/commuting time, school/commuting route, Commuter pass sections (stations, lines, etc.), frequently used stations (other than the nearest station to your home or place of work), lessons (places, time zones, etc.), hobbies, interests, lifestyle, etc. good. The storage unit 40 also stores a location history that is a history of the location and date and time of the user U, a search history that is a history of search queries input by the user U, a browsing history that is a history of content browsed by the user U, Purchase history (payment history), which is the history of user U's product purchases and payment processing, exhibition history and sales history, which is the history of user U's listings on the marketplace, posting history, which is the history of user U's postings etc. may be stored. In addition, the storage unit 40 may store a usage history of various applications installed in the terminal device 10 by the user U, a payment history of payment (electronic payment) using the terminal device 10 of the user U, and the like. . These information can be used as learning data along with sensor data.

[4. Processing procedure]
Next, a processing procedure performed by the terminal device 10 according to the embodiment will be described with reference to FIG. FIG. 4 is a sequence diagram illustrating a processing procedure according to the embodiment; Note that the processing procedure described below is repeatedly executed by the control unit 30 of the terminal device 10 . Here, the processing denoted by the terminal device 10 is common to the terminal device 10 (10A) of the first user U (U1) and the terminal device 10 (10B) of the second user U (U2). Indicates processing.

As shown in FIG. 4, the acquiring unit 35 of the terminal device 10 collects sensor data (self sensor data) from sensors mounted on the terminal device 10 (step S101). For example, the acquisition unit 35 receives various information input by the user U using the input unit 13, various sensor data measured by the sensors 21 to 28 mounted on or connected to the terminal device 10, the positioning unit 14 The positional information of the terminal device 10 positioned by is collected as sensor data.

Subsequently, the transmission unit 31 of the terminal device 10 transmits the sensor data to another terminal device 10 owned by another user U via the communication unit 11 (step S102). For example, the transmission unit 31 of the terminal device 10 (10A) of the first user U (U1) transmits the sensor data to the terminal device 10 (10B) of the second user U (U2) via the communication unit 11. Send to At this time, the receiving unit 32 of the terminal device 10B receives the sensor data (other person's sensor data) of the terminal device 10A via the communication unit 11 . The same is true vice versa. That is, the sensor data is shared between the terminal device 10A and the terminal device 10B. Note that the processing unit 33 of the terminal device 10 may encrypt the sensor data. In this case, the transmission unit 31 may transmit the decryption key (public key or the like) to the other terminal device 10 together with the encrypted sensor data. The method of transmitting sensor data between devices may be P2P, home LAN (home hub format), or file compression.

Subsequently, the comparing unit 36 of the terminal device 10 compares the self sensor data and the other sensor data (step S103). At this time, the comparison unit 36 may compare the self sensor data and the other sensor data based on the position information. When the comparison unit 36 receives sensor data from another person via short-range wireless communication or LAN in the same premises, or when comparing sensor data collected in real time, the comparison unit The sensor data and the other person's sensor data may be directly compared.

Subsequently, the adopting unit 37 of the terminal device 10 adopts either one of the self sensor data and the other person's sensor data (preferred sensor data) according to the comparison result (step S104). For example, the adopting unit 37 adopts other person's sensor data having data not included in the self sensor data. Specifically, the adopting unit 37 adopts other person's sensor data collected by a sensor installed in another terminal device 10 but not installed in the terminal device 10 concerned. Also, the hiring unit 37 employs other person's sensor data that is more accurate than the self sensor data. In the present embodiment, the hiring unit 37 selects, among sensor data collected from other terminal devices 10, data that is not included in the sensor data of the terminal device 10, or data that is more accurate than the sensor data of the terminal device 10. Adopt data.

Subsequently, the learning unit 34 of the terminal device 10A performs on-device machine learning using the adopted sensor data as learning data based on the shared model (step S105). Note that the receiving unit 32 of the terminal device 10A may receive the sharing model provided from the information providing device 100 (server side) via the communication unit 11 . In addition, the transmission unit 31 of the terminal device 10A may transmit, via the communication unit 11, the difference parameter from the shared model as the learning result and the inference result to the information providing device 100 (server side). In the present embodiment, the learning unit 34 performs on-device machine learning using the sensor data adopted according to the comparison result as learning data. All of them may be used as learning data as they are for learning.

[5. Modification]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. So, below, the modification of embodiment is demonstrated.

In the above embodiment, part or all of the processing executed by the information providing device 100 may actually be executed by the terminal device 10 . For example, the processing may be completed stand-alone (by the terminal device 10 alone). In this case, it is assumed that the terminal device 10 has the functions of the information providing device 100 in the above embodiment. In addition, in the above-described embodiment, the terminal device 10 cooperates with the information providing device 100, so from the user U's point of view, it appears that the terminal device 10 is executing the processing of the information providing device 100 as well. That is, from another point of view, it can be said that the terminal device 10 includes the information providing device 100 .

In addition, in the above embodiment, a case is described in which sensor data is shared between individual terminal devices that perform machine learning. You may acquire sensor data from external devices other than the other terminal device which carries out. For example, the terminal device 10 of the user U may acquire sensor data from a fixed terminal device installed on the street or in a storefront, or may acquire sensor data from a mobile object such as a vehicle moving in the city. . Examples of fixed terminal devices installed on the street or in stores include terminals installed in stores, digital signage (electronic signboards), base stations, access points, and the like. In this case, these external devices correspond to other terminal devices owned by other users.

Further, in the above-described embodiment, the terminal device 10 may be a smart phone or the like that realizes advanced camera functions by on-device AI (Artificial Intelligence). Also, the terminal device 10 may be a robot, self-driving car, drone, or the like equipped with an on-device AI.

[6. effect〕
As described above, the terminal device 10 according to the present application includes the receiving unit 32 that receives other person's sensor data collected by the sensor of the other terminal device 10 from another terminal device 10 owned by another user U. and a learning unit 34 that performs machine learning using other person's sensor data as learning data.

In addition, the terminal device 10 according to the present application further includes an acquisition unit 35 that collects self-sensor data from the sensor of the terminal device 10 owned by the user U. Then, the learning unit 34 performs machine learning using the self sensor data and the other sensor data as learning data.

Further, the terminal device 10 according to the present application includes a comparison unit 36 that compares the self sensor data and the other person's sensor data, and one of the self sensor data and the other person's sensor data according to the comparison result. and a hiring unit 37 that employs the . Then, the learning unit 34 performs machine learning using the adopted sensor data as learning data.

Also, the comparison unit 36 compares the self sensor data and the other sensor data on the basis of the position information.

In addition, the hiring unit 37 employs other person's sensor data having data not included in the self sensor data.

The adopting unit 37 also adopts other person's sensor data collected by a sensor installed in another terminal device 10 but not installed in the terminal device 10 concerned.

Also, the hiring unit 37 employs other person's sensor data that is more accurate than the self sensor data.

In addition, the receiving unit 32 receives sensor data of others collected by sensors of other terminal devices 10 from other terminal devices 10 via short-range wireless communication or LAN in the same premises.

In addition, the receiving unit 32 receives, from the other terminal device 10 , sensor data of others collected in real time by the sensors of the other terminal device 10 .

By any one or combination of the above-described processes, the terminal device according to the present application can share sensor data among individual terminal devices that perform machine learning, and can learn using sensor data of other terminal devices.

[7. Hardware configuration]
Also, the terminal device 10 and the information providing device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 5, for example. Hereinafter, the terminal device 10 will be described as an example. FIG. 5 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (random access memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the terminal device 10 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 30 by executing programs loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[8. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing contents.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the control unit can be read as control means or a control circuit.

1 information processing system 10 terminal device 11 communication unit 14 positioning unit 20 sensor unit 30 control unit 31 transmission unit 32 reception unit 33 processing unit 34 learning unit 35 acquisition unit 36 comparison unit 37 employment unit 40 storage unit 100 information providing device

Claims (11)

a receiving unit that receives, from another terminal device owned by another user, other person's sensor data collected by a sensor of the other terminal device;
a learning unit that performs machine learning using the other person's sensor data as learning data;
A terminal device comprising: an acquisition unit that collects self-sensor data from a sensor of the terminal device owned by the user;
further comprising
The terminal device according to claim 1, wherein the learning unit performs machine learning using the self sensor data and the other sensor data as learning data. a comparison unit that compares the self sensor data and the other person's sensor data;
a hiring unit that adopts either one of the self sensor data and the other person's sensor data according to a comparison result;
further comprising
The terminal device according to claim 2, wherein the learning unit performs machine learning using the adopted sensor data as learning data. 4. The terminal device according to claim 3, wherein the comparison unit compares the self sensor data and the other sensor data based on position information. The terminal device according to claim 3 or 4, wherein the adopting unit adopts the other person's sensor data having data not included in the self sensor data. 6. The terminal according to claim 5, wherein the adopting unit adopts the other person's sensor data collected by a sensor installed in the other terminal device but not installed in the terminal device. Device. The terminal device according to any one of claims 3 to 6, wherein the adopting unit adopts the other person's sensor data having higher accuracy than the self sensor data. 3. The receiving unit receives sensor data of others collected by a sensor of the other terminal device from the other terminal device via short-range wireless communication or LAN in the same premises. 8. The terminal device according to any one of 1 to 7. 9. The receiving unit according to any one of claims 1 to 8, wherein the receiving unit receives, from the other terminal device, sensor data of others collected in real time by a sensor of the other terminal device. terminal equipment. An information processing method executed by a terminal device,
a receiving step of receiving, from another terminal device owned by another user, other person's sensor data collected by the sensor of the other terminal device;
a learning step of performing machine learning using the other person's sensor data as learning data;
An information processing method comprising: a receiving procedure for receiving other person's sensor data collected by the sensor of the other terminal device from another terminal device owned by another user;
a learning procedure for performing machine learning using the other person's sensor data as learning data;
An information processing program for executing a computer.

Images