JP7193507B2 - Information processing device, information processing method and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Techniques for transmitting location information of communication devices with low power consumption have been disclosed.

Japanese Patent Application Laid-Open No. 2020-042681

However, in the above-described conventional technology, the manually operated vehicle transmits communication device information including current position information to a communication partner by low power wide area wireless communication (LPWA: Low Power Wide Area). It is nothing more than sequential transmission with a transmission interval. Normally, a communication device does not consume much power just by acquiring position information, but a large amount of power is consumed when the communication device transmits position information to the outside (especially when moving at high speed). Even if the power consumption per transmission is reduced, the power consumption increases according to the number of times the location information is repeatedly transmitted. The power supply capacity may not be insufficient for manually operated vehicles such as those exemplified in the prior art above, but it is difficult to say that the power supply capacity is sufficient for terminal devices (handheld devices) such as smartphones. .

In addition, in recent years, from the perspective of personal information protection, there is a trend toward stricter restrictions on the transmission of location information from personal terminals due to the specifications of terminals such as smartphones. It has become extremely difficult to acquire the location information of the terminal device one by one. For example, regarding permission to use the location information of a personal terminal device, the setting of “always allow” is disappearing from options. In addition, there is a problem that location information cannot be transmitted when an application is running in the background under control of the OS (Operating System) side.

Moreover, terminal devices such as smartphones manufactured in recent years can acquire not only position information but also sensor data such as environmental sounds, acceleration, and atmospheric pressure. Acquisition of these sensor data facilitates estimation of the user's environment, making it possible to provide services more suitable for the user. However, there is also a problem that the business operators cannot acquire such sensor data due to the Personal Information Protection Law and the like.

The present application has been made in view of the above, and an object of the present application is to embed and transmit position information and sensor data.

An information processing apparatus according to the present application includes a first acquisition unit that acquires position information, a second acquisition unit that acquires sensor data from a sensor, a conversion unit that converts the position information and the sensor data into vectors, and a transmission unit that transmits a vector , wherein the conversion unit collectively converts a plurality of the position information and the sensor data into the single vector .

According to one aspect of an embodiment, location information and sensor data may be embedded and transmitted.

FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is an explanatory diagram showing an overview of the future data prediction method according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of an information processing system according to the embodiment; FIG. 4 is a diagram illustrating a configuration example of a terminal device according to the embodiment; FIG. 5 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment; FIG. 6 is a diagram showing an example of a conversion information database. FIG. 7 is a diagram showing an example of a vector information database. FIG. 8 is a sequence diagram illustrating a processing procedure according to the embodiment; FIG. 9 is a diagram illustrating an example of a hardware configuration;

Embodiments for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Also, in the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

[1. Outline of information processing method]
First, an outline of an information processing method performed by an information processing apparatus according to an embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram showing an outline of an information processing method according to an embodiment. In FIG. 1, a case where the terminal device embeds position information and sensor data and transmits them will be described as an example.

As shown in FIG. 1 , the information processing system 1 includes a terminal device 10 and an information providing device 100 . The terminal device 10 and the information providing device 100 are communicably connected to each other by wire or wirelessly via a network N (see FIG. 3).

The terminal device 10 is a smart device such as a smartphone or tablet used by a user U (user), and communicates with an arbitrary server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). It is a portable terminal device capable of performing The terminal device 10 has a screen such as a liquid crystal display, which has a touch panel function. accepts the operation of An operation performed on an area where content is displayed on the screen may be an operation on the content. In addition, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC as well as a smart device.

Here, the terminal device 10 acquires its own position information and sensor data, and converts (vectorizes) the acquired position information and sensor data into a vector. That is, the terminal device 10 performs so-called embedding to generate an embedding vector based on position information and sensor data. The terminal device 10 then transmits an embedding vector based on the positional information and the sensor data to the information providing device 100 . Sensor data are, for example, environmental sounds, acceleration, atmospheric pressure, and the like.

The information providing device 100 receives the embedding vector based on the position information and the sensor data from the terminal device 10, and based on the embedding vector based on the position information and the sensor data, the behavior (location, behavior, movement, etc.) of the user U is determined. It is an information processing device that analyzes and is realized by a server device, a cloud system, or the like.

[1-1. Advantages of converting position information into vectors]
First, the advantage of converting position information into a vector will be described.

Conventionally, the terminal device 10 transmits location information to the information providing device 100. At this time, the terminal device 10 performs rounding of the location information in consideration of the privacy of the user U and in order to reduce the amount of data communication. (Rounding processing) is performed, and the position information is rounded and converted to coarse position information before transmission. For example, if the location information is latitude and longitude information of GPS (Global Positioning System) coordinates, round up or round down (rounding off is also possible) the value below the second decimal place in the decimal angle of latitude and longitude, and roughly Values were converted to location information.

In this case, when the information providing apparatus 100 tries to estimate the behavior of the user U based on the position information, since the rounded position information is a rough value, a large area unit such as a region or a large-scale facility is Even if it can be specified, it may not be possible to specify a narrow place (and/or the difference between indoors and outdoors) such as an adjacent small building or room. Therefore, although the general behavior of the user U can be estimated, the detailed behavior of the user U may not be estimated. For example, if a small building A and a building B (for example, a convenience store and a fast food restaurant) are built adjacent to each other within the range of the location indicated by the rounded position information of the user U, the user U In some cases, it was not possible to specify from the location information which part of B the person was in, or whether the person was inside or outside the building (or site). What the information providing apparatus 100 (server side) really wants to know is not the location information of the user U itself, but the behavior of the user U indicated by the location information (that is, the context of the user U).

This time, when the terminal device 10 converts (vectorizes) its own position information into a vector, the terminal device 10 converts the raw data of the position information before fraction processing (rounding processing) into a vector. Therefore, the terminal device 10 can generate an embedding vector after identifying a small space (and/or the difference between indoors and outdoors) such as an adjacent small building or room. Therefore, the information providing apparatus 100 can know the detailed behavior of the user U based on the embedding vector. That is, the information providing apparatus 100 can acquire specific and detailed information about the behavior of the user U from the terminal apparatus 10 rather than receiving conventional rounded position information. Further, since it is the information providing apparatus 100 (on the server side) that defines the vector values of the embedding vectors and their meanings, even if a third party sees only the vector values of the embedding vectors, what are the vector values? I don't understand what you mean, so it's safe.

[1-2. Outline of information processing method for embedding and transmitting position information and sensor data]
Next, an outline of an information processing method for embedding and transmitting position information and sensor data will be described with reference to FIG.

As shown in FIG. 1, the terminal device 10 of the user U acquires its own location information (step S1).

Then, the terminal device 10 acquires the sensor data together with the position information (step S2).

Then, the terminal device 10 generates a set of position information and sensor data (step S3).

Then, the terminal device 10 embeds the sets of the position information and the sensor data, converts (vectorizes) a plurality of sets of the position information and the sensor data into one vector, and converts the position information and the sensor data into one vector. generates an embedding vector based on (step S4).

Then, the terminal device 10 transmits the embedding vector based on the positional information and the sensor data to the information providing device 100 (step S5).

Then, the information providing apparatus 100 receives the embedding vector based on the positional information and the sensor data from the terminal device 10 (step S6).

Then, the information providing device 100 analyzes the behavior (location, behavior, movement, etc.) of the user U based on the embedding vector received from the terminal device 10 (step S7).

[1-3. Example 1]
Next, a first embodiment of an information processing method for embedding and transmitting the position information will be described.

For example, in step S1, the terminal device 10 possessed by each of a plurality of users U located in each region uses positioning technology such as GNSS (Global Navigation Satellite System) and indoor positioning technology, Each acquires its own location information. At this time, the terminal device 10 acquires the time information indicating the time when the position information was acquired together with its own position information.

At the same time, in step S2, the terminal device 10 acquires sensor data such as environmental sound, acceleration, and atmospheric pressure using sensors mounted on or connected to the terminal device 10 along with the position information.

The connection may be wired connection or wireless connection. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices.

In this way, by acquiring sensor data in addition to position information, the accuracy of estimating the user environment (user U's context, etc.) is improved. For example, sensor data such as user U's environmental sound, acceleration, and air pressure support position information and user environment. In addition, it is possible to recognize changes in the user's environment (changes in environmental sound, acceleration, atmospheric pressure, etc.) at the same position.

At this time, the terminal device 10 accumulates the acquired position information as a position history (for example, location history, action history, etc.). In addition, the terminal device 10 accumulates the acquired various sensor data for each type. Note that the terminal device 10 may associate and accumulate the position information and the sensor data for each acquired time or time zone.

Subsequently, in step S3, the terminal device 10 generates a set of position information and sensor data at the same time or in the same time zone, and arranges the sets of position information and sensor data in time series (time series data). data).

Subsequently, in step S4, the terminal device 10 generates an embedding vector based on sequence data in which pairs of position information and sensor data are arranged in time series. Here, the terminal device 10 uses sequence data as a data set and converts the sequence data into a multidimensional real-valued vector using a neural network-based machine learning method or the like. At this time, the terminal device 10 inputs sequence data to a vector conversion model constructed through machine learning using, for example, RNN (Recurrent Neural Network) or LSTM (Long short-term memory), and outputs an embedding vector. get. The terminal device 10 may receive the vector conversion model from the information providing device 100 .

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Attention can handle data such as sentences in which the sequence is important. Also, the terminal device 10 may use a similar natural language processing model. By using such a model to generate an embedding vector from sequence data, it is possible to generate an embedding vector that emphasizes the order of information.

Alternatively, the terminal device 10 may convert the sequence data into the embedding vector according to a predetermined conversion rule (rule base).

However, sequence data is only an example. In practice, for example, the raw data of the position information may be input to the model in chronological order.

Here, when the terminal device 10 embeds based on the set data of the position information and the sensor data in accordance with the vector conversion model and the predetermined conversion rule, the output value becomes a vector representation. By arranging these output vectors in chronological order, the moving route of the user is shown. For example, although the specific latitude and longitude are unknown, the movement from point A to point B on the vector space is indicated. That is, the embedding vector is the result (output value) of embedding by the terminal device 10 using a set of positional information and sensor data. This embedding vector is a value indicating user behavior (a value indicating context).

Note that the timing at which the terminal device 10 performs the embedding is arbitrary. For example, the accumulated location information may be embedded at the timing when transmission of location information from the terminal device 10 to the information providing device 100 is permitted. Alternatively, accumulation of location information may be started when an application (app) is activated, and the accumulated location information may be embedded at the timing when use of the application is terminated. Also, when the number of accumulated positional information reaches a predetermined number (for example, 100), the accumulated positional information may be embedded. Alternatively, the accumulated position information may be embedded when the user moves from the place where he/she has stayed until then (when the position information changes significantly).

Subsequently, in step S5, the terminal device 10 transmits the embedding vector based on the combination of the positional information and the sensor data to the information providing device 100 via the network N (see FIG. 3).

The timing at which the terminal device 10 transmits the embedding vector based on the combination of the positional information and the sensor data to the information providing device 100 is arbitrary. For example, it may be the timing at which transmission of position information from the terminal device 10 to the information providing device 100 is permitted. Alternatively, it may be at the start of an application (app), during use, at a predetermined time, at a predetermined cycle (every two hours, etc.), or the like. Alternatively, the information providing apparatus 100 may request the terminal apparatus 10 to transmit an embedding vector based on a combination of position information and sensor data.

Further, when the timing is such that transmission of location information from the terminal device 10 to the information providing device 100 is permitted, the terminal device 10 transmits its own information at that time together with an embedding vector based on a combination of location information and sensor data. may be transmitted to the information providing apparatus 100 . That is, the terminal device 10 may transmit a set of the embedding vector based on the set of the position information and the sensor data and its own position information to the information providing device 100 .

Subsequently, in step S6, the information providing device 100 receives the terminal device 10 possessed by each of the plurality of users U located in each area via the network N, based on the sets of position information and sensor data of each of the users U. Receive an embedding vector.

Subsequently, in step S7, the information providing apparatus 100 arranges the embedding vector in the specific space. By arranging the embedding vector in a specific space, the information providing apparatus 100 clearly grasps the similarity and change in behavior of the user U himself and/or the similarity and difference in behavior between the user U and other users. can do. For example, since the terminal device 10 embeds a set of position information (latitude and longitude, etc.) and sensor data, it generates an embedding vector of the same value when the terminal device 10 is at the same position. The information providing apparatus 100 arranges the embedding vectors of the same value at the same position in the specific space.

In this case, the information providing apparatus 100 is not mapped (associated) with a position on the map (latitude and longitude, etc.), so a specific position cannot be specified. It is possible to confirm specific actions of the user U, such as moving to a point, then moving to a point C, and returning to a point A again. Also, the information providing apparatus 100 can confirm the fact that the user U has moved without using the position information.

As described above, in this embodiment, when transmitting user position information and sensor data, the position information and sensor data acquired by the terminal device are not directly transmitted to the receiving side (server side). It embeds information and sensor data before sending it to the receiver. Since the embedded information is difficult to restore on the receiving side, it becomes difficult to specify accurate location information and sensor data, and the problem of personal information is resolved. In other words, not only is the amount of information reduced, but the raw data (location information and sensor data itself) detected by the terminal side cannot be restored by the server side, so it is secure. In addition, by embedding, a group of location information and sensor data is converted into a vector that indicates a specific action taken by the user (going to work, shopping, traveling by train, etc.) and transmitted. This eliminates the need to send messages one by one, and solves the problem of battery consumption.

Furthermore, in this embodiment, information such as environmental sound, acceleration, acceleration, atmospheric pressure, etc. is combined with position information and embedded, and transmitted, thereby significantly reducing the amount of data. Transmitting this information as raw data results in a large amount of data, but by embedding it, it can be transmitted as a single vector.

[1-4. Example 2]
Next, a second embodiment of the information processing method for embedding and transmitting the position information will be described.

For example, in step S5, the terminal device 10 may transmit some of its own positional information to the information providing device 100 along with an embedding vector based on a set of positional information and sensor data. That is, the terminal device 10 may transmit a set of the embedding vector based on the set of the position information and the sensor data and the position information of the terminal device 10 to the information providing device 100 .

For example, when there were no restrictions imposed on the transmission of location information (when location information could be transmitted freely), the terminal side would receive an average of 17 locations per day if “always allowed”. In the past, information was sent to the server side, but recently restrictions have become stricter, and the terminal side is now sending only about three pieces of location information to the server side. Therefore, the terminal device 10 may, for example, generate an embedding vector obtained by vectorizing 14 points of position information out of 17 points of position information, and transmit the embedding vector together with the remaining 3 points of position information to the server side. That is, the terminal device 10 may transmit embedding information obtained by vectorizing part of position information together with sensor data, and the remaining position information itself. The rest of the position information is the number of pieces of position information that can be transmitted by the terminal. In addition, the terminal device 10 may transmit a set of sensor data together with the remaining position information.

The embedding vector indicates the behavior of the user U, but does not indicate the specific position of the user U. Therefore, the information providing apparatus 100 (on the server side) cannot know in which region the action of the user U took place. Therefore, by combining the embedding vector and the rounded position information, the information providing apparatus 100 (on the server side) can roughly know the area where the action of the user U took place. For example, the embedding vector can indicate "user U had lunch (lunch)", but does not indicate "in which region" user U had lunch. By adding rounded position information indicating a rough place (for example, △△ town in ○○ ward) to this, it is possible to know the rough place where the user U had lunch.

Further, it may be implemented in combination with the first embodiment described above. For example, in step S7, when arranging the embedding vector in the specific space, the information providing apparatus 100 uses the three-point positional information to map the specific space and the map. It can be mapped to a position above (latitude, longitude, etc.). As a result, the information providing apparatus 100 can estimate the location to some extent, although not completely.

In addition, since the terminal device 10 embeds position information (latitude and longitude, etc.), it generates an embedding vector of the same value when the terminal device 10 is at the same position. Therefore, the information providing apparatus 100 can also estimate that the user U is staying at the same place based on the embedding vector of the same value. As a result, the information providing apparatus 100 can determine whether the user U is “at home” when the user U stays in the same place for a long time at night, or when the user U stays at the same place for a long time, or when the user U normally stays at the same place during the day. It is also possible to presume that the person is "at work" if he or she stays in a place different from nighttime for a long time.

[1-5. Example 3]
Next, a third embodiment of an information processing method for embedding and transmitting the position information will be described.

For example, in step S3, the terminal device 10 performs actions of the user U represented by two or more sets of position information and sensor data based on sequence data in which sets of position information and sensor data are arranged in time series. An embedding vector may be generated in which (user's context) is indicated by a vector value.

For example, the terminal device 10 inputs sequence data into a vector transformation model generated through machine learning using RNN, LSTM, etc., and acquires an embedding vector as an output. At this time, even if the RNN and LSTM are the same data, the output changes depending on the input order. For example, if you move in the order of "position 1" → "position 2" → "position 3", output "I went to work" (outbound), and then "position 3" → "position 2" → "position 1" If you are moving with , output "I came home" (return trip). By using such a model that provides information according to the time series of information, it is possible to generate an embedding vector based on the characteristics of changes in position information.

In this way, when an embedding vector is generated from the sequence data using the LSTM, an embedding vector indicating whether or not the user has performed a predetermined action is generated from the behavior of the user U indicated by the sequence data (context of the user U). be able to. For example, if the sequence data of the combination of the position information and the sensor data indicates that the user U has moved from the area where the home is located to the area where the workplace is located, the terminal device 10 considers the action of "going to work". An embedding vector can be generated in which the corresponding elements take predetermined values.

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Also, the terminal device 10 may use a similar natural language processing model.

Alternatively, the terminal device 10 may convert the sequence data into the embedding vector according to a predetermined conversion rule. For example, the terminal device 10 determines that the position information #1 obtained first is within a predetermined range from the home of the user U, the position information #2 obtained next is within a predetermined range from the station, and finally If the acquired location information #3 is within a predetermined range from the place of work, an embedding vector may be generated in which the element corresponding to the action of "arrived at work" takes a predetermined value.

However, sequence data is only an example. In practice, for example, the raw data of the position information may be input to the model in chronological order.

In addition, for example, when the location information acquired in 30 minutes does not change significantly, the terminal device 10 obtains the average value, the median value, etc. of the location information acquired in the 30 minutes, and uses this as the location information for the 30 minutes. and an embedding vector may be generated from the position information.

Then, in step S7, the information providing apparatus 100 determines whether the user U is "at home", "goes to work", "goes shopping", or "goes shopping", based on the vector value of the embedding vector based on the combination of the position information and the sensor data. Action (user's context) such as "I am traveling by train" is determined. At this time, the information providing apparatus 100 may determine behavior for each time period based on values indicated by a plurality of elements of the multidimensional embedding vector. For example, the information providing apparatus 100 determines that "I went to work in the morning" when the embedding vector includes a value indicating "I went out in the morning" and a value indicating "I went to work in the morning". You may

Here, the values of the elements of the embedding vector correspond to user U's actions. For example, in the case of a three-dimensional vector, the value of the first element indicates "whether you went to work", the value of the second element indicates "whether you ate out", and the value of the third element indicates "whether you took the train". It may be specified to indicate whether or not In this case, the embedding vector may be a One-hot vector (One Hot Vector). A one-hot vector is a vector such as (1,0,0) in which one component is 1 and all remaining components are 0. However, in practice, the embedding vector is not limited to the one-hot vector.

Also, the values of the elements of the embedding vector may correspond to a plurality of user U actions. For example, the value of the first element indicates "whether or not you went to work and ate out", and the value of the second element indicates "whether or not you went to work and ate out". may

Values of a plurality of elements included in a certain range of the embedding vector may correspond to user U's behavior. For example, if the value up to the third element (upper three digits) is "100", it indicates that "I went to work", and if it is "101", it indicates that "I went to work by train".

Also, the value of each element of the embedding vector may be an evaluation value. For example, if the value up to the third element (upper three digits) is "000", it indicates "not moved", and if it is "100", it indicates that "it has moved 100 kilometers". Also, the value of any element of the embedding vector may be a value indicating the movement direction. For example, if the value of the 4th to 7th elements is "0000", it indicates that the movement direction is "north", "0001" indicates that the movement direction is "north-northeast", and "0010" indicates that the movement direction is "north-northeast". Indicates that the movement direction is "northeast", "0011" indicates that the movement direction is "east northeast", "0100" indicates that the movement direction is "east", and "1000" indicates the movement direction. indicates "south", and "1100" indicates that the direction of travel is "west". Also, the value of each element of the embedding vector may be a healthy living score.

It is assumed that the values of the elements of the embedding vector can be set arbitrarily when learning the model or creating the conversion rule. For example, it is assumed that the server side can make arbitrary settings according to circumstances such as what kind of behavior information the server wants to acquire from the user U side and use for services provided to the user U.

Here, a vector conversion model, a predetermined conversion rule, etc. for converting a pair of position information and sensor data into an embedding vector are used to convert the sequence data of the pair of position information and sensor data into a , ``I went to work'', ``I went shopping'', ``I'm traveling by train'', etc. is. For example, when the information providing apparatus 100 wants to know whether or not the user U has taken a specific action, the information providing apparatus 100 uses a vector conversion model or a predetermined A conversion rule or the like is provided to the terminal device 10 of the user U.

[1-6. Overview of future data prediction method for pairs of position information and sensor data]
Next, with reference to FIG. 2, an outline of a method of predicting future data of pairs of position information and sensor data will be described. FIG. 2 is an explanatory diagram showing an overview of the future data prediction method according to the embodiment.

The terminal device 10 of the user U predicts future data based on the past data of the combination of the position information and the sensor data, if necessary. Prediction of future data for a set of position information and sensor data is merely an example. In practice, the terminal device 10 does not need to predict future data of a set of location information and sensor data.

As shown in FIG. 2, when the terminal device 10 of the user U generates a set of location information and sensor data (step S3 in FIG. 1), the location information and Machine learning is performed on pairs with sensor data, and a prediction model for future data based on past data is constructed.

In the example shown in FIG. 2, a set of position information P and sensor data SD1 and SD2 is generated for each time T, and machine learning of distributed representation is performed as a data set D. In the example shown in FIG. Here, sensor data SD1 indicates environmental sound, and sensor data SD2 indicates acceleration. For example, when the time T is "t-2", a set of the position information P1 and the sensor data SD11 and SD21 is generated and set as the data set D1.

Here, when a set of position information and sensor data at each of times “t−2”, “t−1”, and “t” is input, the terminal device 10 performs , “t+2”, and “t+3” so that sets of position information and sensor data are output. As a result, when the sensor data is combined with the position information at the past and present times, the sensor data corresponding to the position information at the future time is obtained.

For example, the prediction model is an encoder-decoder model. This is a model using two RNNs, and if learning is performed in the order of "(past data) → first RNN (encoder) → vector → second RNN (decoder) → (future data)" , the vector output by the first RNN captures the characteristics of the data well. If the characteristics of the data cannot be captured well, the accuracy of the data will decrease in the future. Conversely, modifying the first RNN to increase the precision of future data will improve the precision of the vector.

Here, the terminal device 10 uses the encoder of the encoder-decoder model to generate the distributed representation. A decoder is then used to predict the future from the generated distributed representation. Thereby, the terminal device 10 estimates a set of position information and sensor data to be acquired in the future from a set of past position information and sensor data.

Thus, the encoder and decoder become separate RNNs. When the encoder is trained, the performance of the encoder can be improved by jointly modifying the encoder and decoder so that the decoder can properly predict the future from the distributed representation output by the encoder.

As a result, even if a situation occurs in which sensor data cannot be acquired at a certain time, the set of position information and sensor data up to that time is input to the prediction model, and the set of position information and sensor data obtained as an output. based on the sensor data that should have been acquired at that time can be inferred. Moreover, it is possible to generate an embedding vector that predicts the future behavior of the user U, not just the embedding vector that indicates the past or current behavior of the user U. Also, the embedding vector generation accuracy is improved.

[2. Configuration example of information processing system]
Next, the configuration of the information processing system 1 including the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 3 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. As shown in FIG. 3, the information processing system 1 according to the embodiment includes a terminal device 10 and an information providing device 100. As shown in FIG. These various devices are communicatively connected via a network N by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

Also, the number of devices included in the information processing system 1 shown in FIG. 3 is not limited to the illustrated one. For example, only one terminal device 10 is shown in FIG. 3 for simplification of illustration, but this is only an example and is not limited, and two or more devices may be provided.

The terminal device 10 is an information processing device used by the user U. For example, the terminal device 10 includes smart devices such as smartphones and tablet terminals, feature phones, PCs (Personal Computers), PDAs (Personal Digital Assistants), car navigation systems, and wearable devices such as smart watches and head-mounted displays. , smart glasses, etc.

In addition, the terminal device 10 is compatible with wireless communication networks such as LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation: fifth generation mobile communication system), Bluetooth (registered trademark), wireless LAN (Local It is possible to communicate with the information providing apparatus 100 by connecting to the network N via short-range wireless communication such as Area Network).

The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, or the like. Note that the information providing apparatus 100 may be realized by cloud computing.

[3. Configuration example of terminal device]
Next, the configuration of the terminal device 10 will be described with reference to FIG. FIG. 4 is a diagram showing a configuration example of the terminal device 10. As shown in FIG. As shown in FIG. 4, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40. Prepare.

(Communication unit 11)
The communication unit 11 is connected to the network N (see FIG. 3) by wire or wirelessly, and transmits and receives information to and from the information providing apparatus 100 via the network N. FIG. For example, the communication unit 11 is implemented by a NIC (Network Interface Card), an antenna, or the like.

(Display unit 12)
The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic EL display (Organic Electro-Luminescent Display). Also, the display unit 12 is a touch panel display, but is not limited to this.

(Input unit 13)
The input unit 13 is an input device that receives various operations from the user U. FIG. The input unit 13 has, for example, buttons and the like for inputting characters, numbers, and the like. Moreover, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13 . Note that the input unit 13 may be a microphone or the like that receives voice input from the user U. FIG. The microphone may be wireless.

(Positioning unit 14)
The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites, and based on the received signals, position information (for example, latitude and longitude). That is, the positioning unit 14 positions the position of the terminal device 10 . GPS is merely an example of GNSS (Global Navigation Satellite System).

Also, the positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position using various communication functions of the terminal device 10 as described below as auxiliary positioning means for position correction and the like.

(Wi-Fi positioning)
For example, the positioning unit 14 measures the position of the terminal device 10 using the Wi-Fi (registered trademark) communication function of the terminal device 10 or the communication network provided by each communication company. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the position of the terminal device 10 by measuring the distance to a nearby base station or access point.

(beacon positioning)
The positioning unit 14 may also use the Bluetooth (registered trademark) function of the terminal device 10 to measure the position. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting with a beacon transmitter connected by the Bluetooth (registered trademark) function.

(geomagnetic positioning)
Further, the positioning unit 14 positions the position of the terminal device 10 based on the geomagnetism pattern of the structure measured in advance and the geomagnetic sensor provided in the terminal device 10 .

(RFID positioning)
Further, for example, if the terminal device 10 has an RFID (Radio Frequency Identification) tag function equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag In this case, the location used is recorded together with the information that the payment was made by the terminal device 10 . The positioning unit 14 may measure the position of the terminal device 10 by acquiring such information. Also, the position may be measured by an optical sensor provided in the terminal device 10, an infrared sensor, or the like.

The positioning unit 14 may measure the position of the terminal device 10 using one or a combination of the positioning means described above, if necessary.

(Sensor unit 20)
The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10 . The connection may be wired connection or wireless connection. For example, the sensor unit 20 may be a detection device other than the terminal device 10, such as a wearable device or a wireless device. In the example shown in FIG. 4, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an atmospheric pressure sensor 23, an air temperature sensor 24, a sound sensor 25, an optical sensor 26, a magnetic sensor 27, and an image sensor ( camera) 28.

The sensors 21 to 28 described above are only examples and are not limited. That is, the sensor unit 20 may be configured to include a part of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28. .

The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10 such as movement direction, speed, and acceleration of the terminal device 10 . The gyro sensor 22 detects physical movements of the terminal device 10 such as inclination in three axial directions based on the angular velocity of the terminal device 10 and the like. The atmospheric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

Since the terminal device 10 includes the above-described acceleration sensor 21, gyro sensor 22, barometric pressure sensor 23, etc., techniques such as pedestrian dead-reckoning (PDR: Pedestrian Dead-Reckoning) using these sensors 21 to 23, etc. , the position of the terminal device 10 can be determined. This makes it possible to acquire indoor position information that is difficult to acquire with a positioning system such as GPS.

For example, a pedometer using the acceleration sensor 21 can calculate the number of steps, walking speed, and distance walked. In addition, by using the gyro sensor 22, it is possible to know the traveling direction, the direction of the line of sight, and the inclination of the body of the user U. Also, from the atmospheric pressure detected by the atmospheric pressure sensor 23, the altitude at which the terminal device 10 of the user U is present and the number of floors can be known.

The temperature sensor 24 detects the temperature around the terminal device 10, for example. The sound sensor 25 detects, for example, sounds around the terminal device 10 (environmental sounds, etc.). The optical sensor 26 detects the illuminance around the terminal device 10 . The magnetic sensor 27 detects, for example, geomagnetism around the terminal device 10 . The image sensor 28 captures an image around the terminal device 10 .

The atmospheric pressure sensor 23, the temperature sensor 24, the sound sensor 25, the optical sensor 26, and the image sensor 28 described above detect the atmospheric pressure, temperature, sound, and illuminance, respectively, or capture an image of the surroundings to detect the terminal device 10. It is possible to detect the surrounding environment and situations. In addition, it is possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and situation of the terminal device 10 .

(control unit 30)
The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input/output port, and various circuits. Also, the control unit 30 may be configured by hardware such as an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31 , a reception unit 32 , a processing unit 33 , an acquisition unit 34 , a conversion unit 35 , a prediction unit 36 and a learning unit 37 .

(Sending unit 31)
The transmission unit 31 can transmit, for example, various information input by the user U using the input unit 13 and various information processed by the processing unit 33 to the information providing apparatus 100 via the communication unit 11. .

The transmitting unit 31 also transmits the embedding vector converted from the positional information by the converting unit 35 to the information providing apparatus 100 via the communication unit 11 . At this time, the transmitting unit 31 may transmit the position information at that time together with the embedding vector.

(Receiver 32)
The receiving unit 32 receives various information provided from the information providing device 100 via the communication unit 11 . For example, the receiving unit 32 receives a vector conversion model or the like provided from the information providing device 100 via the communication unit 11 .

(Processing unit 33)
The processing unit 33 controls the entire terminal device 10 including the display unit 12 and the like. For example, the processing unit 33 can output various types of information transmitted by the transmitting unit 31 and various types of information received by the receiving unit 32 from the information providing apparatus 100 to the display unit 12 for display.

(Acquisition unit 34)
The acquisition unit 34 acquires the location information of the terminal device 10 constantly/periodically/at a predetermined timing. For example, the acquisition unit 34 acquires position information of the terminal device 10 positioned by the positioning unit 14 . In addition, the acquisition unit 34 acquires sensor data detected by each of the sensors 21-28. Also, the acquisition unit 34 estimates and acquires the indoor position information of the terminal device 10 based on the sensor data detected by the sensors 21 to 28 .

The acquisition unit 34 also stores (saves) the acquired position information and sensor data in the storage unit 40 . For example, the acquisition unit 34 accumulates the acquired position information as a position history (for example, location history, action history, etc.). The acquisition unit 34 also accumulates sensor data detected by each of the sensors 21 to 28 for each type. Note that the acquisition unit 34 may store the location information and the sensor data in association with each acquired time or time period.

The acquisition unit 34 also acquires the vector conversion model and the prediction model that the reception unit 32 has received from the information providing device 100 and stores (saves) them in the storage unit 40 .

Alternatively, the acquisition unit 34 acquires a predetermined conversion rule (rule) received by the reception unit 32 from the information providing apparatus 100 and stores (saves) it in the storage unit 40 .

(Converter 35)
The conversion unit 35 performs embedding on pairs of position information and sensor data, converts (vectorizes) a plurality of pairs of position information and sensor data into one vector, and converts (vectorizes) a plurality of pairs of position information and sensor data into a vector based on the position information and sensor data. Generate an embedding vector.

For example, the conversion unit 35 generates a set of position information and sensor data at the same time or in the same time zone, and generates sequence data in which the sets of position information and sensor data are arranged in time series. Then, the conversion unit 35 generates an embedding vector based on the sequence data.

For example, the conversion unit 35 uses sequence data as a data set and converts the sequence data into a multidimensional real-valued vector using a neural network-based machine learning method or the like. That is, the transformation unit 35 inputs sequence data into a vector transformation model constructed through machine learning using, for example, RNN (Recurrent Neural Network) or LSTM (Long short-term memory), and acquires an embedding vector as an output. do.

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Also, the conversion unit 35 may use a similar natural language processing model.

Alternatively, the conversion unit 35 may convert the sequence data into the embedding vector according to a predetermined conversion rule (on a rule basis).

However, sequence data is only an example. In practice, for example, raw data such as position information and sensor data may be input to the model in chronological order as they are.

(Prediction unit 36)
The prediction unit 36 predicts future data based on past data in pairs of position information and sensor data. For example, even if a situation occurs in which sensor data cannot be acquired at a certain time, the prediction unit 36 inputs a set of position information and sensor data up to that time into the prediction model, and outputs the position information and the sensor data. Based on the set with the data, the sensor data that should have been acquired at that time is inferred.

(Learning part 37)
The learning unit 37 performs machine learning on pairs of position information and sensor data using a machine learning method using a neural network, etc., and creates a vector conversion model for converting pairs of position information and sensor data into embedding vectors. to build.

In addition, the learning unit 37 performs machine learning on pairs of position information and sensor data using a neural network-based machine learning method or the like, and builds a future data prediction model based on past data.

For example, the learning unit 37 constructs a vector conversion model and a prediction model by machine learning using RNN (Recurrent Neural Network), LSTM (Long short-term memory), or the like. As for the model, a model of any type and any format can be adopted. For example, the learning unit 37 may employ an SVM (Support Vector Machine) or a DNN (Deep Neural Network) as a model. Here, DNN may be CNN (Convolutional Neural Network) or RNN. The model may also be a model realized by combining a plurality of models, such as a model combining CNN and RNN.

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Also, the learning unit 37 may use a similar natural language processing model.

Learning is, for example, deep learning using DNN. Data mining and other machine learning algorithms may also be used. The learning unit 37 learns the model using the various learning methods described above.

(storage unit 40)
The storage unit 40 is realized by, for example, a semiconductor memory device such as RAM (Random Access Memory) or flash memory, or a storage device such as HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disk. be. Various programs, various data, and the like are stored in the storage unit 40 . Here, the storage unit 40 stores, for example, various information input by the user U using the input unit 13, sensor data detected by the sensors 21 to 28 mounted on or connected to the terminal device 10, Positional information and the like of the terminal device 10 that has been positioned is stored. Further, various information processed by the processing unit 33 and various information received by the receiving unit 32 are stored. The storage unit 40 also stores a vector conversion model, a prediction model, or a predetermined conversion rule.

[4. Configuration example of information processing device]
Next, the configuration of the information providing device 100 according to the embodiment will be described using FIG. FIG. 5 is a diagram showing a configuration example of the information providing device 100 according to the embodiment. As shown in FIG. 5, the information providing device 100 has a communication section 110, a storage section 120, and a control section .

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to the network N (see FIG. 3) by wire or wirelessly.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 5 , the storage unit 120 has a conversion information database 121 and a vector information database 122 .

(Conversion information database 121)
The conversion information database 121 stores various information related to vector conversion models, prediction models, predetermined conversion rules, and the like. FIG. 6 is a diagram showing an example of the conversion information database 121. As shown in FIG. In the example shown in FIG. 6, the conversion information database 121 has items such as "conversion ID", "conversion information", and "provider".

"Conversion ID" indicates identification information for identifying a vector conversion model, a prediction model, or a predetermined conversion rule. The common model is a model managed by the server and common to each user.

"Conversion information" indicates a file in which a vector conversion model or a prediction model identified by a conversion ID, or a predetermined conversion rule or the like is converted into data.

Also, the "provider" indicates the terminal device 10 to which the vector conversion model, the prediction model, or the predetermined conversion rule identified by the conversion ID is provided. For example, a terminal ID may be stored in the "provider". A plurality of terminal devices 10 may be provided. That is, a plurality of terminal IDs may be stored in the "provider". Note that if there is no need to specify the terminal device 10 to be provided, the item "provided destination" may be omitted.

For example, in the example shown in FIG. 6, a vector transformation model or a prediction model, which is transformation information “transformation information #1” identified by transformation ID “T1”, or a predetermined transformation rule, etc. ” indicates that it is provided to

Note that the conversion information database 121 is not limited to the above, and may store various types of information depending on the purpose. For example, the conversion information database 121 may store detailed information of common models identified by conversion IDs.

(Vector information database 122)
The vector information database 122 stores various types of information regarding embedding vectors based on sets of position information and sensor data received from the terminal device 10 . FIG. 7 is a diagram showing an example of the vector information database 122. As shown in FIG. In the example shown in FIG. 7, the vector information database 122 has items such as "terminal ID", "embedding vector", and "determination result".

"Terminal ID" indicates identification information for identifying the terminal device 10 that is the source of the embedding vector. The “terminal ID” may be an arbitrary number assigned by the server, or contact information (telephone number, e-mail address, etc.) of the terminal device 10 . If there is no need to identify the terminal device 10, the item "terminal ID" may be omitted.

Also, “embedding vector” indicates an embedding vector based on a set of position information and sensor data received from the terminal device 10 . A plurality of “embedding vectors” may exist for one terminal device 10 . For example, if the terminal device 10 generates an embedding vector every 30 minutes, the embedding vector for every 30 minutes is stored.

Also, the "determination result" indicates the behavior (location, behavior, movement, etc.) of the user U determined based on the embedding vector based on the combination of the position information and the sensor data.

For example, in the example shown in FIG. 7, the embedding vector "Vector #11" received from the terminal device 10 identified by the terminal ID "Terminal #1" indicates that the behavior of the user U of the terminal device 10 is the determination result "Determination result # 11”.

The vector information database 122 is not limited to the above, and may store various types of information depending on the purpose. For example, the vector information database 122 may store identification information (vector ID, etc.) for identifying each embedding vector for each received data unit. The vector information database 122 may also store information indicating the date and time when each embedding vector was received. The vector information database 122 may also store information on rules for interpreting embedding vectors. The vector information database 122 may also store conversion information such as vector conversion models, prediction models, or predetermined conversion rules.

(control unit 130)
Returning to FIG. 5, the description is continued. The control unit 130 is a controller, and for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like controls the information providing apparatus 100. Various programs (corresponding to an example of an information processing program) stored in the internal storage device are executed by using a storage area such as a RAM as a work area. In the example illustrated in FIG. 5 , the control unit 130 has a collection unit 131 , a determination unit 132 , a generation unit 133 and a provision unit 134 .

(Collection unit 131)
The collection unit 131 collects embedding vectors based on sets of position information and sensor data from individual terminal devices 10 . That is, the collection unit 131 receives the embedding vector based on the combination of the position information and the sensor data from the terminal device 10 via the communication unit 110 .

(Determination unit 132)
The determination unit 132 analyzes the embedding vector based on the set of the collected positional information and sensor data, and determines the behavior (location, behavior, movement, etc.) of the user U.

Further, the determination unit 132 arranges the embedding vectors in a specific space, and determines similarities and changes in behavior of the user U himself and/or similarities and differences in behavior between the user U and other users.

(Generating unit 133)
The generation unit 133 converts position information and sensor data into an embedding vector based on an existing position history (or assumed position information) that accumulates position information collected in the past and an existing sensor history that accumulates sensor data. A vector conversion model, a prediction model, or a predetermined conversion rule (rule) for conversion is generated.

For example, the generation unit 133 generates a vector conversion model that outputs an embedding vector when position information is input by having the model learn a set of position information, sensor data, and an embedding vector as correct data.

The generation unit 133 also performs machine learning on pairs of position information and sensor data using a neural network-based machine learning method or the like, and generates a prediction model of future data based on past data.

Here, the generation unit 133 generates a vector conversion model by machine learning using RNN (Recurrent Neural Network) and LSTM (Long short-term memory). As for the model, a model of any type and any format can be adopted. For example, the generation unit 133 may adopt SVM (Support Vector Machine) or DNN (Deep Neural Network) as a model. Here, DNN may be CNN (Convolutional Neural Network) or RNN. The model may also be a model realized by combining a plurality of models, such as a model combining CNN and RNN.

Note that the RNN and LSTM may be neural networks based on an attention mechanism. Also, the generation unit 133 may use a similar natural language processing model.

Learning is, for example, deep learning using DNN. Data mining and other machine learning algorithms may also be used. The generation unit 133 learns the model using the various learning methods described above.

Also, the generation unit 133 may share a vector conversion model or a prediction model with the learning unit 37 of the terminal device 10 .

(Providing unit 134)
The providing unit 134 provides the terminal device 10 of the user U with the generated vector conversion model, predetermined conversion rules, and the like. That is, the providing unit 134 transmits the generated vector conversion model, predetermined conversion rules, and the like to the terminal device 10 of the user U via the communication unit 110 .

[5. Processing procedure]
Next, processing procedures by the terminal device 10 and the information providing device 100 according to the embodiment will be described with reference to FIG. FIG. 8 is a sequence diagram illustrating a processing procedure according to the embodiment; The processing procedure described below is repeatedly executed by the control unit 30 of the terminal device 10 or the control unit 130 of the information providing device 100 .

As shown in FIG. 8 , the generation unit 133 of the information providing apparatus 100 generates a position based on an existing position history (or assumed position information) in which the position information of the terminal device 10 collected in the past is accumulated or sensor data. A vector conversion model for converting information into an embedding vector, predetermined conversion rules, etc. are generated (step S101). For example, the generation unit 133 generates a vector conversion model that outputs an embedding vector when position information is input by making the model learn a combination of the position information and the embedding vector as correct data.

Then, the providing unit 134 of the information providing device 100 provides the generated vector conversion model, predetermined conversion rules, etc. to the terminal device 10 of the user U (step S102). That is, the providing unit 134 transmits the generated vector conversion model, predetermined conversion rules, and the like to the terminal device 10 of the user U via the communication unit 110 .

Then, the acquiring unit 34 of the terminal device 10 acquires the vector conversion model, the predetermined conversion rule, etc. provided from the information providing device 100 (step S103).

Then, the acquiring unit 34 of the terminal device 10 acquires the location information of the terminal device 10 constantly/periodically/at a predetermined timing (step S104).

For example, the acquisition unit 34 acquires position information of the terminal device 10 positioned by the positioning unit 14 . Also, the acquisition unit 34 estimates and acquires the indoor position information of the terminal device 10 based on the sensor data detected by the sensors 21 to 28 .

In addition to the position information, the acquisition unit 34 acquires sensor data such as environmental sound, acceleration, and atmospheric pressure using the sensors 21 to 28 mounted on or connected to itself (step S105).

The acquisition unit 34 accumulates the acquired position information as a position history (for example, location history, action history, etc.). In addition, the acquisition unit 34 accumulates the acquired various sensor data for each type. Note that the acquisition unit 34 may store the location information and the sensor data in association with each acquired time or time period.

Then, the conversion unit 35 of the terminal device 10 generates a set of position information and sensor data at the same time or in the same time period (step S106).

Then, the conversion unit 35 generates sequence data in which sets of position information and sensor data are arranged in time series (step S107).

At this time, the prediction unit 36 of the terminal device 10 may predict future data based on the past data of the combination of the position information and the sensor data. For example, even if a situation occurs in which sensor data cannot be acquired at a certain time, the prediction unit 36 inputs a set of position information and sensor data up to that time into the prediction model, and outputs the position information and the sensor data. Based on the set with the data, the sensor data that should have been acquired at that time is inferred. In this case, the conversion unit 35 generates sequence data in which pairs of predicted future data position information and sensor data are arranged in time series.

At this time, the learning unit 37 of the terminal device 10 performs machine learning on pairs of position information and sensor data using a machine learning technique using a neural network, etc., and reconstructs a prediction model for future data based on past data. You may

Then, the conversion unit 35 generates an embedding vector based on the sequence data (step S108).

For example, the conversion unit 35 inputs sequence data to a vector conversion model constructed through machine learning using, for example, RNN (Recurrent Neural Network) or LSTM (Long short-term memory), and obtains an embedding vector as an output. do. Note that the RNN and LSTM may be neural networks based on an attention mechanism. Also, the conversion unit 35 may use a similar natural language processing model.

At this time, the learning unit 37 of the terminal device 10 performs machine learning on the set of the position information and the sensor data using a machine learning method using a neural network, etc., and converts the set of the position information and the sensor data into an embedding vector. A vector transformation model for transformation may be reconstructed.

Alternatively, the conversion unit 35 may convert the sequence data into the embedding vector according to a predetermined conversion rule (on a rule basis).

However, sequence data is only an example. In practice, for example, raw data such as position information and sensor data may be input to the model in chronological order as they are.

Then, the transmission unit 31 of the terminal device 10 transmits the embedding vector converted by the conversion unit 35 to the information providing device 100 via the communication unit 11 (step S109).

Then, the collection unit 131 of the information providing device 100 collects the embedding vectors based on the sets of the position information and the sensor data from the individual terminal devices 10 (step S110). That is, the collection unit 131 receives the embedding vector based on the combination of the position information and the sensor data from the terminal device 10 via the communication unit 110 .

Then, the determining unit 132 of the information providing apparatus 100 analyzes the embedding vector based on the set of the collected positional information and sensor data, and determines the behavior (location, behavior, movement, etc.) of the user U (step S111). .

At this time, the determination unit 132 arranges the embedding vector in a specific space, and determines the similarity and change in behavior of the user U himself and/or the similarity and difference in behavior between the user U and other users. good too.

Further, the generation unit 133 of the information providing device 100 may update the vector conversion model, the predetermined conversion rule, etc. based on the embedding vector based on the combination of the position information and the sensor data received from the terminal device 10 .

[6. Modification]
The terminal device 10 and the information providing device 100 described above may be implemented in various different forms other than the above embodiments. So, below, the modification of embodiment is demonstrated.

In the above embodiment, the terminal device 10 generates an embedding vector based on a set of position information and sensor data. At this time, the embedding vector based on position information and the embedding vector based on sensor data are separately generated. may be generated and combined to generate one embedding vector. For example, the terminal device 10 arranges an embedding vector based on positional information in the 1st to 10th elements of a multidimensional embedding vector of 20 or more elements, and places sensor data in the 11th to 20th elements. You may place an embedding vector based on In this case, the embedding vector based on the positional information indicates "user's behavior and location", etc., and the embedding vector based on sensor data indicates "user's environment and situation".

At this time, as vector conversion models, two types of models may be constructed: a model for converting position information into vectors and a model for converting sensor data into vectors. Alternatively, one vector conversion model may individually convert the input positional information and sensor data into vectors, and combine these two types of vectors.

In the above-described embodiment, the information providing apparatus 100 determines whether the user U "went to the mountains", "went to the sea", or "went to the countryside" based on the vector value of the embedding vector based on the set of position information and sensor data. Actions such as "went" and "went abroad" may be determined. Also, actions such as "went on a trip" or "went on a business trip" by the user U may be determined. Also, actions such as "day trip" or "stayed" by the user U may be determined.

In addition, in the above-described embodiment, when the terminal device 10 does not want to inform the information providing device 100 (server side) of detailed information about the behavior of the user U, the terminal device 10 stores location information or before embedding location information. The information may be fractioned (rounded) to round the position information and convert the position information into coarse position information. For example, when the position information is latitude and longitude information of GPS (Global Positioning System) coordinates, the terminal device 10 rounds up the value after the decimal point of the decimal angle of the latitude and longitude (the minute or less in the degree-minute-second system). Alternatively, truncation (rounding off is also possible) is performed, and the position information is converted into rough values. As a result, it is possible to limit the content of the information about the behavior of the user U that is transmitted from the terminal device 10 to the information providing device 100 (server side). In addition, it is possible to reduce the amount of positional information as learning data for machine learning, thereby reducing the processing load.

Also, in the above embodiment, the values of the respective elements of the embedding vector are represented by "0" and "1" for the sake of convenience. , may be a number having an integer part and a fractional part. That is, any real number may be used. As a result, each element of the multi-dimensional embedding vector can express multiple pieces of information and detailed information.

Further, in the above embodiment, the terminal device 10 accumulates the acquired position information and sensor data, but it may be accumulated in a vectorized state. For example, based on the location indicated by each piece of position information, the terminal device 10 sets the combination of position information and sensor data to "at home", "at work", "at store", "outdoors", and "at home". , etc., and stored. Then, when the same vector values are continuous along the time series, they may be collectively used as an embedding vector.

Further, in the above embodiment, the terminal device 10 or the information providing device 100 individually adjusts the vector conversion model, the prediction model, or the predetermined conversion rule (rule) according to the characteristics and circumstances of each user U. may For example, a vector conversion model, a prediction model, or a predetermined conversion rule may be customized.

In the above embodiment, a vector conversion model for converting a combination of position information and sensor data into an embedding vector using federated learning or the like, or a prediction model for future data based on past data. etc. may be generated. For example, in federated learning, each terminal device 10 performs machine learning using pairs of position information and sensor data as learning data for a vector conversion model, a prediction model, etc., and the original model and the learned model. The difference data from the model is transmitted to the information providing device 100 . The information providing device 100 receives difference data from each terminal device 10, integrates and learns them, and generates a common model for each model. Then, the information providing device 100 provides each terminal device 10 with parameters of the generated common model and the like. In this way, by using federated learning, it is possible to perform learning without providing a set of terminal location information and sensor data, which is learning data, to the server side. Privacy can be considered.

[7. effect〕
As described above, the information processing device (terminal device 10) according to the present application includes a first acquisition unit (acquisition unit 34) that acquires position information, a second acquisition unit (acquisition unit 34) that acquires sensor data from sensors, ), a conversion unit 35 that converts position information and sensor data into vectors, and a transmission unit that transmits the vectors. At this time, the conversion unit 35 collectively converts the plurality of pieces of position information and sensor data into a single vector. The conversion unit 35 converts the position information and the sensor data into vectors using a vector conversion model obtained by machine learning or according to predetermined conversion rules.

Thereby, the information processing device can embed and transmit the position information and the sensor data.

[8. Hardware configuration]
Also, the terminal device 10 and the information providing device 100 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 9, for example. The information providing apparatus 100 will be described below as an example. FIG. 9 is a diagram illustrating an example of a hardware configuration; The computer 1000 is connected to an output device 1010 and an input device 1020, and an arithmetic device 1030, a primary storage device 1040, a secondary storage device 1050, an output I/F (Interface) 1060, an input I/F 1070, and a network I/F 1080 are buses. It has a form connected by 1090.

The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic unit 1030 is implemented by, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or the like.

The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used for various calculations by the arithmetic device 1030 . The secondary storage device 1050 is a storage device in which data used for various calculations by the arithmetic device 1030 and various databases are registered. State Drive), flash memory, or the like. The secondary storage device 1050 may be an internal storage or an external storage. Also, the secondary storage device 1050 may be a removable storage medium such as a USB memory or an SD (Secure Digital) memory card. Also, the secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

The output I/F 1060 is an interface for transmitting information to be output to the output device 1010 that outputs various information such as a display, a projector, and a printer. (Digital Visual Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and other standardized connectors. Also, the input I/F 1070 is an interface for receiving information from various input devices 1020 such as a mouse, keyboard, keypad, buttons, scanner, etc., and is realized by, for example, USB.

Also, the output I/F 1060 and the input I/F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

Also, the output device 1010 and the input device 1020 may be integrated like a touch panel. In this case, the output I/F 1060 and the input I/F 1070 may also be integrated as an input/output I/F.

Note that the input device 1020 includes, for example, optical recording media such as CDs (Compact Discs), DVDs (Digital Versatile Discs), PDs (Phase change rewritable discs), magneto-optical recording media such as MOs (Magneto-Optical discs), and tapes. It may be a device that reads information from a medium, a magnetic recording medium, a semiconductor memory, or the like.

Network I/F 1080 receives data from other devices via network N and sends the data to arithmetic device 1030, and also transmits data generated by arithmetic device 1030 via network N to other devices.

Arithmetic device 1030 controls output device 1010 and input device 1020 via output I/F 1060 and input I/F 1070 . For example, arithmetic device 1030 loads a program from input device 1020 or secondary storage device 1050 onto primary storage device 1040 and executes the loaded program.

For example, when the computer 1000 functions as the information providing device 100 , the arithmetic device 1030 of the computer 1000 implements the functions of the control unit 130 by executing a program loaded on the primary storage device 1040 . Further, arithmetic device 1030 of computer 1000 may load a program acquired from another device via network I/F 1080 onto primary storage device 1040 and execute the loaded program. Further, the arithmetic unit 1030 of the computer 1000 may cooperate with another device via the network I/F 1080, and call functions, data, etc. of the program from another program of the other device for use.

[9. others〕
Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. In addition, the components described above include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within the so-called equivalent range. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, replacements, or modifications of components can be made without departing from the gist of the above-described embodiments.

Further, among the processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, the information providing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing content.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the acquisition unit can be read as acquisition means or an acquisition circuit.

1 information processing system 10 terminal device 14 positioning unit 30 control unit 31 transmission unit 32 reception unit 33 processing unit 34 acquisition unit 35 conversion unit 36 prediction unit 37 learning unit 100 information providing device 110 communication unit 120 storage unit 121 conversion information database 122 vector Information database 130 control unit 131 collection unit 132 determination unit 133 generation unit 134 provision unit

Claims (20)

a first acquisition unit that acquires position information;
a second acquisition unit that acquires sensor data from the sensor;
a conversion unit that converts the position information and the sensor data into a vector;
a transmitter that transmits the vector;
with
The conversion unit collectively converts the plurality of position information and the sensor data into the single vector.
An information processing device characterized by: The conversion unit converts sequence data obtained by arranging the plurality of position information and the sensor data in time series into the multidimensional vector.
The information processing apparatus according to claim 1, characterized by: a first acquisition unit that acquires position information;
a second acquisition unit that acquires sensor data from the sensor;
a conversion unit that converts the position information and the sensor data into a vector;
a transmitter that transmits the vector;
with
The conversion unit converts sequence data obtained by arranging the plurality of position information and the sensor data in time series into the multidimensional vector.
An information processing device characterized by: a prediction unit that predicts the future location information and the sensor data based on the location information and the sensor data;
4. The information processing apparatus according to any one of claims 1 to 3, further comprising: a first acquisition unit that acquires position information;
a second acquisition unit that acquires sensor data from the sensor;
a prediction unit that predicts the future location information and the sensor data based on the location information and the sensor data;
a conversion unit that converts the position information and the sensor data into a vector;
a transmitter that transmits the vector;
An information processing device comprising: The prediction unit uses a prediction model obtained by machine learning to predict the future position information and the sensor data.
6. The information processing apparatus according to claim 4, wherein: The conversion unit converts a part of the position information and the sensor data out of the plurality of position information into a vector,
The transmitting unit transmits the vector converted from part of the position information and the sensor data, and the remaining position information.
7. The information processing apparatus according to any one of claims 1 to 6, characterized by: a first acquisition unit that acquires position information;
a second acquisition unit that acquires sensor data from the sensor;
a conversion unit that converts the position information and the sensor data into a vector;
a transmitter that transmits the vector;
with
The conversion unit converts a part of the position information and the sensor data out of the plurality of position information into a vector,
The transmitting unit transmits the vector converted from part of the position information and the sensor data and the remaining position information.
An information processing device characterized by: The sensor data includes at least one of environmental sound, acceleration, and atmospheric pressure.
The information processing apparatus according to any one of claims 1 to 8 , characterized in that: The conversion unit converts the position information and the sensor data into the vector using a vector conversion model obtained by machine learning.
The information processing apparatus according to any one of claims 1 to 9 , characterized in that: The conversion unit converts the position information and the sensor data into the vector according to a predetermined conversion rule.
The information processing apparatus according to any one of claims 1 to 10 , characterized by: the vector is indicative of user behavior;
The information processing apparatus according to any one of claims 1 to 11 , characterized by: An information processing method executed by an information processing device,
a first acquisition step of acquiring location information;
a second acquiring step of acquiring sensor data from the sensor;
a conversion step of converting the position information and the sensor data into a vector;
a transmitting step of transmitting the vector;
including
In the converting step, a plurality of the position information and the sensor data are collectively converted into a single vector;
An information processing method characterized by: a first acquisition procedure for acquiring location information;
a second acquisition procedure for acquiring sensor data from the sensor;
a conversion procedure for converting the position information and the sensor data into a vector;
a transmission procedure for transmitting the vector;
on the computer, and
In the conversion step, a plurality of the position information and the sensor data are collectively converted into a single vector.
An information processing program characterized by: An information processing method executed by an information processing device,
a first acquisition step of acquiring location information;
a second acquiring step of acquiring sensor data from the sensor;
a conversion step of converting the position information and the sensor data into a vector;
a transmitting step of transmitting the vector;
including
In the conversion step, sequence data obtained by arranging the plurality of position information and the sensor data in time series is converted into the multidimensional vector.
An information processing method characterized by: a first acquisition procedure for acquiring location information;
a second acquisition procedure for acquiring sensor data from the sensor;
a conversion procedure for converting the position information and the sensor data into a vector;
a transmission procedure for transmitting the vector;
on the computer, and
In the conversion step, sequence data obtained by arranging the plurality of position information and the sensor data in time series is converted into the multidimensional vector.
An information processing program characterized by: An information processing method executed by an information processing device,
a first acquisition step of acquiring location information;
a second acquiring step of acquiring sensor data from the sensor;
a prediction step of predicting the future location information and the sensor data based on the location information and the sensor data;
a conversion step of converting the position information and the sensor data into a vector;
a transmitting step of transmitting the vector;
An information processing method comprising: a first acquisition procedure for acquiring location information;
a second acquisition procedure for acquiring sensor data from the sensor;
a prediction procedure for predicting the future location information and the sensor data based on the location information and the sensor data;
a conversion procedure for converting the position information and the sensor data into a vector;
a transmission procedure for transmitting the vector;
An information processing program characterized by causing a computer to execute An information processing method executed by an information processing device,
a first acquisition step of acquiring location information;
a second acquiring step of acquiring sensor data from the sensor;
a conversion step of converting the position information and the sensor data into a vector;
a transmitting step of transmitting the vector;
including
In the converting step, part of the position information and the sensor data among the plurality of position information are converted into a vector;
In the transmitting step, the vector converted from the part of the position information and the sensor data and the remaining position information are transmitted.
An information processing method characterized by: a first acquisition procedure for acquiring location information;
a second acquisition procedure for acquiring sensor data from the sensor;
a conversion procedure for converting the position information and the sensor data into a vector;
a transmission procedure for transmitting the vector;
on the computer, and
In the conversion step, part of the position information and the sensor data among the plurality of position information are converted into a vector;
In the transmission step, the vector converted from the part of the position information and the sensor data and the remaining position information are transmitted.
An information processing program characterized by:

Images