JP7105176B2 - Machine learning device, judgment device, program and machine learning method - Google Patents

Description

The present invention relates to a machine learning device, a determination device, a program, and a machine learning method.

Techniques for detecting the visit of a user of a wireless communication terminal by analyzing a log relating to wireless signals received by a wireless LAN access point are known (see Patent Documents 1 to 3, for example).
[Prior art documents]
[Patent Literature]
[Patent Document 1] Patent No. 5859161 [Patent Document 2] Patent No. 5802716 [Patent Document 3] Patent No. 6169124

According to the conventional technology, it is determined whether or not the user of the communication terminal that transmitted the wireless signal has visited the store by using the radio wave intensity of the wireless signal. Although it is possible to detect store visits with relatively high accuracy using the determination method described above, in recent years, there has been a demand for further improvement in store visit detection accuracy.

In a first aspect of the invention, a machine learning device is provided. The machine learning device receives, for example, a data point containing information about a radio signal received by a radio receiver disposed within an area having a predetermined geographical range, and the radio that emitted the radio signal. A model generation unit is provided for generating a learning model used to determine whether the user of the terminal was within the area. The machine learning device described above includes, for example, an adjustment unit that adjusts a reference value for determining whether or not the user exists inside the area in the learning model generated by the model generation unit. In the above machine learning device, for example, the model generation unit (a) uses an initial value according to the learning model as the reference value, and (b) (i) a plurality of radio signals received by one or more radio receiving devices. and (ii) whether the user of the wireless terminal that emitted each of the plurality of wireless signals was within the area in which the wireless receiving device that received each of the plurality of wireless signals was arranged. A learning model is generated by executing machine learning using a learning data set containing a label indicating whether or not. In the above-described machine learning device, for example, the adjustment unit increases the true positive rate of the determination result of the learning model generated by the model generation unit compared to the case where the initial value is used as the reference value. value is determined as the adjusted reference value.

In each of the above machine learning devices, (i) terminal identification information for identifying the wireless terminal that transmitted each of the plurality of wireless signals, (ii) reception indicating the time at which each of the plurality of wireless signals was received A record acquisition unit that acquires a plurality of records associated with time information and (iii) signal information about radio waves of each of the plurality of radio signals may be provided. Each of the above machine learning devices includes (i) user identification information for identifying each of the one or more users involved in a specific action, and (ii) action time indicating the time at which the specific action was performed. An action history acquisition unit that acquires a plurality of pieces of action history information associated with information may be provided. The above machine learning device includes (i) terminal identification information and reception time information included in each of the plurality of records acquired by the record acquisition unit, and (ii) a plurality of action history information acquired by the action history acquisition unit. A plurality of records and a plurality of action history information are collated based on the user identification information and the action time information included in each, and terminal identification information included in each of at least some of the plurality of records and at least one a labeling unit that generates a learning data set by associating information about radio waves indicated by signal information included in each of the records of the area with a label indicating that the user was present inside the area; you can In the machine learning device described above, the user identification information included in at least part of the plurality of action history information may be terminal identification information of a wireless terminal used by the user identified by the user identification information.

The machine learning device obtains correspondence information indicating a correspondence relationship between each of the one or more pieces of user identification information and terminal identification information of a wireless terminal used by a user identified by the user identification information. You may have a department. In the machine learning device described above, the labeling unit refers to the correspondence information acquired by the correspondence acquisition unit, and performs (i) terminal identification information and reception time included in each of the plurality of records acquired by the record acquisition unit. and (ii) the user identification information and the action time information included in each of the plurality of action history information acquired by the action history acquisition unit. You may generate a dataset.

Each of the above machine learning devices includes (i) terminal identification information for identifying the wireless terminal that transmitted each of the plurality of wireless signals, (ii) reception indicating the time at which each of the plurality of wireless signals was received Acquiring a plurality of records associated with time information, (iii) reception position information indicating the position at which each of the plurality of wireless signals was received, and (iv) signal information regarding radio waves of each of the plurality of wireless signals. A record acquisition unit may be provided. In each of the above machine learning devices, (i) area identification information for identifying each of the one or more areas, (ii) time period during which entry to the area identified by the area identification information is restricted a restriction information acquisition unit that acquires one or more pieces of entry restriction information shown in FIG. The above machine learning device includes (i) receiving position information included in each of the plurality of records acquired by the record acquisition unit, and (ii) included in each of the one or more entry restriction information acquired by the restriction information acquisition unit. A plurality of records and one or more entry restriction information are collated based on the area identification information stored, and the terminal identification information contained in each of at least some of the plurality of records and each of at least some of the records A labeling unit may be provided that generates a learning data set by associating information about radio waves indicated by the included signal information with a label indicating that the user was not inside the area.

The above-described machine learning device obtains a plurality of records acquired by the record acquisition unit, based on the terminal identification information and the reception time information included in each of the plurality of records, for each terminal identification information and a predetermined condition. An aggregator may be provided for aggregating for each satisfying period. In the machine learning device described above, each aggregating unit includes (i) terminal identification information for identifying a wireless terminal that has transmitted each of the plurality of wireless signals, and (ii) the target of aggregation processing by the aggregating unit. Aggregation period information indicating a period, (iii) One or more features related to one or more radio signals transmitted during the period indicated by the aggregation period information by a radio terminal identified by the terminal identification information among the plurality of radio signals One or more aggregated records may be generated to which feature information indicating quantity is associated. In the machine learning device described above, the labeling unit may generate the learning data set by labeling at least part of the one or more consolidated records.

In the above machine learning device, the learning model may calculate the value of a function used for estimating the uncertainty of prediction or judgment of the learning model for each data point input to the learning model. . In the machine learning device described above, the learning model may determine whether or not the user is present inside the area based on the calculated function value and the reference value.

In the machine learning device described above, the adjustment unit may calculate data representing a receiver operating characteristic curve (ROC curve) of the learning model generated by the model generation unit. In the machine learning device described above, the adjustment unit compares a point on the ROC curve and an initial value as a reference value to a true positive determination result of the learning model generated by the model generation unit. A reference value corresponding to the point at which the rate increases may be determined as the second value.

In the machine learning device described above, the model generator may generate a learning model for each wireless receiving device or for each area. In the machine learning device described above, the wireless receiving device may be a relay device that relays wireless communication.

In a second aspect of the invention, a determination device is provided. The determination device described above includes, for example, the machine learning device described above. The determination device described above includes, for example, a determination target acquisition unit that acquires determination target data including information about each of the plurality of radio signals received by at least one of the one or more radio reception devices. For example, the determination device (i) inputs the determination target data acquired by the determination target acquisition unit into the learning model generated by the model generation unit, and (ii) uses the second value determined by the adjustment unit as the reference value. Using the value, whether each user of one or more wireless terminals that transmitted wireless signals related to the determination target data existed within an area in which the wireless receiving device that received the wireless signals related to the determination target data was arranged A judgment unit for judging whether or not is provided.

In a third aspect of the invention, a machine learning method is provided. The above machine learning method, for example, is input with data points containing information about radio signals received by radio receivers located within an area having a predetermined geographic range, and the radio that originated the radio signal. There is a model generation stage for generating a learning model that is used to determine if the user of the terminal was within the area. The above machine learning method, for example, has an adjustment stage in which the learning model generated in the model generation stage adjusts the reference value for determining whether or not the user was inside the area. In the above machine learning method, for example, the model generation step includes (a) using an initial value according to the learning model as a reference value, and (b) (i) a plurality of radio signals received by one or more radio receiving devices. and (ii) whether the user of the wireless terminal that emitted each of the plurality of wireless signals was within the area in which the wireless receiving device that received each of the plurality of wireless signals was arranged. generating a learning model by performing machine learning using a training data set that includes a label indicating whether or not; In the above-described machine learning method, for example, the adjustment stage increases the true positive rate of the judgment result of the learning model generated in the model generation stage compared to the case where the initial value is used as the reference value. determining the value as the adjusted reference value.

In a fourth aspect of the invention, a program is provided. A non-transitory computer-readable medium storing the above program may be provided. The program is, for example, a program for causing a computer to function as the machine learning device according to the first aspect. The program is, for example, a program for causing a computer to function as the determination device according to the second aspect. The program may be a program for causing a computer to execute the machine learning method according to the third aspect.

It should be noted that the above summary of the invention does not list all the necessary features of the invention. Subcombinations of these feature groups can also be inventions.

An example of the system configuration of the data analysis system 100 is shown schematically. An example of the internal configuration of the management server 160 is shown schematically. An example of the internal configuration of the storage unit 242 is shown schematically. An example of information processing in the input data generator 244 is shown schematically. An example of the internal configuration of the data set generator 252 is shown schematically. An example of the internal configuration of the learning unit 254 is shown schematically. An example of the internal configuration of the threshold adjuster 256 is schematically shown. An example of information processing in the threshold adjustment unit 256 is schematically shown. An example of a data table 900 is shown schematically. An example of a data table 1000 is shown schematically. An example of a data table 1100 is shown schematically. An example of a data table 1200 is shown schematically. An example of a data table 1300 is shown schematically. An example of a data table 1400 is shown schematically. An example of a data table 1500 is shown schematically. An example of a data table 1600 is shown schematically. An example of a data table 1700 is shown schematically. An example of information processing in the analysis unit 240 is shown schematically. An example of information processing in the analysis unit 240 is shown schematically. An example of a data table 1700 is shown schematically. An example of the system configuration of the computer 3000 is shown schematically.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution of the invention. In addition, in the drawings, the same or similar parts may be denoted by the same reference numerals to omit redundant description.

In each embodiment, each user preferably agrees to receive services using personal information.

[Overview of data analysis system 100]
FIG. 1 schematically shows an example of a system configuration of a data analysis system 100. As shown in FIG. In this embodiment, the data analysis system 100 has a management server 160 . In the present embodiment, for the purpose of facilitating understanding of the data analysis system 100, the management server 160 responds to a request from the communication terminal 108 by determining the number of customers who visited the store 12 or 14 (the number of customers who visited the store 14). The data analysis system 100 will be described in detail by taking as an example a case of estimating , and transmitting information indicating the above estimation result as a response to the above request.

In this embodiment, store 12 has a predetermined geographical range. Inside the store 12, an access point 122 and a visit confirmation device 142 are arranged. In this embodiment, store 14 has a predetermined geographic area. Inside the store 14, an access point 124 and a visit confirmation device 144 are arranged. At least one of access point 122 , access point 124 , visit confirmation device 142 and visit confirmation device 144 may form part of data analysis system 100 .

In FIG. 1 , a user 22 of a communication terminal 102 having a wireless communication function exists inside a store 12 . In this embodiment, communication terminal 102 transmits and receives information to and from access point 122 using radio signals. For example, communication terminal 102 emits a radio signal and access point 122 receives the radio signal. Further, in this embodiment, the user 22 uses various services provided by the store visit confirmation device 142 . At this time, the store visit confirmation device 142 acquires the identification information assigned to the user 22 by the provider or operator of the above service, the identification information of the communication terminal 102, and the like. Thus, the visit confirmation device 142 detects the visit of the user 22 .

In FIG. 1, a user 24 of a communication terminal 104 having a wireless communication function exists inside a store 14 . In this embodiment, communication terminal 104 uses radio signals to transmit and receive information to and from access point 124 . For example, communication terminal 104 emits a radio signal and access point 124 receives the radio signal. Further, in this embodiment, the user 24 uses various services provided by the store visit confirmation device 144 . As a result, the store visit confirmation device 144 detects the visit of the user 24 by the same procedure as when the store visit confirmation device 142 detects the visit of the user 22 .

In the present embodiment, details of the data analysis system 100 will be described by taking as an example a case where the number of access points arranged inside each of the stores 12 and 14 is one. However, the data analysis system 100 is not limited to this embodiment. In other embodiments, multiple access points may be located within at least one of store 12 and store 14 .

[Collect log information]
According to the data analysis system 100 according to the present embodiment, the management server 160 collects, from the access point 122, information regarding one or more wireless signals received by the access point 122 from communication terminals. Management server 160 also collects information from access point 124 regarding each of the one or more wireless signals received by access point 124 .

Information on wireless signals (sometimes referred to as log information) includes information for identifying the access point that received the wireless signal, and the time at which the wireless signal was received (sometimes referred to as reception time). (sometimes referred to as reception time information), information for identifying the communication terminal that transmitted the radio signal (sometimes referred to as terminal identification information), the radio It may include information on the radio wave of the signal (sometimes referred to as signal information). The signal information includes, for example, at least one of information indicating the radio wave intensity of the radio signal, information indicating the frequency band of the radio signal, and information indicating the connection stage of communication.

A location registration request stage, an authentication stage, an association stage, a connection completion stage, and the like are exemplified as connection stages of communication between a communication terminal and an access point in a mobile communication system. The connection stages of communication between a communication terminal and an access point in a wireless communication system other than the mobile communication system include a probe request stage, an authentication stage, an association stage, a connection completion stage, a disassociation stage, and a detach stage (referred to as a detachment stage). may be used.) and the like are exemplified.

Details of the log information will be described later, but a specific example of the log information is shown in FIG. 12, for example. In FIG. 12, data table 1200 includes APID 1222 of the access point that received the wireless signal, information 1224 indicating the reception time of the wireless signal, terminal ID 1226 of the communication terminal that transmitted the wireless signal, and radio wave of the wireless signal. information 1228 about is associated with and stored. Each row of the data table 1200 represents an example of log information.

The APID 1222 is identification information assigned to each access point by the management server 160, and may be an example of identification information of each access point. Examples of the information 1228 on the radio wave of the radio signal include information indicating radio wave intensity, information indicating the frequency band, information indicating the connection stage, and the like.

[Collection of visit confirmation information]
In this embodiment, the management server 160 collects information about customers who have been confirmed to visit the store 12 from the store visit confirmation device 142 . The management server 160 collects information about customers who are confirmed to have visited the store 14 from the store visit confirmation device 144 . Information about a customer whose visit has been confirmed (sometimes referred to as visit confirmation information) includes information for identifying the customer (sometimes referred to as a user ID), It may include information indicating the time (sometimes referred to as visit time), information indicating the store visited by the customer, information indicating means for confirming the visit of the customer, and the like.

The information for identifying the customer may be identification information assigned to each customer by any service provider or operator, or may be the terminal ID of the communication terminal used by each customer. . The information indicating the visit time may be information indicating the date when the customer visits the store, or may be information indicating the date and time when the customer visits the store.

Although the details of the store-visit confirmation information will be described later, a specific example of the store-visit confirmation information is shown in FIG. 13, for example. In FIG. 13, the data table 1300 stores the user ID 1322 of the user, the position information 1324 of the location where the behavior of the user was confirmed, and the information 1326 indicating the time when the behavior of the user was confirmed. . Each row of the data table 1300 represents an example of store visit confirmation information.

As the user ID, for example, (i) the credit card number acquired by the store visit confirmation device 142 or the store visit confirmation device 144, the membership number or account of the membership service, the MAC address of the communication terminal, (ii) the store visit confirmation device 142 Alternatively, the name, credit card number, membership service member number or account entered in the store visit confirmation device 144 is used.

[Generate training data set]
According to this embodiment, the management server 160 analyzes the collected store visit confirmation information and log information, and generates a learning data set that can be used as teacher data for the learning model used in the determination process described above. . For example, the management server 160 first collates the collected visit confirmation information and log information, and associates at least part of one or more pieces of visit confirmation information with at least part of one or more pieces of log information.

For example, the management server 160 includes (i) information for identifying a customer and information indicating the time when the customer comes to the store, which are included in each store visit confirmation information, and (ii) terminal identification information, which is included in each log information. The store visit confirmation information and the log information are collated using the information and the reception time information as keys. If it is possible to obtain information indicating the correspondence between the information for identifying the customer and the terminal identification information (sometimes referred to as correspondence information), the management server 160 can obtain such information. The visit confirmation information and the log information may be collated by referring to them.

Although the details of the correspondence information will be described later, a specific example of the correspondence information is shown in FIG. 11, for example. In FIG. 11, data table 1100 stores terminal ID 1122 of each communication terminal and user ID 1124 of the user of each communication terminal in association with each other. Each row of the data table 1100 represents an example of correspondence information.

The details of the procedure for the management server 160 to collate the store visit confirmation information and the log information will be described later. The management server 160 first reads log information recorded in a specific line from the data table 1300 . Next, the management server 160 (i) refers to the data table 1100 using the terminal ID 1226 included in the log information as a key, and (ii) extracts the user ID 1124 associated with the terminal ID 1122 that matches the terminal ID 1226. . Next, the management server 160 refers to the data table 1300 using (i) the extracted user ID 1124 and the information 1224 indicating the reception time of the radio signal included in the log information as keys, and (ii) the user ID 1322 is Store-visit confirmation information that matches the user ID 1124 and whose information 1326 indicating the time when the user's behavior was confirmed matches the information 1224 indicating the reception time is extracted. The management server 160 determines that the extracted store visit confirmation information can be associated with the read log information.

Next, the management server 160 stores in the log information associated with the store visit confirmation information the store or the like where the access point where the user of the communication terminal indicated by the log information received the radio signal associated with the log information is located. A label (sometimes referred to as a positive label) indicating that it existed inside the As a result, a positive-labeled learning data set is generated.

The details of the procedure for the management server 160 to generate the learning data set will be described later, but the management server 160 generates the learning data set according to the following procedure, for example. Management server 160 first refers to a database in which the APID of each access point is associated with the store ID of the store where each access point is located, and accesses the access points that have received radio signals related to the above log information. Acquire the identification information of the store or the like where points are distributed.

A specific example of the above database is shown in FIG. 9, for example. In FIG. 9, data table 900 stores APID 922 of each access point, information 924 indicating the MAC address of each access point, and store ID 926 of the store where each access point is arranged, in association with each other.

For example, the management server 160 (i) refers to the data table 900 using the APID 1222 included in the log information as a key, and (ii) extracts the shop ID 926 associated with the APID 922 matching the APID 1222 . The management server 160, for example, (i) the terminal ID 1226 included in the log information, the information 1224 indicating the reception time of the wireless signal, and the information 1228 regarding the radio wave of the wireless signal, (ii) the extracted shop ID 926, ( iii) generating data points (sometimes referred to as training data points) forming part of the training data set by associating them with positive labels; The management server 160 similarly processes each of the pieces of log information associated with the store visit confirmation information, thereby generating a learning data set including a plurality of learning data points.

Details of the learning data set will be described later, and a specific example of the learning data set is shown in FIG. 15, for example. In FIG. 15, each row of data table 1500 represents an example of a training data point. 15, data table 1500 includes terminal ID 1422 of the communication terminal that transmitted the wireless signal, store ID 1424 of the store where the access point that received the wireless signal is located, information 1426 indicating the date of reception of the wireless signal, and wireless communication terminal ID 1424. Information 1428 indicating the feature amount of the signal and information 1532 indicating the type of label are associated and stored.

In the data table 1500, the terminal ID 1422 records, for example, the terminal ID 1226 included in the log information. For the store ID 1424, for example, the store ID 926 extracted above is recorded. The information 1426 indicating the reception date of the radio signal records, for example, the time indicated by the information 1224 indicating the reception time of the radio signal included in the log information, or information indicating the date. The information 1428 indicating the characteristic amount of the radio signal records, for example, the information 1228 on the radio wave of the radio signal included in the log information, or a part thereof. Information indicating positive is recorded in the information 1532 indicating the type of label. Note that the information 1428 indicating the characteristic amount of the radio signal may record a statistic obtained by statistically processing the information 1228 regarding radio waves of the radio signal included in the plurality of pieces of log information.

[Building a learning model]
According to this embodiment, the management server 160 generates a learning model using the learning data set. The management server 160 generates a learning model using items indicated by positive labels included in each data point constituting the learning data set as objective variables.

The management server 160 generates a learning model using at least various physical quantities related to radio waves of radio signals indicated by signal information included in each data point constituting the learning data set or statistical values of the physical quantities as feature quantities. good. The management server 160 may generate a learning model using, as a feature quantity, the identification information of the access point that received the wireless signal included in each data point forming the learning data set. The management server 160 may generate a learning model using reception time information included in each data point constituting the learning data set as a feature amount.

In the above learning model, for example, a data point containing information about a radio signal received by an access point placed inside a store is input, and the user of the communication terminal that transmitted the radio signal is It is used to determine whether the The learning model described above may be a classifier that classifies input data points (sometimes referred to as samples) into appropriate classes.

For example, the learning model classifies the input data points into a class related to wireless signals transmitted by communication terminals inside the store, etc., and a class related to wireless signals transmitted by communication terminals not located inside the store, etc. do. A class related to a radio signal transmitted by a communication terminal inside a store or the like may be an example of a class indicating that the user of the communication terminal that transmitted the radio signal was present inside the store or the like. A class related to a radio signal transmitted by a communication terminal that does not exist inside a store, etc. may be an example of a class indicating that the user of the communication terminal that transmitted the radio signal was not present inside the store, etc. .

The learning model may calculate, for each data point input to the learning model, the value of a function used to estimate the uncertainty of the prediction or decision of the learning model. A function used to estimate the uncertainty of prediction or judgment of a learning model is a function for calculating the probability that an input data point belongs to each class (sometimes referred to as classification probability). , a decision function for evaluating the uncertainty of the prediction result or determination result of the learning model, and the like.

In the above learning model, (i) the calculated classification probability or the value of the decision function, and (ii) the above learning model determines whether the user of the communication terminal was inside the store or the like. It is determined whether or not the user exists within the area based on a threshold for classification (sometimes referred to as a threshold for classification). As described above, the learning model has the function of outputting the value of the function used to estimate the uncertainty of the prediction or decision of the learning model. In the learning model, an initial value corresponding to the type of function is set as a threshold for classification.

According to this embodiment, the management server 160 first executes machine learning processing using part of the learning data set. In machine learning processing, model learning and model verification are performed.

As described above, according to the present embodiment, the learning data set is generated based on store visit confirmation information, for example. In this case, the learning data set is data obtained by assigning a positive label to "data related to radio signals transmitted by the communication terminal of the user who entered the store 12 and used the service provided by the store visit confirmation device 142". becomes a set.

Therefore, when machine learning processing is performed using a learning data set generated based on the store visit confirmation information, the input data points are (i) "(a) entered into the store 12, (ii) a class to which "(b) enters the store 12 and is provided by the store visit confirmation device 142; Radio signals transmitted by communication terminals of users who did not use the service" and "(c) Radio signals transmitted by communication terminals of users who did not enter the store 12" belong to this class with high accuracy. A learning model is constructed for

However, in many cases in actual business, a trained learning model classifies input data points into (i) classes related to radio signals emitted by communication terminals existing inside a store, etc., and (ii) stores It is required to classify into a class related to radio signals transmitted by a communication terminal that does not exist inside such as a class. In other words, the learned learning model is based on the following: (i) "(a) the communication terminal of the user who entered the store 12 and used the service provided by the store visit confirmation device 142 transmitted A class to which "(b) a radio signal transmitted by a communication terminal of a user who entered the store 12 and did not use the service provided by the store visit confirmation device 142" belongs; and (ii) "( c) It is required to classify with high accuracy into the class to which "radio signals transmitted by communication terminals of users who have not entered the shop 12" belongs.

When the machine learning process is performed using the learning data set generated based on the store visit confirmation information, the learned learning model has (a) and (b) as positive labels Based on the data set Compared to the case of using another learned model, more "(b) the communication terminal of the user who entered the store 12 and did not use the service provided by the store visit confirmation device 142 "radio signal" may be determined as negative. Therefore, in order to improve the evaluation of the learning model, the learning model should be evaluated as follows: "(b) a wireless communication terminal of a user who entered the store 12 and did not use the service provided by the store visit confirmation device 142; It is preferable to increase the percentage of positives that "signal".

However, the learning model increases the percentage of positive determinations of "(b) a wireless signal transmitted by a communication terminal of a user who entered the store 12 and did not use the service provided by the store visit confirmation device 142". , there is a possibility that the learning model will determine positively "(c) the radio signal transmitted by the communication terminal of the user who did not enter the store 12". Therefore, while suppressing an increase in the percentage of positive determinations of "(c) the radio signal transmitted by the communication terminal of the user who did not enter the store 12", It is preferable to increase the percentage of positive determinations of "radio signals transmitted by communication terminals of users who have entered the store and have not used the service provided by the store visit confirmation device 142".

Therefore, the management server 160 next uses the learning data set that was not used in the machine learning process to adjust the classification threshold. For example, the management server 160 sets the threshold for classification so that the true positive rate of the determination result of the learning model is higher than when the initial value corresponding to the learning model is used as the threshold for classification. adjust. The management server 160 may adjust the threshold for classification so that the false positive rate of the determination result of the learning model is smaller than when the initial value is used as the threshold for classification.

According to this embodiment, in the actual determination process, the management server 160 uses the learned learning model and the adjusted threshold value to create new data points that are not used for learning and verification (determination target data and (i) a class related to wireless signals transmitted by communication terminals existing inside a store, etc., and (ii) a class related to wireless signals transmitted by communication terminals not located inside a store, etc. and According to one embodiment, when the adjusted threshold value is used as the threshold for classification, compared with the case where the initial value corresponding to the learning model is used as the threshold for classification, the judgment result of the learning model is higher. Higher true positive rate. According to another embodiment, when the adjusted threshold value is used as the classification threshold value, compared with the case where the initial value corresponding to the learning model is used as the classification threshold value, the determination result of the learning model is The true positive rate of is increased, and the false positive rate of the determination result of the learning model is decreased.

Stores such as the store 12 and the store 14 may be an example of an area. Data analysis system 100 may be an example of a machine learning device and a determination device. Communication terminals, such as communication terminal 102 and communication terminal 104, may be examples of wireless terminals. Access points, such as access point 122 and access point 124, may be examples of record retrievers, wireless receivers, and relays. Management server 160 may be an example of a machine learning device and a determination device. The management server 160 may be an example of a record acquisition unit, an action history acquisition unit, and a label assignment unit.

Each of the one or more pieces of log information may be an example of a record. Each of the one or more store visit confirmation information may be an example of action history information. Each of the one or more pieces of input data may be an example of an aggregation record. A threshold for classification may be an example of a reference value. The adjusted threshold may be an example of the second value.

According to this embodiment, the management server 160 uses the learned learning model constructed according to the above procedure to perform the actual determination process. As described above, according to the present embodiment, a learning model is constructed using a learning data set generated based on store visit confirmation information. This provides sufficient determination accuracy in many cases. Furthermore, according to this embodiment, the threshold for classification is adjusted prior to the determination process. The learning model then classifies the input data points using the adjusted classification thresholds. This further improves the determination accuracy.

Generally, in supervised machine learning, the problem is how to collect a large amount of teacher data. In particular, when machine learning is used to determine whether or not a communication terminal exists in a specific area based on information related to the radio signal of the communication terminal, it is necessary to collect a large amount of training data in the first place. difficult. Therefore, the inventors, for example, pay attention to the radio signal transmitted by the communication terminal of the customer visiting the store 12 where the access point 122 and the store visit confirmation device 142 are installed, and use the log information related to the radio signal as training data. I thought of it. However, even in this case, if the customer enters the store 12 but does not use the service provided by the store visit confirmation device 142, the customer's data is used as training data. It may not be possible.

In this regard, in the present embodiment, for example, a positive label is a "radio signal emitted by a communication terminal of a customer who enters the store 12 and uses the service provided by the store visit confirmation device 142". After a learning model is constructed using the data set as teacher data, the threshold for classification is adjusted. As a result, (i) a learning data set with a positive label of "a wireless signal transmitted by a communication terminal of a customer who entered the store 12 and used the service provided by the store visit confirmation device 142" is obtained as (ii ) "a wireless signal transmitted by a communication terminal of a customer who entered the store 12 and used the service provided by the store visit confirmation device 142" It can be used instead of a learning data set with a positive label of "radio signal emitted by a communication terminal of a customer who did not use the service."

[Overview of Each Part of Data Analysis System 100]
In this embodiment, the communication network 10 transmits information between at least one of the communication terminal 108 , the access point 122 , the access point 124 , the store visit confirmation device 142 and the store visit confirmation device 144 and the management server 160 . The communication network 10 may be a wired communication transmission line, a wireless communication transmission line, or a combination of a wireless communication transmission line and a wired communication transmission line.

Communication network 10 may include wireless communication networks, the Internet, P2P networks, leased lines, VPNs, power line communication lines, and the like. The communication system in the wireless communication network may be (i) a mobile communication system such as a 3G system, an LTE system, or a 4G system; Communication), a wireless LAN system such as WiFi (registered trademark), a wireless MAN system such as WiMAX (registered trademark), a wireless data communication system such as a wireless WAN system.

In this embodiment, each of communication terminal 102 and communication terminal 104 has a wireless communication function and transmits a wireless signal. The wireless communication method may be a mobile communication method or a wireless data communication method. Communication terminal 102 transmits and receives information to and from other communication devices via access point 122 and communication network 10, for example. Communication terminal 104 transmits and receives information to and from other communication devices via access point 124 and communication network 10, for example.

Details of each of communication terminal 102 and communication terminal 104 are not particularly limited. Each of the communication terminal 102 and the communication terminal 104 is exemplified by a personal computer, a mobile terminal, and the like. Examples of mobile terminals include mobile phones, smart phones, PDAs, tablets, notebook or laptop computers, wearable computers, and the like. Each of communication terminal 102 and communication terminal 104 may be a mobile terminal.

In this embodiment, the communication terminal 108 transmits and receives information to and from the management server 160 via the communication network 10 . Communication terminal 108 may transmit various requests to management server 160 . Communication terminal 108 may obtain a response to the above request from management server 160 . For example, communication terminal 108 requests to perform processing for estimating the number of customers who have visited store 12 or store 14 .

Details of the communication terminal 108 are not particularly limited. Examples of the communication terminal 108 include a personal computer and a mobile terminal. Examples of mobile terminals include mobile phones, smart phones, PDAs, tablets, notebook or laptop computers, wearable computers, and the like.

In this embodiment, the access point 122 relays communication of the communication terminal 102 using a wireless LAN system. For example, access point 122 allows information to be conveyed between communication terminal 102 and communication network 10 . Various signals are exchanged between the access point 122 and the communication terminal 102 even before communication is established. For example, after the connection between the access point 122 and the communication terminal 102 is completed, and before the communication terminal 102 starts communication with another information processing device, a probe request is issued between the access point 122 and the communication terminal 102. Related processing, authentication processing, association processing, etc. are executed.

In this embodiment, the access point 122 receives radio signals output by the communication terminal 102 . The access point 122 may determine that the wireless signal has been received when the radio field strength of the wireless signal that has reached the access point 122 is equal to or greater than a predetermined value. Access point 122 analyzes the received radio signal and acquires identification information (sometimes referred to as a terminal ID) for identifying communication terminal 102 .

The radio signal may be a signal received after communication is established, or a signal received before communication is established. Examples of signals received before communication is established include probe requests, authentication requests, association requests, and the like. A signal received after communication is established may be a disassociation request or a detach request (sometimes referred to as a detach request). A MAC address is exemplified as the identification information of the communication terminal 102 included in these signals.

The terminal ID of communication terminal 102 may be an example of terminal identification information. The MAC address of communication terminal 102 may be an example of terminal identification information.

The access point 122 stores, for example, log information in which the terminal ID of the communication terminal 102 that transmitted the wireless signal, time information indicating the time at which the wireless signal was received, and information on the radio wave of the wireless signal are associated with each other. , to the management server 160 . The log information includes identification information of the access point 122, the terminal ID of the communication terminal 102 that transmitted the radio signal, information indicating the time when the radio signal was received, and information about the radio wave of the radio signal. It may be information that has been Examples of the identification information of the access point 122 include SSID, MAC address, IP address, and the like.

In this embodiment, the access point 124 relays communication of the communication terminal 104 using a wireless LAN system. For example, access point 124 allows information to be conveyed between communication terminal 104 and communication network 10 . Various signals are exchanged between the access point 124 and the communication terminal 104 even before communication is established. For example, after the connection between the access point 124 and the communication terminal 104 is completed, and before the communication terminal 104 starts communication with another information processing device, a probe request is issued between the access point 124 and the communication terminal 104. Related processing, authentication processing, association processing, etc. are executed.

In this embodiment, the access point 124 receives radio signals output by the communication terminal 104 . The access point 124 may determine that the wireless signal has been received when the radio signal strength of the wireless signal that reaches the access point 124 is equal to or greater than a predetermined value. Access point 124 analyzes the received radio signal and acquires identification information (sometimes referred to as a terminal ID) for identifying communication terminal 104 .

The radio signal may be a signal received after communication is established, or a signal received before communication is established. Examples of signals received before communication is established include probe requests, authentication requests, association requests, and the like. A signal received after communication is established may be a disassociation request or a detach request (sometimes referred to as a detach request). A MAC address is exemplified as the identification information of the communication terminal 104 included in these signals.

The terminal ID of communication terminal 104 may be an example of terminal identification information. The MAC address of communication terminal 104 may be an example of terminal identification information.

The access point 124 stores, for example, log information in which the terminal ID of the communication terminal 104 that transmitted the radio signal, information indicating the time at which the radio signal was received, and information on the radio wave of the radio signal are associated with each other. Send to the management server 160 . The log information includes identification information of the access point 124, the terminal ID of the communication terminal 104 that transmitted the radio signal, information indicating the time when the radio signal was received, and information about the radio wave of the radio signal. It may be information that has been Examples of the identification information of the access point 124 include SSID, MAC address, IP address, and the like.

In this embodiment, the store visit confirmation device 142 detects the behavior of the user 22 inside the store 12 and generates store visit confirmation information indicating that the user 22 of the communication terminal 102 was inside the store 12 . The store visit confirmation device 142 transmits the generated store visit confirmation information to the management server 160 . The store visit confirmation information is information in which identification information (sometimes referred to as a user ID) for identifying the user 22 involved in the above action is associated with information indicating the time at which the action was performed. can be The store visit confirmation information is information in which the user ID of the user 22 involved in the action, information indicating the time when the action was performed, and information indicating the position of the place where the action was performed are associated with each other. good too. The user ID is preferably data that does not lead to individual identification.

The store visit confirmation device 142 may establish communication with the communication terminal 102 by short-range wireless communication or wired communication, and acquire the terminal ID of the communication terminal 102 . In this case, the store-visit confirmation information may be information in which the terminal ID of the communication terminal 102, information indicating the time when the terminal ID was acquired, and information indicating the location of the store 12 are associated with each other. The information indicating the location of the store 12 is exemplified by the name of the store 12, the address of the store 12, the location coordinates of the store 12, and the like. The visit confirmation information may be information in which the terminal ID of the communication terminal 102 and information indicating the time when the terminal ID was acquired are associated with each other.

The actions of the user 22 inside the store 12 include (i) an act of purchasing a product inside the store 12, (ii) an act of using a service provided inside the store 12, and (iii) an act inside the store 12. , the act of using the biometric authentication system, (iv) the act of establishing communication between the communication terminal 102 and the store visit confirmation device 142 by short-range wireless communication or wired communication inside the store 12, and (v) the entrance/exit management system. An act of entering the inside of the store 12 using it is exemplified. The service provided inside the store 12 may be a membership service. Examples of membership services include credit card services, point card services, point services, point programs, loyalty programs, mileage services, and mileage programs. The entrance/exit management system utilizes at least one of the biometric information of the user 22, the user ID stored in the ID card lent to the user 22, and the terminal ID of the communication terminal 102 to control the entry and exit of the user 22. You may manage at least one of

The store visit confirmation device 142 is not particularly limited in detail as long as it is a device into which the user ID of the user 22 or the terminal ID of the communication terminal 102 and the information indicating the time are inputted. Also, the type of user ID is not particularly limited. The store visit confirmation device 142 includes a system for inputting the name or identification information of a customer who used the service provided at the store 12, a POS system, a system for acquiring a membership number for a membership service, an entrance/exit management system, A biometrics authentication system etc. are illustrated. The system in which the names or identification information of customers who use the services provided at the storefront of the store 12 are entered may be a system in which the staff of the store 12 enter the names or identification information. Information input to each system or information acquired by each system may be an example of a user ID. The type of information input to each system or the type of information acquired by each system may be an example of the type of user ID.

In this embodiment, the store visit confirmation device 144 detects the behavior of the user 24 inside the store 14 and generates store visit confirmation information indicating that the user 24 of the communication terminal 104 was inside the store 14 . The store visit confirmation device 144 transmits the generated store visit confirmation information to the management server 160 . The store visit confirmation information is information in which identification information (sometimes referred to as a user ID) for identifying the user 24 involved in the above action is associated with information indicating the time at which the action was performed. can be The store visit confirmation information is information in which the user ID of the user 24 involved in the action, information indicating the time when the action was performed, and information indicating the position of the place where the action was performed are associated with each other. good too. The user ID is preferably data that cannot be used to identify an individual, or data that makes it difficult to identify an individual.

The store visit confirmation device 144 may establish communication with the communication terminal 104 by short-range wireless communication or wired communication, and acquire the terminal ID of the communication terminal 104 . In this case, the store-visit confirmation information may be information in which the terminal ID of the communication terminal 104, information indicating the time when the terminal ID was obtained, and information indicating the location of the store 14 are associated with each other. The information indicating the location of the store 14 is exemplified by the name of the store 14, the address of the store 14, the location coordinates of the store 14, and the like. The store visit confirmation information may be information in which the terminal ID of the communication terminal 104 and information indicating the time when the terminal ID was acquired are associated with each other.

The actions of the user 24 inside the store 14 include (i) the act of purchasing products inside the store 14, (ii) the act of using services provided inside the store 14, and (iii) the inside of the store 14. , the act of using the biometric authentication system, (iv) the act of establishing communication between the communication terminal 104 and the visit confirmation device 144 by short-range wireless communication or wired communication inside the store 14, and (v) the entrance/exit management system. An act of entering the inside of the store 14 using it is exemplified. The service provided inside the store 14 may be a membership service. Examples of membership services include credit card services, point card services, point services, point programs, loyalty programs, mileage services, and mileage programs. The entrance/exit management system utilizes at least one of the biometric information of the user 24, the user ID stored in the ID card lent to the user 24, and the terminal ID of the communication terminal 104 to control the entry and exit of the user 24. You may manage at least one of

The store visit confirmation device 144 is not particularly limited in detail as long as it is a device to which the user ID of the user 24 or the terminal ID of the communication terminal 104 and the information indicating the time are inputted. The store visit confirmation device 144 includes a system for inputting the name or identification information of a customer who uses the service provided at the store 14, a POS system, a system for acquiring a membership number for a membership service, an entrance/exit management system, A biometrics authentication system etc. are illustrated. The system in which the names or identification information of customers who use the services provided at the shop 14 are input may be a system in which the staff of the shop 14 inputs the names or identification information.

In this embodiment, the management server 160 manages information processing in the data analysis system 100 . The management server 160 may manage log information. The management server 160 acquires one or more pieces of log information from at least one of the access points 122 and 124, for example. The management server 160 stores the acquired log information in the database. The management server 160 may manage store visit confirmation information. For example, the management server 160 acquires one or more items of store visit confirmation information from at least one of the store visit confirmation device 142 and the store visit confirmation device 144 . The management server 160 stores the acquired visit confirmation information in the database.

In this embodiment, the management server 160 analyzes the collected log information, and the user of the communication terminal that has transmitted the radio signal related to each piece of log information It is determined whether or not it exists inside the More specifically, the management server 160 may generate a learning model used for the determination process described above. Log information or data obtained by processing log information based on a predetermined algorithm is input to the learning model. The above learning model outputs a determination result as to whether or not the user of the communication terminal that transmitted the radio signal related to the above log information was present inside the store or the like where the access point that transmitted the radio signal is located. do. Management server 160 may adjust the threshold for classification. The management server 160 may perform the determination process described above using the learning model described above and the adjusted threshold.

In this embodiment, the management server 160 acquires various requests from the communication terminal 108 . The management server 160 executes processing in response to the request, and transmits the execution result of the processing to the communication terminal 108 as a response to the request. For example, the management server 160 executes processing for estimating the number of customers who have visited the store 12 or 14 in response to a request from the communication terminal 108 . The management server 160 may execute processing for estimating the number of passers-by passing outside the store 12 or 14 in response to a request from the communication terminal 108 . Details of the management server 160 will be described later.

In this embodiment, the management server 160 stores (i) user identification information for identifying each of the one or more users involved in a specific action, and (ii) the time at which the specific action was performed. The details of the management server 160 will be described by taking as an example a case where store visit confirmation information is used as action history information associated with indicated action time information. However, the action history information is not limited to this embodiment.

Examples of the action history information include product purchase history and service usage history. Examples of product purchase history or service usage history include credit card usage history, purchase history based on POS information, entry/exit history of specific facilities, and usage history of services that provide privileges to users who have registered as members. be. A point card service, a point service, a point program, a loyalty program, a mileage service, a mileage program, and the like are examples of services that give benefits to users who have registered as members. Services that give benefits to users who have registered as members may be provided by railway companies, airlines, expressway operating companies, companies, corporate groups, and the like.

The credit card usage history includes, for example, information in which the credit card number, the store where the credit card was used, and the time when the credit card was used are associated with each other. The usage history of the service that gives privileges to users who have registered as members includes, for example, information in which the member ID, the store where the product was purchased or the service was used, and the time when the service was used are associated with each other. Credit card numbers, member IDs, etc. may be replaced with further anonymized identification symbols.

Another example of action history information is a bid request (sometimes referred to as a bid request) in an advertisement bidding service such as an RTB (Real Time Bidding) service. For example, when a user accesses a web page having an ad space, a bid request is issued from a SSP (Supply Side Platform) to a DSP (Demand Side Platform) via AdExchange.

The bid request includes a user ID that identifies the user browsing the web page having the ad space. A user ID is, for example, a cookie. A bid request may include location information such as latitude and longitude, as in the case where the user is browsing the web page via a web browser running on a mobile terminal. In addition, a bid deadline of about 100 msec is set in the bid request sent from AdExchange to the DSP, and the advertising space is bought and sold almost in real time. Therefore, one or more bid requests can be used as action history information by correlating and storing the bid request with the time when the bid request was issued or received or the bid deadline of the bid request.

Another example of action history information may be image data or video data to which information about the position and time at which an image or video was shot is added. In this embodiment, first, features of a subject are extracted from image data or video data using a subject recognition algorithm. The subject can be identified by the extracted features of the subject. Next, by associating the feature of the subject with the information about the position and time when the image or video was taken, the image data or video data can be used as action history information.

Another example of action history information may be information on SNS (Social Networking Service) including location information. The action history information may be information in which information including the location information is associated with information indicating the time when the information was uploaded on the SNS. Information on SNS including location information includes tweets with Geo tags, Facebook (registered trademark) check-in information, and image data or video data with information about the location and time when an image or video was taken. etc. are exemplified. For example, a Tweet with a Goe tag includes information in which a Twitter (registered trademark) account, location information at which the Tweet was sent, and time at which the Tweet was sent are associated with each other.

In this embodiment, the wireless receiving device is an access point, and an area having a predetermined geographical range is a store where the access point is installed. explained. However, the management server 160 is not limited to this embodiment. In another embodiment, the radio receiver may be a base station forming part of a mobile communication network. In this case, the area with the predetermined geographical range may be the cell of each base station.

Further, in the present embodiment, the details of the data analysis system 100 will be described by taking as an example a case where the communication terminal 108 requests execution of processing for estimating the number of customers who have visited the store 12 or 14. was done. However, the data analysis system 100 is not limited to this embodiment. In another embodiment, communication terminal 108 may request that a process be performed to estimate the number of passersby passing outside store 12 or store 14 .

[Specific configuration of each part of data analysis system 100]
Each part of the data analysis system 100 may be implemented by hardware, software, or a combination of hardware and software. When at least part of the components of the data analysis system 100 are implemented by software, the components implemented by the software are programs that define the operations of the components in an information processing device with a general configuration. may be implemented by invoking the

The program may be stored in a computer-readable medium such as a CD-ROM, DVD-ROM, memory, hard disk, etc., or may be stored in a storage device connected to a network. The program may be installed on a computer forming at least part of the data analysis system 100 from a computer-readable medium or network-connected storage device. By executing the program, the computer may function as at least part of each part of the data analysis system 100 .

A program that causes a computer to function as at least part of each unit of data analysis system 100 may include a module that defines the operation of each unit of data analysis system 100 . These programs or modules act on a data processing device, an input device, an output device, a storage device, etc. to cause the computer to function as each part of the data analysis system 100, or instruct the computer to perform information processing methods in each part of the data analysis system 100. or let it run.

The information processing described in the program functions as concrete means in which the software related to the program and various hardware resources of the data analysis system 100 cooperate when the program is read into the computer. . The data analysis system 100 corresponding to the purpose of use of the computer in this embodiment is constructed by the specific means described above realizing the calculation or processing of information according to the purpose of use of the computer.

FIG. 2 schematically shows an example of the internal configuration of the management server 160. As shown in FIG. In this embodiment, the management server 160 includes an input/output control unit 220 , an information collection unit 230 , an analysis unit 240 , and a visitor number estimation unit 270 . In this embodiment, the analysis unit 240 has a storage unit 242 , an input data generation unit 244 , a model construction unit 250 and a determination unit 260 . In this embodiment, the model builder 250 includes a dataset generator 252 , a learner 254 and a threshold adjuster 256 . In this embodiment, each unit of the management server 160 may send and receive information to and from each other.

The information collection unit 230 may be an example of a record acquisition unit, an action history acquisition unit, and a correspondence relationship acquisition unit. The input data generation unit 244 may be an example of a record acquisition unit, a determination target acquisition unit, and an aggregating unit. The data set generation unit 252 may be an example of a record acquisition unit, an action history acquisition unit, a correspondence acquisition unit, a label assignment unit, and a determination target acquisition unit. The learning unit 254 may be an example of a model generation unit. The threshold adjuster 256 may be an example of an adjuster. The determination unit 260 may be an example of a determination unit.

In this embodiment, the input/output control unit 220 controls input/output of information in the management server 160 . For example, the input/output control unit 220 controls at least one of an input device, an output device, and a communication device (not shown) of the management server 160 .

In this embodiment, the information collection unit 230 collects various types of information and stores the collected information in the storage unit 242 . In one embodiment, information collector 230 acquires one or more pieces of log information from at least one of access point 122 and access point 124 . In another embodiment, the information collection unit 230 acquires one or more items of store visit confirmation information from at least one of the store visit confirmation device 142 and the store visit confirmation device 144 .

In yet another embodiment, the information collecting unit 230 acquires correspondence information indicating the correspondence between each of one or more user IDs and the terminal ID of the communication terminal used by the user identified by the user ID. . In one embodiment, the information collection unit 230 acquires correspondence information from an external information processing device. In another embodiment, the information collection unit 230 generates correspondence information based on one or more pieces of log information and one or more pieces of visit confirmation information.

In this embodiment, the analysis unit 240 analyzes at least part of the one or more log information and the one or more visitor confirmation information collected by the information collection unit 230 . The analysis unit 240 determines whether or not the user of the communication terminal that transmitted the radio signal related to each piece of log information was present inside the store or the like where the access point that transmitted the radio signal is located. The analysis unit 240 receives log information or data obtained by processing the log information based on a predetermined algorithm, and the user of the communication terminal that transmits the radio signal related to the log information A learning model may be generated for determining whether or not the user is inside a store or the like where the access point that transmitted the wireless signal is located. The analysis unit 240 may adjust the reference value for the learning model to determine whether the user was present inside the store or the like.

In this embodiment, the storage unit 242 stores various information used in information processing in the management server 160 . The storage unit 242 stores various types of information generated during information processing in the management server 160 . Details of the storage unit 242 will be described later.

In the present embodiment, the input data generation unit 244 generates input data to be input to the learning model in the learning process of the learning model in the learning unit 254 or the determination process in the determination unit 260 . In this embodiment, the input data generation unit 244 collects a plurality of pieces of log information acquired by the information collection unit 230 based on the terminal ID and time information included in each of the pieces of log information, for each terminal identification ID, and , to generate one or more pieces of input data by aggregating for each period that satisfies a predetermined condition.

Each of the one or more pieces of input data indicates (i) a terminal ID for identifying the communication terminal that transmitted each of the plurality of radio signals, and (ii) a period subject to aggregation processing by the input data generation unit 244. Aggregation period information, (iii) a feature indicating one or more feature amounts relating to one or more radio signals transmitted during a period indicated by the aggregation period information by a communication terminal identified by a terminal ID among a plurality of radio signals; It may be information associated with information. Each of the one or more pieces of input data may be an example of an aggregation record.

Examples of one or more types of feature values include radio wave intensity during the unit period, length of stay during the unit period, number of visits during the stay period, radio wave intensity during the stay period, store attributes, and the like. Other examples of the types of one or more feature values include radio wave intensity per unit period for each frequency band, length of stay per unit period for each frequency band, number of visits per frequency band during stay period, For example, the radio wave intensity during the stay period for each frequency band is exemplified. Other examples of the types of one or more feature values include the radio field strength during the unit period for each connection stage, the length of stay during the unit period for each connection stage, the number of visits during the stay period for each connection stage, and each connection stage. radio wave intensity during the period of stay in the

The period of stay shall be defined as a wireless signal from a specific communication terminal, whose radio wave intensity is higher than the predetermined strength, during a unit period having a predetermined length at an access point located at a specific store, etc. may be the period between the time the radio signal is first detected and the time it was last detected. The number of visits during the period of stay is the period during which the radio signal from the above-mentioned specific communication terminal continues to be detected and the radio signal strength is greater than a predetermined strength during the specific period of stay. (sometimes referred to as duration) may be the number of times that appears.

The attributes of the store are exemplified by the area where the store is opened, the type of store opened, the location conditions of the store, the structure of the store, and combinations thereof. Examples of store opening districts include city centers, suburbs, in front of stations, roadsides, tourist spots, and the like. Examples of the form of opening a store include a shopping street, a store within a store, and a single store. Examples of store location conditions include a street store, an aerial store, and an underground store.

At least one feature amount value may be a statistical value of an amount indicated by signal information included in each of a plurality of pieces of log information aggregated into single input data. Examples of statistical values include maximum values, minimum values, average values, and median values. The above statistic value may be a statistic value for a unit period or a statistic value for a length of stay. The above statistic may be a statistic over a period of time.

In this embodiment, part of the input data generated by the input data generation unit 244 becomes a learning data set used as at least one of training data, verification data, and test data for machine learning in the learning unit 254. Of the input data generated by the data generation unit 244 , data that is not used as the learning data set becomes determination target data that is subjected to determination processing in the determination unit 260 . More specifically, the input data labeled by the information processing in the data set generation unit 252 becomes the learning data set, and the input data not labeled by the information processing in the data set generation unit 252 is used for the determination. It becomes the target data.

In this embodiment, the model construction unit 250 generates a learning model. The learning model described above is input with data points containing information about radio signals received by radio receivers located within an area having a predetermined geographic range, and the user of the radio terminal that originated the radio signal. existed inside the area. The learning model may calculate, for each data point input to the learning model, the value of a function used to estimate the uncertainty of the prediction or decision of the learning model. The above learning model may determine whether or not the user was present inside the area based on the calculated function value and the threshold for classification.

In this embodiment, the model construction unit 250 generates a learning data set used for machine learning of the learning model described above. In this embodiment, the model builder 250 adjusts the threshold for classification. As a result of the adjustment described above, the model construction unit 250 determines a new threshold for classification that the learned model uses in the determination processing in the determination unit 260 .

In this embodiment, the data set generation unit 252 generates a learning data set used for machine learning in the learning unit 254 . The dataset generator 252 generates a learning dataset using at least one of the log information acquired by the information collector 230 and the input data generated by the input data generator 244, for example.

In one embodiment, the dataset generation unit 252 generates a learning dataset by assigning a positive or negative label to part of the log information acquired by the information collection unit 230 . The dataset generation unit 252 may store the generated learning dataset in the storage unit 242 .

For example, the data set generation unit 252 compares one or more collected store visit confirmation information with one or more collected log information to obtain (i) store visit confirmation information related to a specific user, and (ii) the relevant Log information related to the communication terminal used by the user, wherein the difference between the reception time indicated by the log information and the arrival time indicated by the store visit confirmation information is equal to or greater than a predetermined value Associate small log information with . The data analysis system 100 generates a learning data set, for example, by assigning a positive label to log information associated with store visit confirmation information.

In another embodiment, the dataset generator 252 generates a learning dataset by adding a positive or negative label to part of the input data generated by the input data generator 244 . The dataset generation unit 252 may store the generated learning dataset in the storage unit 242 .

For example, the data set generation unit 252 compares one or more pieces of collected store visit confirmation information with one or more pieces of input data to obtain (i) store visit confirmation information about a specific user and (ii) store visit confirmation information used by the user. input data relating to a communication terminal, wherein the difference between the reception time indicated by the input data and the visit time indicated by the store visit confirmation information is equal to or smaller than a predetermined value and associate. The data set generation unit 252 generates a learning data set, for example, by assigning a positive label to the input data associated with the store visit confirmation information.

The data set generation unit 252 may generate determination target data to be subjected to determination processing in the determination unit 260 . The determination target data may include information regarding multiple wireless signals received by at least one of the one or more access points.

In one embodiment, the data set generation unit 252 outputs, among the log information acquired by the information collection unit 230, log information that has not been labeled as data to be determined. In another embodiment, the data set generation unit 252 outputs, among the input data generated by the input data generation unit 244, unlabeled input data as determination target data. The data set generation unit 252 may store the generated determination target data in the storage unit 242 .

[Specific example 1-1 of method for generating learning data set]
In this embodiment, the data set generation unit 252 accesses the storage unit 242 and acquires multiple pieces of log information. In each of the plurality of pieces of log information, (i) terminal identification information for identifying the communication terminal that transmitted each of the plurality of radio signals, and (ii) reception time indicating the time when each of the plurality of radio signals was received The information may be associated with (iii) signal information about radio waves of each of the plurality of wireless signals.

The data set generation unit 252 accesses the storage unit 242 and acquires a plurality of store visit confirmation information. In each of the plurality of store visit confirmation information, (i) user identification information for identifying each of the one or more users involved in a specific action, and (ii) action time information indicating the time when the specific action was performed. may be associated. More specifically, each of the plurality of store visit confirmation information is information in which user identification information of the user and information indicating the user's store visit time (sometimes referred to as store visit time information) are associated with each other. It's okay. The visit time information may be an example of action time information.

The data set generation unit 252 generates (i) terminal identification information and reception time information included in each of the plurality of pieces of log information, and (ii) user identification information and visit time information included in each of the plurality of pieces of store visit confirmation information. Based on this, multiple pieces of log information and multiple pieces of store visit confirmation information are collated, and the terminal identification information included in each of at least a portion of the log information among the plurality of pieces of log information, and the terminal identification information included in each of the at least a portion of the log information. A learning data set is generated by associating the information about radio waves indicated by the signal information received with the positive label. As a result, a learning data set is generated in which at least some of the pieces of log information are positively labeled.

More specifically, the data set generation unit 252 first uses the terminal identification information and the reception time information included in the specific log information as keys for verification to select (i) (ii) the user indicated by the user identification information included in the visit confirmation information matches the user of the communication terminal indicated by the terminal identification information included in the specific log information; and The store visit confirmation information whose time indicated by the store visit time information matches the time indicated by the reception time information included in the specific log information is extracted.

Next, the data set generator 252 associates the extracted store visit confirmation information with the specific log information. Next, the data set generation unit 252 generates a learning data set, for example, by assigning a positive label to the log information associated with the store visit confirmation information. The data set generation unit 252 may perform the above processing for each of the plurality of pieces of log information. A particular log information may be an example of at least some records.

In one embodiment, when the user identification information included in at least a part of the plurality of store visit confirmation information is the terminal identification information of the communication terminal used by the user identified by the user identification information, the data set generation unit 252 , store visit confirmation information including specific terminal identification information, (ii) log information including the specific terminal identification information, the reception time indicated by the reception time information included in the log information, and the store visit confirmation The difference between the store visit time indicated by the store visit time information included in the information is associated with log information that is equal to or smaller than a predetermined value. The data set generation unit 252 generates the learning data set, for example, by assigning a positive label to the log information associated with the store visit confirmation information.

In another embodiment, when the user identification information included in at least a part of the plurality of store visit confirmation information is not the terminal identification information of the communication terminal used by the user identified by the user identification information, the data set generation unit 252 , accesses the storage unit 242 and acquires the correspondence information related to the user identification information. Correspondence information may be information indicating a correspondence relationship between user identification information and terminal identification information of a communication terminal used by a user identified by the user identification information.

The data set generation unit 252 refers to the correspondence information, and (i) the terminal identification information and reception time information included in each of the plurality of log information, and (ii) the information included in each of the plurality of store visit confirmation information. A plurality of pieces of log information and a plurality of pieces of store visit confirmation information are collated based on the user identification information and the store visit time information, and at least a portion of the plurality of pieces of log information and at least a portion of the plurality of store visit confirmation information are associated. The data set generation unit 252 generates the learning data set, for example, by assigning a positive label to the log information associated with the store visit confirmation information.

More specifically, in the procedure for associating at least a portion of a plurality of pieces of log information with at least a portion of a plurality of pieces of store-visit confirmation information, the data set generation unit 252 first generates the Using the terminal identification information and the reception time information received as a key for verification, store visit confirmation information that matches the above-described conditions is extracted from a plurality of store visit confirmation information. Next, the data set generator 252 associates the extracted store visit confirmation information with the specific log information.

According to the present embodiment, in the procedure for extracting store-visit confirmation information that matches the above-described conditions from among a plurality of store-visit confirmation information, the data set generation unit 252 uses the correspondence information to generate a specific It is determined whether or not the user indicated by the user identification information included in the visit confirmation information matches the user of the communication terminal indicated by the terminal identification information included in the specific log information. In one embodiment, the data set generator 252 uses the correspondence information to convert the terminal identification information included in the log information into corresponding user identification information. Thereby, the data set generation unit 252 can determine whether or not the log information and the visit confirmation information are information related to the same user or communication terminal, using the user identification information as a key. In another embodiment, the data set generation unit 252 converts the user identification information included in the visit confirmation information into corresponding terminal identification information using the correspondence information. Thereby, the data set generation unit 252 can determine whether or not the log information and the visit confirmation information are information related to the same user or communication terminal, using the terminal identification information as a key.

In each of the above embodiments, when the difference between the arrival time indicated by the arrival time information and the reception time indicated by the reception time information is equal to or smaller than a predetermined value, the arrival time information may be determined to match the reception time indicated by the reception time information. Also, the predetermined value regarding the difference between the visit time and the reception time may be n times the transmission interval of the specific type of radio signal. n may be an integer of 1 or more, an integer of 2 or more, an integer of 3 or more, an integer of 4 or more, or an integer of 5 or more It may be an integer of 10 or more. n may be an integer of 50 or less, an integer of 30 or less, an integer of 20 or less, an integer of 10 or less, or an integer of 5 or less. It may be an integer of 3 or less. Specific types of radio signals are exemplified by various communication control signals such as probe requests and location registration requests.

[Specific example 1-2 of method for generating learning data set]
In this embodiment, the dataset generation unit 252 accesses the storage unit 242 to obtain one or more pieces of input data. In each of the one or more pieces of input data, (i) terminal identification information of the communication terminal that transmitted the wireless signal to be aggregated, (ii) aggregation period information indicating the period of time to be aggregated, (iii) Feature information indicating one or more feature amounts related to wireless signals to be aggregated may be associated.

The terminal identification information included in the input data may be an example of the terminal identification information included in the record. Aggregation period information included in the input data may be an example of information based on reception time information. The feature information included in the input data may be an example of information about radio waves indicated by the signal information.

The data set generation unit 252 accesses the storage unit 242 and acquires one or more store visit confirmation information. In each of the one or more store visit confirmation information, (i) user identification information for identifying each of the one or more users involved in a specific action, and (ii) action time information indicating the time when the specific action was performed may be associated with. More specifically, each of the one or more pieces of visit confirmation information may be information in which user identification information of a user and visit time information indicating the visit time of the user are associated with each other.

The data set generation unit 252 generates (i) terminal identification information and aggregation period information contained in each of the one or more pieces of input data, and (ii) a plurality of store visit confirmation information Based on user identification information and store visit time information included in each, one or more input data and a plurality of store visit confirmation information are collated, and terminal identification included in each of at least part of the one or more input data A learning data set is generated by associating the information, the information about the feature quantity indicated by the feature information included in each of the at least part of the input data, and the positive label. As a result, a learning data set is generated in which at least a portion of one or more pieces of input data are positively labeled.

More specifically, the data set generation unit 252 first uses the terminal identification information and aggregation period information included in the specific input data as keys for verification to select (i) (ii) the user indicated by the user identification information included in the visit confirmation information matches the user of the communication terminal indicated by the terminal identification information included in the specific input data; and The time indicated by the store visit time information is extracted as store visit confirmation information included in the period indicated by the aggregation period information included in the specific input data.

Next, the data set generator 252 associates the extracted store visit confirmation information with the specific input data. Next, the data set generation unit 252 generates a learning data set, for example, by assigning a positive label to the input data associated with the store visit confirmation information. The dataset generator 252 may perform the above process for each of the one or more pieces of input data.

[Specific example 2-1 of method for generating learning data set]
In this embodiment, the data set generation unit 252 accesses the storage unit 242 and acquires multiple pieces of log information. In each of the plurality of pieces of log information, (i) terminal identification information for identifying the communication terminal that transmitted each of the plurality of radio signals, and (ii) reception time indicating the time when each of the plurality of radio signals was received information, (iii) reception position information indicating a position where each of the plurality of radio signals is received, and (iv) signal information regarding radio waves of each of the plurality of radio signals may be associated.

The data set generation unit 252 accesses the storage unit 242 and acquires the entry restriction information of the store arranged at the position indicated by the reception position information included in each of the plurality of pieces of log information. Each of the one or more pieces of entry restriction information indicates (i) a store ID for identifying each of the one or more stores, and (ii) a time period during which access to the store identified by the store ID is restricted. information may be associated with each other. The store ID may be an example of area identification information.

The data set generation unit 252 collates a plurality of pieces of log information and one or more access restriction information based on the reception position information included in each of the plurality of log information and the store ID included in the acquired access restriction information. and extracts access restriction information corresponding to the position indicated by the reception position information included in each of the plurality of log information. The data set generator 252, for example, assigns a negative label to the log information from which the corresponding entry restriction information has been extracted. A negative label indicates that the user of the communication terminal that transmitted the wireless signal was not inside the store or the like where the access point that received the wireless signal is located. As a result, the terminal identification information included in each of at least some of the log information among the plurality of log information, the information on radio waves indicated by the signal information included in each of at least some of the log information, and the negative label. An associated training data set is generated.
[Specific example 2-2 of method for generating learning data set]
In this embodiment, the dataset generation unit 252 accesses the storage unit 242 to obtain one or more pieces of input data. In each of the one or more pieces of input data, (i) the terminal ID of the communication terminal that transmitted the one or more wireless signals to be aggregated, (ii) the access point that received the wireless signal to be aggregated. (iii) aggregation period information indicating a period for which aggregation processing is performed; you can

The data set generation unit 252 accesses the storage unit 242 and acquires entry restriction information of the store indicated by the store ID included in each of the one or more pieces of input data. In each of the one or more items of access restriction information, (i) a store ID and (ii) information indicating a time zone during which access to the store identified by the store ID is restricted may be associated.

The data set generation unit 252 generates a plurality of pieces of log information and one or more entry data based on the store ID included in each of the one or more pieces of input data and the store ID included in the entry restriction information stored in the storage unit 242 . The restriction information is collated to extract entry restriction information corresponding to each of the one or more pieces of input data. The data set generator 252, for example, assigns a negative label to the input data from which the corresponding entry restriction information has been extracted. As a result, the terminal identification information included in each of at least a portion of the log information among the plurality of log information, the information on the radio waves indicated by the signal information included in each of the at least a portion of the log information, and the negative label. An associated training data set is generated.

In this embodiment, the learning unit 254 generates a learning model by machine learning. In the above learning model, for example, a data point containing information about a radio signal received by an access point placed inside a specific store is input, and the user of the communication terminal that transmitted the radio signal It is used to determine whether it existed inside the The learning unit 254 may generate a different learning model for each store, or may generate a common learning model for a plurality of stores. Learning unit 254 may generate a different learning model for each access point, or may generate a common learning model for a plurality of access points.

For example, the learning unit 254 accesses the storage unit 242 and acquires the learning data set generated by the data set generation unit 252 . As described above, in each learning model, an appropriate initial value corresponding to the learning model is set as a threshold for classification. Therefore, the learning unit 254 first uses the initial value as a threshold for classification and performs machine learning using the learning data set to generate a learned learning model. .

The learning unit 254 may perform cross-validation to evaluate the generalization performance of the generated learning model. The learning unit 254 may perform a grid search with cross-validation to determine the hyperparameters of the learning model.

The learning unit 254 may divide the learning data set generated by the data set generation unit 252 into training data and test data. The learning unit 254 may divide the learning data set generated by the data set generation unit 252 into training data, verification data, and test data. The learning unit 254 may divide the verification data into verification data used for verification for determining parameters included in the learning model and verification data used for verification for adjusting the threshold for classification. . Verification data used for verification for adjusting the threshold for classification is used by the threshold adjuster 256, for example.

In this embodiment, the threshold adjuster 256 adjusts the classification threshold for the learning model generated by the learner 254 . For example, the threshold adjusting unit 256 determines, as the adjusted threshold, a value that increases the true positive rate of the determination result of the learning model compared to when the initial value is used as the threshold for classification.

For example, the threshold adjustment unit 256 first calculates data indicating a receiver operating characteristic curve (ROC curve) of the learning model generated by the learning unit 254 . For example, the threshold adjustment unit 256 gradually changes the classification threshold by using a data set that is not used for machine learning in the learning unit 254 among the learning data sets generated by the data set generation unit 252. while calculating the true positive rate and false positive rate of the learning model. Thereby, data representing the ROC curve is calculated. Next, the threshold adjustment unit 256 corresponds to a point on the ROC curve at which the true positive rate of the determination result of the learning model increases compared to when the initial value is used as the threshold for classification. is determined as the adjusted threshold.

In the present embodiment, the determination unit 260 determines determination target data using the learning model generated by the learning unit 254 and the adjusted threshold determined by the threshold adjustment unit 256 . For example, the determination unit 260 accesses the storage unit 242 and acquires determination target data generated by the data set generation unit 252 . The determination unit 260 inputs the determination target data to the learning model generated by the learning unit 254 . The learning unit learning unit 254 uses the adjusted threshold value determined by the threshold adjustment unit 256 to determine whether the user of the communication terminal indicated by the terminal ID included in the determination target data is identified by the store ID included in the determination target data. It is determined whether or not it existed inside the store where the The determination unit 260 may store output data indicating the determination result in the storage unit 242 .

In this embodiment, the visitor number estimation unit 270 estimates, for example, the number of customers who visited a specific store during a specific period in response to a request from the communication terminal 108 . The visitor number estimation unit 270 calculates the number of communication terminal users who were present at a specific store during a specific period based on the output data stored in the storage unit 242, for example. The number-of-stores estimating unit 270 estimates the number of customers who visited a particular store during a particular period, for example, by multiplying the calculated number of communication terminal users by an appropriate coefficient. As described above, the management server 160, instead of the number-of-stores estimating unit 270, or together with the number-of-stores estimating unit 270, responds to requests from the communication terminal 108 by A passerby number estimation unit (not shown) for estimating the number of people may be provided.

FIG. 3 schematically shows an example of the internal configuration of the storage unit 242. As shown in FIG. AP information storage unit 322, store information storage unit 324, correspondence information storage unit 326, log information storage unit 332, store visit information storage unit 334, input data storage unit 342, data set storage unit 352, target A data storage unit 354 , an output data storage unit 362 and a model information storage unit 372 are provided.

In this embodiment, the AP information storage unit 322 stores information regarding each of one or more access points. The AP information storage unit 322 stores, for example, the identification information of each access point and the identification information of the store where each access point is arranged in association with each other. Examples of the identification information of each access point include identification information (sometimes referred to as APID) assigned to each access point by the management server 160, MAC address of each access point, and the like.

In this embodiment, the shop information storage unit 324 stores information about each of one or more shops. The store information storage unit 324 stores identification information of each store, location information of each store, information indicating attributes of each store, and entry restriction information of each store, for example, in association with each other. Examples of the identification information of each store include identification information (sometimes referred to as a store ID) assigned to each store by the management server 160, the name of each store, the address of each store, and the like.

In this embodiment, the correspondence information storage unit 326 stores one or more pieces of correspondence information. In this embodiment, the log information storage unit 332 stores one or more pieces of log information. In this embodiment, the store visit information storage unit 334 stores one or more pieces of store visit confirmation information. In this embodiment, the input data storage unit 342 stores one or more pieces of input data. In this embodiment, the dataset storage unit 352 stores one or more learning datasets. The data set storage unit 352 may store learning data sets for each store. The dataset storage unit 352 may store a learning dataset for each access point.

In this embodiment, the target data storage unit 354 stores one or more determination target data. In this embodiment, the output data storage unit 362 stores one or more output data. In this embodiment, the model information storage unit 372 stores information about each of one or more learning models. The model information storage unit 372 may store information on learning models for each store. The model information storage unit 372 may store information regarding learning models for each access point. The information about the learning model includes, for example, an inference model program, information indicating parameter values of the inference model, and information indicating the adjusted threshold value determined by the threshold adjustment unit 256 .

FIG. 4 schematically shows an example of information processing in the input data generator 244. As shown in FIG. Using FIG. 4, as an example, a case where a plurality of pieces of log information related to radio signals transmitted by the communication terminal 102 among the radio signals received by the access point 122 in a specific unit period are aggregated into a single input data. An example of aggregation processing in the input data generator 244 will be described. In FIG. 4 , triangular marks indicate timings at which access point 122 receives a wireless signal transmitted by communication terminal 102 and having a radio signal strength greater than a predetermined strength.

As shown in FIG. 4, among the radio signals transmitted by communication terminal 102 in a specific unit period, n pieces of log information related to radio signals detected by access point 122 are classified into groups 422 and 424. obtain. n may be an integer of 2 or greater. For example, group 422 includes ma pieces of log information. Also, the group 424 includes mb pieces of log information. ma may be an integer of 1 or more, or an integer of 2 or more. mb may be an integer of 1 or more, or an integer of 2 or more. The number of groups included in the unit period indicates the number of visits during the period of stay.

For example, the input data generation unit 244 sorts a plurality of pieces of log information related to wireless signals detected at the access point 122 in a specific unit period, using reception time indicated by each piece of log information as a key. If the difference between the reception times of two temporally adjacent pieces of log information is equal to or smaller than a predetermined value, the input data generator 244 determines that the two pieces of log information belong to the same group. do. On the other hand, if the difference between the reception times of two temporally adjacent pieces of log information is greater than or equal to a predetermined value, the input data generation unit 244 determines that the two pieces of log information belong to different groups. to decide.

When the ma pieces of log information included in the group 422 are arranged according to the reception time indicated by each piece of log information, the difference ΔTa between the reception times of two pieces of log information temporally adjacent to each other is the above predetermined value. equal to or less than that value. Similarly, when mb pieces of log information included in group 424 are arranged according to the reception time indicated by each piece of log information, the difference ΔTb between the reception times of two temporally adjacent pieces of log information is the above predetermined is equal to or less than the specified value. On the other hand, the difference ΔTab between the reception time indicated by the last log information detected among the log information included in the group 422 and the reception time indicated by the first detected log information among the log information included in the group 424 is greater than or equal to the above predetermined value.

For example, the input data generation unit 244 aggregates a part of n pieces of log information to generate 1 to n−1 pieces of input data. The input data generation unit 244 may store the generated one or more input data in the storage unit 242 . In one embodiment, the input data generator 244 aggregates multiple pieces of log information into single input data for each unit period. In another embodiment, the input data generator 244 aggregates multiple pieces of log information into single input data for each duration (ie, for each group).

More specifically, the input data generator 244 first classifies a plurality of pieces of collected log information for each communication terminal indicated by each piece of log information. As a result, for example, n pieces of log information regarding radio signals transmitted by the communication terminal 102 within a specific unit period can be obtained.

Next, the input data generation unit 244 aggregates the log information classified for each communication terminal for each arbitrary period based on the reception time indicated by the reception time information of each log information. A unit period, a duration period, and the like are exemplified as the arbitrary period.

For example, the input data generation unit 244 performs arbitrary statistical processing on various physical quantities related to radio waves of radio signals indicated by signal information included in one or more pieces of log information detected during each period. , the feature amount of one or more pieces of log information detected during each period is calculated. The input data generation unit 244 converts representative values of various physical quantities related to radio waves of radio signals indicated by signal information included in one or more pieces of log information detected during each period to one or more pieces of log information detected during each period. It may be a feature amount of log information.

The input data generation unit 244 associates, for example, the terminal identification information of the communication terminal related to one or more pieces of log information detected during each period, the information indicating the period to be aggregated, and the feature amount. to generate the input data. Thereby, one or more pieces of log information detected during each period are aggregated into single input data.

FIG. 5 schematically shows an example of the internal configuration of the dataset generator 252. As shown in FIG. In this embodiment, the dataset generation unit 252 includes a label assignment unit 552 , a dataset output unit 554 and a target data output unit 556 .

In this embodiment, the label assigning unit 552 assigns a positive or negative label to part of the log information acquired by the information collecting unit 230 . The label assigning unit 552 may assign a positive or negative label to part of the input data generated by the input data generating unit 244 . The details of the procedure for labeling a portion of log information or input data may be as described above.

In this embodiment, the dataset output unit 554 outputs log information or input data labeled by the labeling unit 552 as a learning dataset. The dataset output unit 554 may store the learning dataset in the dataset storage unit 352 .

In this embodiment, the target data output unit 556 outputs log information or input data to which no label has been assigned by the label assigning unit 552 as determination target data. The target data output unit 556 may store the determination target data in the target data storage unit 354 .

FIG. 6 schematically shows an example of the internal configuration of the learning section 254. As shown in FIG. In this embodiment, the learning unit 254 comprises a class classifier 652 , a class classifier 654 and a default setting storage unit 656 .

In this embodiment, classifier 652 is the learning model for store 12 and classifier 654 is the learning model for store 14 . In this embodiment, the initial setting storage unit 656 stores initial settings for machine learning of the classifiers 652 and 654 . The initial setting storage unit 656 may store information regarding the initial value of the threshold for classification of each classifier as the initial setting. The learning unit 254 uses the initial settings stored in the initial setting storage unit 656 and the learning data set stored in the data set storage unit 352 to perform machine learning of the class classifiers 652 and 654. Run.

When the classifier 652 and the classifier 654 calculate the classification probability regarding the determination result, compare the classification probability and the threshold for classification, and determine whether or not the user exists inside the store or the like. , the initial value of the threshold for classification is 0.5. When the classifier 652 and the classifier 654 calculate a decision function regarding the determination result, compare the decision function and the threshold for classification, and determine whether or not the user was present inside a store or the like , the initial value of the threshold for classification is zero.

FIG. 7 schematically shows an example of the internal configuration of the threshold adjuster 256. As shown in FIG. In this embodiment, the threshold adjuster 256 includes an ROC curve calculator 752 , an AUC score calculator 754 , an actuation point determiner 756 and an adjustment value determiner 758 .

In this embodiment, the ROC curve calculator 752 calculates data indicating the receiver operating characteristic curve (ROC curve) of the learning model generated by the learning unit 254 . The procedure for calculating the data representing the ROC curve may be as described above. In this embodiment, the AUC score calculator 754 calculates an AUC score indicating the area of the area under the ROC curve (AUC).

In this embodiment, the true positive rate of the determination result of the learning model increases compared to the case where the operating point determination unit 756 and the initial value are used as the threshold for classification, which is the point on the ROC curve. A point is determined as the actuation point of the learning model. Details of the procedure for determining the actuation point will be described later.

In this embodiment, the adjustment value determination unit 758 determines a value used as the threshold after adjustment. The adjustment value determiner 758 may determine the threshold corresponding to the actuation point determined by the actuation point determiner 756 as the adjusted threshold.

FIG. 8 schematically shows an example of information processing in the threshold adjuster 256. As shown in FIG. FIG. 8 schematically shows an example of a process in which the actuation point determination section 756 determines an actuation point. In FIG. 8, the ROC curve 820 indicates the ROC curve of the learning model learned using the learning data set generated based on the store visit confirmation information. Point 832 shows the true positive rate and false positive rate when the initial value is used as the threshold for classification. Points 834 are (i) the ROC curve 820, (ii) a point with a true positive rate of 1 and a false positive rate of 0 (sometimes referred to as the best point), and a true positive rate of 0. and a straight line 840 connecting points with a false positive rate of 1.

In this embodiment, actuation point determiner 756 determines point 834 as the actuation point. Here, when the learning process is executed using the learning data set generated based on the store visit confirmation information, the learned learning model may be, for example, "visit the store 12 and use the store visit confirmation device 142 to visit the store. are positive, and "data points related to users who have visited the store 12 but have not been confirmed by the store visit confirmation device 142" and "data points related to users who have not visited the store 12" are classified as positive. data points” to be classified as negative.

Therefore, in the operation points, compared with the case where the initial value is used as the threshold for classification, "the data point related to the user who visited the store 12 and was confirmed by the store visit confirmation device 142" is positive, The true positive rate increases when "data points related to users who have visited the store 12 but are not confirmed to have visited the store by the store visit confirmation device 142" and "data points related to users who have not visited the store 12" are negative. . In addition, compared with the case where the initial value is used as the threshold for classification, "the data point regarding the user who visited the store 12 and was confirmed by the store visit confirmation device 142" is positive, and "the store 12 is visited. However, the false-positive rate increases when "data points regarding users whose visits have not been confirmed by the visit confirmation device 142" and "data points regarding users who have not visited the store 12" are negative.

However, in the determination unit 260, for example, "data points related to users who exist inside the store 12" are positive, and "data points related to users who do not exist inside the store 12" are negative. Processing is performed. In other words, the determination unit 260 determines, for example, “data points regarding a user who visited the store 12 and was confirmed by the store visit confirmation device 142” and “a user who visited the store 12 but was confirmed by the store visit confirmation device 142. Determination processing using a learning model is performed by setting a positive data point for a user who has not visited the store 12 and a negative data point for a user who has not visited the store 12 .

Here, the behavior of "a user who has visited the store 12 but whose visit has not been confirmed by the store visit confirmation device 142" is more likely than that of a "user who has not visited the store 12". There is a high possibility that the behavior is similar to that of "a user whose visit has been confirmed by the visit confirmation device 142". Therefore, by using the threshold value corresponding to the actuation point determined by the actuation point determination unit 756 as the threshold value for classification in the determination unit 260, compared to the case where the initial value is used as the threshold value for classification, It is expected that the true positive rate will increase when "data points related to users existing inside the store 12" are positive and "data points related to users not present inside the store 12" are negative. In addition, compared to the case where the initial value is used as the threshold for classification, "data points related to users existing inside the store 12" are positive, and "data points related to users not existing inside the store 12" are positive. It is expected that the false-positive rate in the case of negative results will be reduced.

Note that the method of determining the actuation point is not limited to this embodiment. In another embodiment, operating point determiner 756 calculates the distance between the best point and each point on ROC curve 820 . The actuation point determination unit 756 determines the point with the shortest distance as the actuation point.

A specific example of information processing in the management server 160 will be described with reference to FIGS. 9 to 20. FIG. 9 to 17 schematically show examples of various data tables stored in the storage unit 242. FIG. FIG. 18 schematically shows an example of information processing in the analysis unit 240. As shown in FIG. Specifically, FIG. 9 schematically shows an example of a learning data set generation process, a learning model generation process, and a classification threshold adjustment process in the analysis unit 240 . FIG. 19 schematically shows an example of information processing in the analysis unit 240. As shown in FIG. FIG. 19 schematically shows an example of determination processing in the analysis unit 240. As shown in FIG. FIG. 20 schematically shows an example of a data table 1800 in which judgment results are recorded.

FIG. 9 schematically shows an example of a data table 900. As shown in FIG. The data table 900 may be an example of a database stored in the AP information storage unit 322. FIG. Each row of data table 900 may be an example of information about each access point. In this embodiment, the data table 900 stores APID 922 of each access point, information 924 indicating the MAC address of each access point, and store ID 926 of the store where each access point is arranged, in association with each other. The APID 922, the information 924 indicating the MAC address, and the shop ID 926 may be examples of access point identification information.

FIG. 10 schematically shows an example of the data table 1000. As shown in FIG. Data table 1000 may be an example of a database stored in store information storage unit 324 . Each row of the data table 1000 may be an example of information about each store. In this embodiment, the data table 1000 includes store ID 1022 of each store, store name 1024 of each store, store type 1026 of each store, location information 1028 of each store, and information 1030 indicating business hours of each store. and are stored in association with each other. Items recorded in the store type 1026 include, for example, a store within a store, a single store in an urban area or a store in a shopping district, and a single roadside store. Store type 1026 may be an example of a store attribute. The information 1030 indicating business hours may be an example of entry restriction information.

FIG. 11 schematically shows an example of a data table 1100. As shown in FIG. The data table 1100 may be an example of a database stored in the correspondence information storage unit 326. FIG. Each row of the data table 1100 may be an example of correspondence information. In this embodiment, the data table 1100 stores the terminal ID 1122 of each communication terminal and the user ID 1124 of the user of each communication terminal in association with each other. Examples of the user ID 1124 include a credit card number, a membership number for a point-giving service, an SNS account, and the like.

FIG. 12 schematically shows an example of a data table 1200. As shown in FIG. The data table 1200 may be an example of a database stored in the log information storage unit 332. FIG. Each row of data table 1200 may be an example of log information. In this embodiment, the data table 1200 includes the APID 1222 of the access point that received the wireless signal, information 1224 indicating the reception time of the wireless signal, the terminal ID 1226 of the communication terminal that transmitted the wireless signal, and information about the radio wave of the wireless signal. 1228 are stored in association with each other. Examples of the information 1228 on the radio wave of the radio signal include information indicating radio wave intensity, information indicating the frequency band, information indicating the connection stage, and the like. The radio wave intensity is represented by RSSI, for example.

FIG. 13 schematically shows an example of a data table 1300. As shown in FIG. The data table 1300 may be an example of a database stored in the visit information storage unit 334, for example. Each row of the data table 1300 may be an example of store visit confirmation information. In this embodiment, the data table 1300 stores the user ID 1322 of the user, the position information 1324 of the location where the user's behavior was confirmed, and the information 1326 indicating the time when the user's behavior was confirmed in association with each other. The user ID 1322 may include information indicating the type of user ID and information indicating details of the user ID. The details of the user ID may be a specific number string or character string such as a credit card number, a membership number for a point-giving service, the name of the user, or the MAC address of the communication terminal used by the user. Examples of the location information 1324 of the location where the user's behavior was confirmed include the name of the location where the user's behavior was confirmed, the address of the location, the location coordinates of the location, and the like.

FIG. 14 schematically shows an example of a data table 1400. As shown in FIG. The data table 1400 may be an example of a database stored in the input data storage unit 342, for example. Each row of data table 1400 may be an example of input data. In this embodiment, the data table 1400 includes a terminal ID 1422 of the communication terminal that transmitted the wireless signal, a store ID 1424 of the store where the access point that received the wireless signal is arranged, information 1426 indicating the reception date of the wireless signal, Information 1428 indicating the feature quantity of the radio signal is stored in association with the information 1428 . The radio signal feature amount may include a plurality of items. The store ID 1424 is an example of identification information of a store and may be an example of identification information of an access point. The date of reception may be an example of a unit period in which the radio signal was received.

FIG. 15 schematically shows an example of a data table 1500. As shown in FIG. Data table 1500 may be an example of a database stored in dataset storage unit 352 . Data table 1500 may be an example of a training data set. Each row of data table 1500 may be an example data point for training. In FIG. 15, items similar to those in the data table 1400 are denoted by the same reference numerals as those in the data table 1400, and description thereof may be omitted.

In this embodiment, the data table 1500 includes a terminal ID 1422 of the communication terminal that transmitted the wireless signal, a store ID 1424 of the store where the access point that received the wireless signal is arranged, information 1426 indicating the reception date of the wireless signal, Information 1428 indicating the feature quantity of the radio signal and information 1532 indicating the type of label are associated and stored. Examples of label types include "Visit" indicating that the user exists within the area of the store, and "Non-visit" indicating that the user exists outside the area of the store.

FIG. 16 schematically shows an example of a data table 1600. As shown in FIG. The data table 1600 may be an example of a database stored in the target data storage unit 354. FIG. Each row of the data table 1600 may be an example of determination target data. In FIG. 16, items similar to those in the data table 1400 are denoted by the same reference numerals as those in the data table 1400, and description thereof may be omitted.

In this embodiment, the data table 1600 includes a terminal ID 1422 of the communication terminal that transmitted the wireless signal, a store ID 1424 of the store where the access point that received the wireless signal is arranged, information 1426 indicating the reception date of the wireless signal, Information 1428 indicating the feature quantity of the radio signal, information 1632 indicating the determination result, and information 1634 indicating the classification probability or the value of the decision function are stored in association with each other. Since the determination target data is data used in determination processing in the determination unit 260, in this embodiment, the columns of the information 1632 indicating the determination result and the information 1634 indicating the classification probability or the value of the decision function are blank. ing.

FIG. 17 schematically shows an example of a data table 1700. As shown in FIG. The data table 1700 may be an example of a database stored in the output data storage unit 362, for example. Each row of data table 1700 may be an example of output data. In FIG. 17, items similar to data table 1400, data table 1500, or data table 1600 are denoted by the same reference numerals as data table 1400, data table 1500, or data table 1600, and description thereof may be omitted. be.

In this embodiment, the data table 1700 includes a terminal ID 1422 of the communication terminal that transmitted the wireless signal, a store ID 1424 of the store where the access point that received the wireless signal is arranged, information 1426 indicating the reception date of the wireless signal, Information 1532 indicating the label type and information 1632 indicating the determination result are stored in association with each other. In this embodiment, the data table 1700 stores only information corresponding to the learning data set generated by the data set generation unit 252, and does not yet store information about the determination result of the determination target data.

FIG. 18 schematically shows an example of a learning data set generation process, a learning model generation process, and a classification threshold adjustment process in the analysis unit 240 . According to the present embodiment, first, the input data generation unit 244 aggregates at least part of the plurality of pieces of log information stored in the log information storage unit 332 to generate one or more pieces of input data.

The input data generation unit 244 refers to at least one of information about each access point stored in the AP information storage unit 322 and information about each store stored in the store information storage unit 324, and generates one or more input data may be generated. The input data generator 244 stores the generated input data in the input data storage 342 .

Specifically, input data generator 244 first stores a plurality of pieces of log information contained in data table 1200 stored in log information storage 332, using terminal ID 1226 of the communication terminal that transmitted the radio signal as a first key. , using information 1224 indicating the reception time of the radio signal as the second key. Next, the input data generation unit 244 classifies the plurality of pieces of log information regarding each communication terminal for each unit period. The input data generator 244 aggregates one or more pieces of log information classified by unit period into single input data. This generates input data.

For example, the input data generator 244 calculates a feature amount of one or more pieces of log information detected during each period. As described with reference to FIG. 4, the input data generation unit 244, for various physical quantities related to radio waves of radio signals indicated by signal information included in one or more pieces of log information detected during each period, You may calculate said feature-value by implementing arbitrary statistical processing. The input data generator 244 may determine a representative value of physical quantities indicated by a plurality of pieces of log information as the feature quantity.

The input data generator 244 stores the generated input data in the data table 1400. FIG. For example, the input data generation unit 244 enters the terminal ID 1226 of one or more pieces of log information aggregated into the input data as the terminal ID 1422 of the input data. The input data generation unit 244 refers to the data table 900 and acquires the store ID 926 corresponding to the APID 1222 of one or more pieces of log information aggregated in the input data. The input data generation unit 244 enters the store ID 926 as the store ID 1424 of the input data. The input data generation unit 244 enters information indicating the unit period that is the aggregation period of the input data as the information 1426 indicating the reception date of the input data. The input data generation unit 244 enters, as the information 1428 indicating the feature amount of the input data, the statistical value or representative value of the physical quantity indicated by one or more pieces of log information aggregated in the input data.

Next, the data set generation unit 252 generates a learning data set by giving a positive or negative label to at least part of the input data stored in the input data storage unit 342 . The data set generation unit 252 may refer to the store visit confirmation information stored in the store visit information storage unit 334 and assign a positive or negative label to at least part of the input data. The data set generation unit 252 refers to the store visit confirmation information stored in the store visit information storage unit 334 and the correspondence information stored in the correspondence information storage unit 326, and determines whether at least a part of the input data is positive or negative. may be labeled.

The data set generation unit 252 may generate a learning data set by performing a process of assigning only positive labels. The data set generation unit 252 may generate a learning data set by executing a process of assigning only negative labels. The data set generation unit 252 may generate a learning data set by performing a process of assigning positive or negative labels. The data set generation unit 252 stores the generated learning data set in the data set storage unit 352 .

The data set generation unit 252 may output, among the input data stored in the input data storage unit 342, the input data to which the positive or negative label has not been assigned as the determination target data. The data set generation section 252 may store the determination target data in the target data storage section 354 .

The dataset generator 252 may output the learning dataset as part of the output data. The dataset generator 252 may store the above output data in the output data storage 362 .

Specifically, the data set generation unit 252 compares, for example, one or more input data included in the data table 1400 with one or more store visit confirmation information included in the data table 1300 to obtain one or more input data. is given a positive label to at least part of the data set for learning stored in the data table 1500 . In the present embodiment, the user ID 1322 of the store visit confirmation information is the MAC address of the communication terminal 102 obtained by communication between the store visit confirmation device 142 installed at the shop front of the store 12 and the communication terminal 102 of the user 22, The location information 1324 of the place where the behavior of the user in the store visit confirmation information was confirmed is the store name of the store 12, and the information 1326 indicating the time when the user behavior in the store visit confirmation information was confirmed is the MAC address obtained. The details of the process by which the data set generation unit 252 generates the data points forming the learning data set will be described with reference to the case where the time is the time of the day.

The data set generation unit 252 first reads one piece of input data that constitutes the data table 1400 stored in the input data storage unit 342 . Next, using the terminal ID 1422 included in the read input data as a key, the data set generation unit 252 selects one or more items of store visit confirmation information from the data table 1300 stored in the store visit information storage unit 334. Store visit confirmation information whose ID 1322 matches the key is extracted. Using the information 1426 indicating the reception date included in the read input data as a key, the data set generation unit 252 extracts the store visit confirmation information acquired during the unit period indicated by the key from among the extracted store visit confirmation information. to extract This associates the input data with the visit confirmation information.

When the store-visit confirmation information is extracted, the data set generation unit 252 assigns a positive label to the input data to generate data points forming a learning data set. Specifically, the data set generation unit 252 extracts the terminal ID 1422, the store ID 1424, the information 1426 indicating the reception date, the information 1428 indicating the characteristic amount, and the information 1532 indicating the label type included in the read input data. are associated with each other and stored in the data table 1500 . A label indicating positive is entered in the information 1532 indicating the type of label.

In this case, the data set generation unit 252 associates the information 1426 indicating the terminal ID 1422, the store ID 1424, and the reception date included in the read input data with the information 1532 indicating the label type, and outputs the output data. may be stored in the data table 1700 as part of the A label indicating positive is entered in the information 1532 indicating the type of label. In addition, since the determination unit 260 does not need to determine the labeled data points, the information 1732 indicating the determination result may include information indicating that determination processing is unnecessary.

On the other hand, if the store-visit confirmation information is not extracted, the data set generation unit 252 adds columns used in the determination processing of the determination unit 260 to the input data to generate determination target data. Specifically, the data set generation unit 252 converts the terminal ID 1422, the store ID 1424, the information 1426 indicating the reception date, and the information 1428 indicating the feature amount included in the read input data, and the determination result of the determination unit 260. A column into which information 1732 indicating is input is associated with a column in which information 1734 indicating the classification probability or the value of the decision function calculated by the model in the determination processing of the determination unit 260 is input, and stored in the data table 1600 .

Next, the learning unit 254 uses the learning data set stored in the data set storage unit 352 and the initial value of the threshold for classification stored in the initial setting storage unit 656 to perform the learned learning. Generate a model. The learning unit 254 stores information about the learned learning model in the model information storage unit 372 .

Next, the threshold adjuster 256 adjusts the threshold for classification. The threshold adjuster 256 determines the adjusted threshold. The threshold adjuster 256 stores information indicating the adjusted threshold in the model information storage 372 .

According to the present embodiment, the log information about the radio signal and the store visit confirmation information are associated to generate a labeled learning data set. As a result, it is possible to relatively easily build a learning model used for determining whether or not a user carrying a wireless terminal that transmits a wireless signal exists inside a specific store or the like.

Regarding this point, it is possible to determine whether a user carrying a wireless terminal that transmits a wireless signal exists inside a specific store, etc. by determining whether the user enters or leaves the store, etc., using log information related to wireless signals. When determining whether the The reception status of radio waves at the access points placed inside the store, etc. depends on (i) the location conditions of the store, etc., (ii) the structure of the store, etc., (iii) the arrangement of the access points inside the store, etc., and (iv) It is affected by the position, size and material of structures or obstacles or people placed around the access point.

Therefore, depending on the situation of the store or the like, it may be difficult to adopt a rule-based approach to determine entry or exit into the store or the like. On the other hand, when adopting an approach using machine learning, a large amount of learning data sets are required to be used as at least one of machine learning training data, verification data, and test data. As described above, according to the present embodiment, a large amount of learning data sets can be prepared, so it is possible to execute the determination process using machine learning.

FIG. 19 schematically shows an example of information processing in the analysis unit 240. As shown in FIG. FIG. 19 schematically shows an example of determination processing in the analysis unit 240. As shown in FIG. According to the present embodiment, first, the determination unit 260 acquires the learned learning model program stored in the model information storage unit 372 and the information indicating the adjusted threshold.

Next, the determination unit 260 reads one determination target data from the data table 1600 stored in the target data storage unit 354 . The determination unit 260 inputs the read determination target data to the learning model. The learning model uses the adjusted threshold value to output a determination result indicating whether or not the user corresponding to the determination target data exists inside the store.

For example, the learning model calculates a classification probability value for the read determination target data. For example, if the value of the classification probability is greater than the adjusted threshold, the learning model outputs a positive determination result. On the other hand, when the value of the classification probability is equal to or less than the adjusted threshold value, the learning model outputs a negative judgment result. Specifically, the determination unit 260 associates the terminal ID 1422, the store ID 1424, and the information 1426 indicating the reception date included in the determination target data with the information 1632 indicating the determination result of the determination unit 260, and stores the information in the data table. 1700.

According to this embodiment, as the classification threshold used by the learning model, the adjusted value determined using the ROC curve is used instead of the initial value determined according to the learning model. As a result, compared to the case where the initial value is used as the threshold for classification, "data points related to users existing inside the store 12" are positive, and "data points related to users not present inside the store 12" are positive. is expected to increase the true positive rate when In addition, compared to the case where the initial value is used as the threshold for classification, "data points related to users existing inside the store 12" are positive, and "data points related to users not existing inside the store 12" are positive. It is expected that the false-positive rate in the case of negative results will be reduced.

FIG. 20 schematically shows an example of a data table 1700 in which judgment results are recorded. A data table 1700 shown in FIG. 20 differs from the data table 1700 shown in FIG. 18 in that a row regarding determination target data is added. In the row related to the determination target data, the determination result output by the determination unit 260 is entered in the column of information 1632 indicating the determination result. On the other hand, in the information 1532 indicating the type of label, information indicating that no label is attached is entered.

FIG. 21 illustrates an example computer 3000 upon which aspects of the present invention may be embodied, in whole or in part. For example, management server 160 is realized by computer 3000 .

Programs installed on the computer 3000 cause the computer 3000 to function as one or more "parts" of an operation or one or more "parts" of an apparatus according to embodiments of the invention, or to and/or cause the computer 3000 to perform processes or steps of processes according to embodiments of the present invention. Such programs may be executed by CPU 3012 to cause computer 3000 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

Computer 3000 according to this embodiment includes CPU 3012 , RAM 3014 , graphics controller 3016 , and display device 3018 , which are interconnected by host controller 3010 . Computer 3000 also includes input/output units such as communication interface 3022 , hard disk drive 3024 , DVD-ROM drive 3026 and IC card drive, which are connected to host controller 3010 via input/output controller 3020 . The computer also includes legacy input/output units such as ROM 3030 and keyboard 3042 , which are connected to input/output controller 3020 via input/output chip 3040 .

The CPU 3012 operates according to programs stored in the ROM 3030 and RAM 3014, thereby controlling each unit. Graphics controller 3016 retrieves image data generated by CPU 3012 into a frame buffer or the like provided in RAM 3014 or itself, and causes the image data to be displayed on display device 3018 .

Communication interface 3022 communicates with other electronic devices over a network. Hard disk drive 3024 stores programs and data used by CPU 3012 within computer 3000 . DVD-ROM drive 3026 reads programs or data from DVD-ROM 3001 and provides programs or data to hard disk drive 3024 via RAM 3014 . The IC card drive reads programs and data from IC cards and/or writes programs and data to IC cards.

ROM 3030 stores therein programs that are dependent on the hardware of computer 3000, such as a boot program that is executed by computer 3000 upon activation. Input/output chip 3040 may also connect various input/output units to input/output controller 3020 via parallel ports, serial ports, keyboard ports, mouse ports, and the like.

A program is provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The program is read from a computer-readable storage medium, installed in hard disk drive 3024 , RAM 3014 , or ROM 3030 , which are also examples of computer-readable storage media, and executed by CPU 3012 . The information processing described within these programs is read by computer 3000 to provide coordination between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information operations or processing according to the use of computer 3000 .

For example, when communication is performed between the computer 3000 and an external device, the CPU 3012 executes a communication program loaded into the RAM 3014 and sends communication processing to the communication interface 3022 based on the processing described in the communication program. you can command. Under the control of the CPU 3012, the communication interface 3022 reads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 3014, the hard disk drive 3024, the DVD-ROM 3001, or an IC card, and transmits the read transmission data. Data is transmitted to the network, or received data received from the network is written in a receive buffer area or the like provided on the recording medium.

In addition, the CPU 3012 causes the RAM 3014 to read all or necessary portions of files or databases stored in external recording media such as the hard disk drive 3024, DVD-ROM drive 3026 (DVD-ROM 3001), IC card, etc. Various types of processing may be performed on the data in RAM 3014 . CPU 3012 may then write back the processed data to an external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on recording media and subjected to information processing. CPU 3012 performs various types of operations on data read from RAM 3014, information processing, conditional decisions, conditional branching, unconditional branching, and information retrieval as specified throughout this disclosure and by instruction sequences of programs. Various types of processing may be performed, including /replace, etc., and the results written back to RAM 3014 . Also, the CPU 3012 may search for information in a file in a recording medium, a database, or the like. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 selects the first attribute from among the plurality of entries. search for an entry that matches the specified condition of the attribute value of the attribute, read the attribute value of the second attribute stored in the entry, and thereby determine the first attribute that satisfies the predetermined condition An attribute value of the associated second attribute may be obtained.

The programs or software modules described above may be stored in a computer readable storage medium on or near computer 3000 . In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the above program can be transferred via a network. provided to the computer 3000;

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the scope of claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The execution order of each process such as actions, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is particularly "before", "before" etc., and it should be noted that it can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if the description is made using "first," "next," etc. for convenience, it means that it is essential to carry out in this order. not a thing

10 communication network 12 store 14 store 22 user 24 user 100 data analysis system 102 communication terminal 104 communication terminal 108 communication terminal 122 access point 124 access point 142 visit confirmation device 144 visit confirmation device 160 management server 220 input/output control unit 230 information collection unit 240 analysis unit 242 storage unit 244 input data generation unit 250 model construction unit 252 data set generation unit 254 learning unit 256 threshold adjustment unit 260 determination unit 270 number of visitors estimation unit 322 AP information storage unit 324 store information storage unit 326 corresponding information storage unit 332 Log information storage unit 334 Store visit information storage unit 342 Input data storage unit 352 Data set storage unit 354 Target data storage unit 362 Output data storage unit 372 Model information storage unit 422 Group 424 Group 552 Labeling unit 554 Data set output unit 556 Target Data output unit 652 Class classifier 654 Class classifier 656 Initial setting storage unit 752 ROC curve calculation unit 754 AUC score calculation unit 756 Operating point determination unit 758 Adjustment value determination unit 820 ROC curve 832 Point 834 Point 840 Straight line 900 Data table 922 APID
924 Information 926 Store ID
1000 Data table 1022 Store ID
1024 Store name 1026 Store type 1028 Location information 1030 Information 1100 Data table 1122 Terminal ID
1124 User ID
1200 data table 1222 APID
1224 Information 1226 Terminal ID
1228 information 1300 data table 1322 user ID
1324 Location information 1326 Information 1400 Data table 1422 Terminal ID
1424 Store ID
1426 information 1428 information 1500 data table 1532 information 1600 data table 1632 information 1634 information 1700 data table 1732 information 1734 information 1800 data table 3000 computer 3001 DVD-ROM
3010 host controller 3012 CPU
3014 RAM
3016 graphic controller 3018 display device 3020 input/output controller 3022 communication interface 3024 hard disk drive 3026 DVD-ROM drive 3030 ROM
3040 input/output chip 3042 keyboard

Claims (15)

A data point containing information relating to a radio signal received by a radio receiver located within an area having a predetermined geographic extent is entered such that a user of a radio terminal that originated the radio signal is located within the area. a model generation unit that generates a learning model used to determine whether or not there was a
an adjustment unit that adjusts a reference value for determining whether the learning model generated by the model generation unit determines whether the user exists inside the area;
with
The model generation unit
(a) using an initial value according to the learning model as the reference value, (b) (i) information about each of the plurality of radio signals received by one or more of the radio receiving devices, and ) a label indicating whether or not the user of the wireless terminal that transmitted each of the plurality of wireless signals was present within the area in which the wireless receiving device that received each of the plurality of wireless signals was arranged; Generate the learning model by performing machine learning using a learning data set containing
The adjustment unit
A second value that increases the true positive rate of the determination result of the learning model generated by the model generating unit as compared to the case where the initial value is used as the reference value is the adjusted reference value determine as
Machine learning device. (i) terminal identification information for identifying the wireless terminal that transmitted each of the plurality of wireless signals; (ii) reception time information indicating the time at which each of the plurality of wireless signals was received; , (iii) a record acquisition unit that acquires a plurality of records associated with signal information about radio waves of each of the plurality of radio signals;
Each of them is associated with (i) user identification information for identifying each of the one or more users involved in a specific action, and (ii) action time information indicating the time when the specific action was performed. an action history acquisition unit that acquires a plurality of action history information;
(i) the terminal identification information and the reception time information included in each of the plurality of records acquired by the record acquisition unit; and (ii) each of the plurality of action history information acquired by the action history acquisition unit. The plurality of records and the plurality of action history information are collated based on the user identification information and the action time information included in the terminal identification included in each of at least some of the plurality of records information, information about radio waves indicated by the signal information included in each of the at least some records, and the label indicating that the user was present in the area, thereby performing the learning. a labeling unit that generates a dataset for
further comprising
The machine learning device according to claim 1. The user identification information included in at least part of the plurality of action history information is the terminal identification information of a wireless terminal used by the user identified by the user identification information.
The machine learning device according to claim 2. further comprising a correspondence acquisition unit that acquires correspondence relationship information indicating a correspondence relationship between each of the one or more pieces of user identification information and the terminal identification information of a wireless terminal used by a user identified by the user identification information,
The labeling unit is
With reference to the correspondence information acquired by the correspondence acquisition unit, (i) the terminal identification information and the reception time information included in each of the plurality of records acquired by the record acquisition unit, and (ii) ) based on the user identification information and the action time information included in each of the plurality of action history information acquired by the action history acquiring unit, the plurality of records and the plurality of action history information are collated, and the learning generate a dataset for
The machine learning device according to claim 2. In each, (i) terminal identification information for identifying the wireless terminal that transmitted each of the plurality of wireless signals, (ii) reception time information indicating the time each of the plurality of wireless signals was received, ( iii) receiving position information indicating the position at which each of the plurality of radio signals was received; and (iv) a record for acquiring a plurality of records associated with signal information regarding radio waves of each of the plurality of radio signals. an acquisition unit;
(i) area identification information for identifying each of the one or more areas; a restriction information acquisition unit that acquires access restriction information;
(i) the reception position information included in each of the plurality of records acquired by the record acquisition unit; and (ii) the information included in each of the one or more entry restriction information acquired by the restriction information acquisition unit. Based on the area identification information, the plurality of records and the one or more pieces of entry restriction information are collated, and the terminal identification information included in each of at least a portion of the plurality of records and the at least a portion of the A label for generating the learning data set by associating information on radio waves indicated by the signal information included in each record with the label indicating that the user was not inside the area. a granting unit;
further comprising
The machine learning device according to claim 1. The plurality of records acquired by the record acquisition unit is based on the terminal identification information and the reception time information included in each of the plurality of records, and each of the terminal identification information satisfies a predetermined condition. further comprising an aggregating unit that aggregates for each period,
The aggregating unit is
In each, (i) terminal identification information for identifying the wireless terminal that transmitted each of the plurality of wireless signals, (ii) aggregation period information indicating a period subject to aggregation processing by the aggregation unit, ( iii) feature information indicating one or more feature amounts relating to one or more radio signals transmitted during the period indicated by the aggregation period information by the radio terminal identified by the terminal identification information among the plurality of radio signals; generate one or more aggregated records associated with
The machine learning device according to any one of claims 2 to 5. The labeling unit generates the learning data set by giving the label to at least part of the one or more aggregated records.
The machine learning device according to claim 6. The learning model calculates, for each data point input to the learning model, the value of a function used to estimate the uncertainty of the prediction or decision of the learning model;
Determining whether the user exists within the area based on the calculated value of the function and the reference value;
The machine learning device according to any one of claims 1 to 7. The adjustment unit
calculating data indicating a receiver operating characteristic curve (ROC curve) of the learning model generated by the model generating unit;
A point on the ROC curve at which the true positive rate of the determination result of the learning model generated by the model generation unit is greater than when the initial value is used as the reference value determining a corresponding reference value as the second value;
The machine learning device according to any one of claims 1 to 8. The model generation unit generates the learning model for each wireless receiving device or each area.
The machine learning device according to any one of claims 1 to 9. The wireless receiving device is a relay device that relays wireless communication,
The machine learning device according to any one of claims 1 to 10. a machine learning device according to any one of claims 1 to 11;
a determination target acquiring unit that acquires determination target data including information about each of the plurality of radio signals received by at least one of the one or more wireless receiving devices;
(i) inputting the determination target data acquired by the determination target acquisition unit into the learning model generated by the model generation unit; and (ii) using the second value determined by the adjustment unit as the reference value. , the users of the one or more wireless terminals that have transmitted the radio signal related to the determination target data exist within the area in which the wireless receiving device that received the wireless signal related to the determination target data is arranged a determination unit that determines whether or not
An information processing device. A program for causing a computer to function as the machine learning device according to any one of claims 1 to 11. A program for causing a computer to function as the information processing apparatus according to claim 12. A data point containing information relating to a radio signal received by a radio receiver located within an area having a predetermined geographic extent is entered such that a user of a radio terminal that originated the radio signal is located within the area. a model generation stage that generates a learning model that is used to determine whether the
an adjustment step of adjusting a reference value for determining whether or not the user was present in the area in the learning model generated in the model generation step;
has
The model generation step includes:
(a) using an initial value according to the learning model as the reference value, (b) (i) information about each of the plurality of radio signals received by one or more of the radio receiving devices, and ) a label indicating whether or not the user of the wireless terminal that transmitted each of the plurality of wireless signals was present within the area in which the wireless receiving device that received each of the plurality of wireless signals was arranged; generating the learning model by performing machine learning using a training data set comprising
The adjustment step includes:
A second value that increases the true positive rate of the determination result of the learning model generated in the model generation stage compared to the case where the initial value is used as the reference value is the adjusted reference value including determining as
machine learning method.

Images