CN111382334A - Data processing method and device, computer and readable storage medium - Google Patents

Data processing method and device, computer and readable storage medium Download PDF

Info

Publication number
CN111382334A
CN111382334A CN202010166886.2A CN202010166886A CN111382334A CN 111382334 A CN111382334 A CN 111382334A CN 202010166886 A CN202010166886 A CN 202010166886A CN 111382334 A CN111382334 A CN 111382334A
Authority
CN
China
Prior art keywords
target
area
terminal device
index
mapping relation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010166886.2A
Other languages
Chinese (zh)
Other versions
CN111382334B (en
Inventor
李一非
孙福宁
许迅腾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202010166886.2A priority Critical patent/CN111382334B/en
Publication of CN111382334A publication Critical patent/CN111382334A/en
Application granted granted Critical
Publication of CN111382334B publication Critical patent/CN111382334B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Telephonic Communication Services (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application discloses a data processing method, a data processing device, a computer and a readable storage medium, which are applied to the field of big data and used for counting massive terminals in an area. The method comprises the following steps: acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in the statistical area set, taking the area where the positioning information is located as a target area and acquiring a historical record index; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when no target incidence relation exists in the mapping relation according to the historical record index, adding the target incidence relation into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation. By adopting the method and the device, the generation of repeated data can be reduced, and the accuracy of the data statistical result is improved.

Description

Data processing method and device, computer and readable storage medium
Technical Field
The present application relates to the field of computing technologies, and in particular, to a data processing method, an apparatus, a computer, and a readable storage medium.
Background
With the development of the internet, online data are more and more, including various transaction data, positioning data and the like, and statistical analysis of these data is also a very important thing, so far, statistical analysis of data, such as Storm and the like, is generally realized based on a distributed real-time big data processing system. Storm is an open-source distributed real-time big data processing framework. Specifically, the real-time computation framework represented by Storm is stateless for the statistical values of the data, wherein stateless means that the statistical data in each time window is only related to the data flow at the current time. For example, for the order quantity in the trading platform, the data in the trading platform can be counted based on the real-time computing framework, wherein the real-time computing framework can directly count the positioning information of the terminal device in each counting time period to obtain the pedestrian volume of each area, and since the positioning information is counted, the same data can be repeatedly counted, so that the obtained statistical information may have some redundant data, which causes inaccuracy of data statistics.
Disclosure of Invention
The embodiment of the application provides a data processing method and device, which can avoid the statistics of repeated data and ensure the accuracy of data statistics results.
An embodiment of the present application provides a data processing method, including:
acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in a statistical area set, taking the area where the positioning information is located as a target area to acquire a historical record index; the historical record index is generated according to the mapping relation between the counted area and the terminal equipment;
acquiring a target association relation between the target terminal equipment and the target area;
when the mapping relation is detected to be not existed in the mapping relation according to the historical record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation;
and updating the mapping relation according to the target association relation between the target terminal equipment and the target area when the target association relation is detected to exist in the mapping relation according to the historical record index.
Wherein, the history index comprises a state bit array;
the method further comprises the following steps:
acquiring a target state hash value of the target terminal device, and acquiring the state bit array, wherein the state bit array comprises a plurality of state values, and the state values are generated according to the state hash value of the terminal device in the mapping relation;
acquiring a target state value of the target terminal equipment from the state bit array according to the target state hash value;
and if at least one missing state value exists in the target state value, determining that the target association relation does not exist in the mapping relation.
Wherein, the history index also comprises a history equipment identification index;
the method further comprises the following steps:
if the target state values are all valid state values, acquiring a target terminal device identifier of the target terminal device, and acquiring the historical device identifier index; the historical equipment identifier index is generated by an identifier mapping relation between the area identifier and the terminal equipment identifier;
filtering and detecting a target identifier mapping relation between the target terminal device identifier and the target area identifier of the target area according to the historical device identifier index;
if the historical device identifier index is filtered and detected to have no target identifier mapping relation, determining that the target association relation does not exist in the mapping relation;
and if the historical equipment identifier index is filtered and detected to have the target identifier mapping relation, determining that the target association relation exists in the mapping relation.
Wherein, the obtaining the target state value of the target terminal device from the state bit array according to the target state hash value includes:
acquiring a corresponding target state position of the target state hash value in the state bit array;
and acquiring the state value at the target state position in the state bit array as the target state value of the target terminal equipment.
Wherein, the filtering and detecting the target identifier mapping relationship between the target terminal device identifier and the target area identifier of the target area according to the historical device identifier index includes:
acquiring a target area identifier of the target area, and acquiring an equipment identifier list corresponding to the target area identifier from the historical equipment identifier index;
if the device identifier list comprises the target terminal device identifier, determining that a target identifier mapping relation between the target terminal device identifier and the target area identifier exists in the historical device identifier index;
and if the equipment identification list does not comprise the target terminal equipment identification, determining that a target identification mapping relation between the target terminal equipment identification and the target area identification does not exist in the historical equipment identification index.
Wherein, after the step of determining that the target association relationship does not exist in the mapping relationship if the target state value has at least one missing state value, the method further comprises:
and updating the state value at the target state position to be a valid state value in the state bit array.
Wherein, the method further comprises:
if the mapping relation is null, an initial state bit array is created, and the state value of the initial state bit array is set as the missing state value to obtain a state bit array.
Wherein, the target terminal device carries the target terminal device identifier, and the method further comprises:
acquiring the mapping relation, and acquiring a history area identifier associated with the target terminal equipment identifier according to the mapping relation;
acquiring the number of the historical region identifications as the number of regions;
and taking the area number and the historical area identification as the device position statistical information of the target terminal device.
Wherein, the method further comprises:
acquiring a target area identifier of the target area;
acquiring a device set associated with the target area identifier according to the mapping relation, wherein the device set comprises historical terminal devices;
and acquiring the number of the historical terminal devices as the device number statistical information of the target area.
Wherein, when it is detected that the target association relationship exists in the mapping relationship according to the history index, updating the mapping relationship according to the target association relationship between the target terminal device and the target area includes:
if the target association relation exists in the mapping relation according to the historical record index, acquiring first system network time;
acquiring historical record information associated with the target area and the target terminal equipment from the mapping relation;
and updating the target recording time in the historical recording information to the first system network time.
Wherein, when it is detected that the target association relationship exists in the mapping relationship according to the history index, updating the mapping relationship according to the target association relationship between the target terminal device and the target area includes:
if the target association relationship exists in the mapping relationship according to the historical record index, acquiring a target time set corresponding to the target association relationship from the mapping relationship;
and acquiring second system network time, and adding the second system network time to the target time set.
Wherein, the method further comprises:
acquiring at least two recording times included in the target time set and a statistical time period of the mapping relation;
acquiring a time interval between adjacent recording times in the at least two recording times, and determining a time interval which is the same as the statistical time period as an effective time interval;
and determining the total value of the effective time interval as the recording duration of the target terminal equipment in the target area.
Wherein, the method further comprises:
if the area where the positioning information is located is not in the statistical area set, filtering and detecting the target terminal equipment according to the historical record index;
and if the filtering detects that the target terminal equipment does not exist in the historical record index, filtering and deleting the positioning information of the target terminal equipment.
Before the step of obtaining the positioning information of the target terminal device, the method further includes:
requesting to acquire the mapping relation between the counted area and the terminal equipment, and recording the request times i; i is a positive integer;
if the mapping relation is successfully acquired at the ith time, generating the historical record index according to the mapping relation;
if the mapping relation fails to be acquired at the ith time and i is smaller than N, a time interval t is elapsediThen, the mapping relation is requested to be obtained in the (i + 1) th time; n is a positive integer greater than 1;
if the mapping relation is successfully acquired at the (i + 1) th time, generating the historical record index according to the mapping relation;
and if the mapping relation fails to be acquired at the (i + 1) th time and i +1 is N, the acquisition of the mapping relation is suspended.
An embodiment of the present application provides a data processing apparatus, where the apparatus includes:
a first obtaining module, configured to obtain location information of a target terminal device, and if an area where the location information is located is in a statistical area set, take the area where the location information is located as a target area, and obtain a history index; the historical record index is generated according to the mapping relation between the counted area and the terminal equipment;
a second obtaining module, configured to obtain a target association relationship between the target terminal device and the target area;
a relation adding module, configured to add, when it is detected that the target association relation does not exist in the mapping relation according to the history index, a target association relation between the target terminal device and the target area to the mapping relation;
and the relation updating module is used for updating the mapping relation according to the target association relation between the target terminal equipment and the target area when the target association relation is detected to exist in the mapping relation according to the historical record index.
Wherein, the history index comprises a state bit array;
the above-mentioned device still includes:
a third obtaining module, configured to obtain a target state hash value of the target terminal device, and obtain the state bit array, where the state bit array includes a plurality of state values, and the state values are generated according to the state hash value of the terminal device in the mapping relationship;
a state obtaining module, configured to obtain a target state value of the target terminal device from the state bit array according to the target state hash value;
a first determining module, configured to determine that the target association relationship does not exist in the mapping relationship if at least one missing state value exists in the target state value.
Wherein, the history index also comprises a history equipment identification index; the above-mentioned device still includes:
a fourth obtaining module, configured to obtain a target terminal device identifier of the target terminal device and obtain the historical device identifier index if the target state values are all valid state values; the historical equipment identifier index is generated by an identifier mapping relation between the area identifier and the terminal equipment identifier;
a first filtering module, configured to filter and detect a target identifier mapping relationship between the target terminal device identifier and a target area identifier of the target area according to the historical device identifier index;
a second determining module, configured to determine that the target association relationship does not exist in the mapping relationship if it is detected by filtering that the target identifier mapping relationship does not exist in the historical device identifier index;
the second determining module is further configured to determine that the target association relationship exists in the mapping relationship if the filtering detects that the target identifier mapping relationship exists in the historical device identifier index.
Wherein, the state acquisition module includes:
an array obtaining unit, configured to obtain a corresponding target state position of the target state hash value in the state bit array;
and a state acquiring unit, configured to acquire, in the state bit array, a state value at the target state position as a target state value of the target terminal device.
Wherein, above-mentioned first filtration module includes:
a hash obtaining unit, configured to obtain a target area identifier of the target area, and obtain an equipment identifier list corresponding to the target area identifier from the historical equipment identifier index;
a first determining unit, configured to determine that a target identifier mapping relationship between the target terminal device identifier and a target area identifier of the target area exists in the historical device identifier index if the device identifier list includes the target terminal device identifier;
a second determining unit, configured to determine that a target identifier mapping relationship between the target terminal device identifier and the target area identifier of the target area does not exist in the historical device identifier index if the device identifier list does not include the target terminal device identifier.
Wherein, above-mentioned device still includes:
and the index updating module is used for updating the state value at the target state position into an effective state value in the state bit array.
Wherein, above-mentioned device still includes:
and the creating module is used for creating an initial state bit array if the mapping relation is null, and setting the state value of the initial state bit array as a missing state value to obtain a state bit array.
Wherein, above-mentioned target terminal equipment carries the terminal equipment label of the goal, above-mentioned apparatus also includes:
a history region obtaining module, configured to obtain the mapping relationship, and obtain a history region identifier associated with the target terminal device identifier according to the mapping relationship;
the first statistical module is used for acquiring the number of the historical region identifications as the number of regions;
and a first generation module, configured to use the number of regions and the historical region identifier as device location statistics information of the target terminal device.
Wherein, above-mentioned device still includes:
a target area obtaining module, configured to obtain a target area identifier of the target area;
the device acquisition module is used for acquiring a device set associated with the target area identifier according to the mapping relation, wherein the device set comprises historical terminal devices;
and the second statistical module is used for acquiring the number of the historical terminal devices as the device number statistical information of the target area.
Wherein, the relationship updating module comprises:
a time obtaining unit, configured to obtain a first system network time if it is detected that the target association relationship exists in the mapping relationship according to the history index;
a record obtaining unit, configured to obtain history information associated with the target area and the target terminal device from the mapping relationship;
and a time updating unit for updating the target recording time in the history information to the first system network time.
Wherein, the relationship updating module further comprises:
a set obtaining unit, configured to obtain a target time set corresponding to the target association relationship from the mapping relationship if it is detected that the target association relationship exists in the mapping relationship according to the history index;
and the time adding unit is used for acquiring the network time of a second system and adding the network time of the second system to the target time set.
Wherein, above-mentioned device still includes:
a set obtaining module, configured to obtain at least two recording times included in the target time set and a statistical time period of the mapping relationship;
an interval obtaining module, configured to obtain a time interval between adjacent recording times of the at least two recording times, and determine a time interval that is the same as the statistical time period as an effective time interval;
and a duration determining module, configured to determine the total value of the effective time interval as a recording duration of the target terminal device in the target area.
Wherein, above-mentioned device still includes:
the first obtaining module is further configured to, if the area where the positioning information is located is not in the statistical area set, perform filtering detection on the target terminal device according to the history index;
and the information deleting module is used for filtering and deleting the positioning information of the target terminal equipment if the history index is detected to be free from the target terminal equipment by filtering.
Wherein, above-mentioned device still includes:
the relation acquisition module is used for requesting to acquire the mapping relation between the counted area and the terminal equipment and recording the request times i; i is a positive integer;
an index generating module, configured to generate the history index according to the mapping relationship if the mapping relationship is successfully obtained at the ith time;
the relationship obtaining module is further configured to, if obtaining the mapping relationship at the ith time fails and i is smaller than N, elapse a time interval tiThen, the mapping relation is requested to be obtained in the (i + 1) th time; n is a positive integer greater than 1;
the index generating module is further configured to generate the history index according to the mapping relationship if the mapping relationship is successfully obtained in the (i + 1) th time;
and the exception handling module is used for suspending the acquisition of the mapping relation if the acquisition of the mapping relation at the (i + 1) th time fails and i +1 is N.
One aspect of the embodiments of the present application provides a computer device, including a processor, a memory, and an input/output interface;
the processor is respectively connected to the memory and the input/output interface, where the input/output interface is used for inputting data and outputting data, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the data processing method implemented in one aspect of the embodiment of the present application.
An aspect of the embodiments of the present application provides a computer-readable storage medium, in which a computer program is stored, where the computer program includes program instructions, and the program instructions, when executed by a processor, perform a data processing method implemented in an aspect of the embodiments of the present application.
The embodiment of the application has the following beneficial effects:
according to the embodiment of the application, the positioning information of the target terminal equipment is obtained, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is used as the target area, and the historical record index is obtained; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when the mapping relation is detected to be not existed in the history record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area. Therefore, by recording the mapping relation of the historical data, the target association relation of the current data can be screened based on the mapping relation, the statistics of the repeated data is avoided, and the accuracy of the data statistics result is ensured.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1a is a diagram of a data processing network architecture provided by an embodiment of the present application;
FIG. 1b is a schematic diagram of a computer device according to an embodiment of the present disclosure;
FIG. 1c is a schematic diagram of another computer device according to an embodiment of the present application;
fig. 2 is a data processing application scenario architecture diagram provided in an embodiment of the present application;
FIG. 3 is a block diagram of a data processing flow provided in an embodiment of the present application;
fig. 4 is a flowchart of a data processing method provided in an embodiment of the present application;
fig. 5 is a schematic view of a target area acquisition scene provided in an embodiment of the present application;
fig. 6 is a schematic diagram of a specific flow of data processing according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a state bit array generation and filtering scenario provided by an embodiment of the present application;
FIG. 8 is a schematic diagram of a data processing scenario provided by an embodiment of the present application;
FIG. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
According to the method and the device, the related data of the massive terminal equipment needs to be acquired and processed, the big data technology can be applied to acquire the positioning information of the massive terminal equipment, and the acquired massive terminal equipment is filtered and counted. The big data technology is one of cloud technologies, and is specifically introduced as follows:
1. cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.
Cloud technology (Cloud technology) is based on a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied in a Cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.
2. Big data (Big data) refers to a data set which cannot be captured, managed and processed by a conventional software tool within a certain time range, and is a massive, high-growth-rate and diversified information asset which can have stronger decision-making power, insight discovery power and flow optimization capability only by a new processing mode. With the advent of the cloud era, big data has attracted more and more attention, and the big data needs special technology to effectively process a large amount of data within a tolerance elapsed time. The method is suitable for the technology of big data, and comprises a large-scale parallel processing database, data mining, a distributed file system, a distributed database, a cloud computing platform, the Internet and an extensible storage system.
Specifically, please refer to fig. 1a, fig. 1a is a data processing network architecture diagram provided in an embodiment of the present application, and as shown in fig. 1a, the data processing system includes a computer device 101, a mapping relation 103 between a counted area and a terminal device is stored in the computer device 101, and a history index 104 generated according to the mapping relation 103 is stored. The computer device 101 acquires the positioning information of the terminal device 102a, the positioning information of the terminal device 102b, the positioning information of the terminal device 102c, and the like, and takes one terminal device as an example, for example, the terminal device 102a as a target terminal device. The computer device 101 acquires the positioning information of the target terminal device 102a, and if the area where the positioning information is located is in the statistical area set, acquires a history index 104 with the area where the positioning information is located as the target area, the history index 104 being generated from the mapping relation 103 between the counted area and the terminal device.
The computer apparatus 101 obtains an object association relationship between the target terminal apparatus 102a and the target area, and if it is detected that the mapping relationship 103 does not include the object association relationship according to the history index 104, it indicates that the target terminal apparatus 102a has not appeared in the target area before the current data processing, that is, there is no association relationship with the target area, and may add the object association relationship to the mapping relationship 103, or update the history index 104 according to the mapping relationship 103. When the history index 104 is updated according to the mapping relation 103, a new history index 104 can be directly generated according to the mapping relation 103 containing the target association relation; alternatively, after the target association relationship is added to the mapping relationship 103, the history index 104 is updated according to the target association relationship, that is, the mapping between the target terminal apparatus 102a and the target area is added to the history index 104.
For example, the computer apparatus 101 creates an initial history index, the initial history index is null, and assuming that the mapping relation 103 includes the association relation between the terminal apparatus 1 and the area 1 and the association relation between the terminal apparatus 2 and the area 1, the computer apparatus 101 updates the initial history index by the mapping relation 103 to obtain a history index, that is, adds the mapping between the terminal apparatus 1 and the area 1 and the mapping between the terminal apparatus 2 and the area 1 to the initial history index to obtain a history index 104, and it can be considered that the history index 104 is a mapping table between the terminal apparatus and the area, and the history index 104 may include "area 1| [ terminal apparatus 1, terminal apparatus 2 ]". The mapping relationship may be stored in an equipment statistics database, which belongs to the computer equipment 101. The computer device 101 obtains the positioning information of the terminal device 3, the area 3 where the positioning information of the terminal device 3 is located is in a statistical area set, the area 3 where the positioning information of the terminal device 3 is located is used as a target area, a target association relation between the terminal device 3 and the area 3 is obtained, the target association relation is detected according to the historical record index 104, the target association relation between the terminal device 3 and the area 3 does not exist in the historical record index 104 is obtained, the mapping relation 103 does not include the target association relation, the target association relation is added to the historical record index 104, so that the historical record index 104 is updated, and an' area 1| [ terminal device 1, terminal device 2] is obtained; area 3| [ terminal device 3] ".
Similarly, when the computer device 101 acquires the positioning information of the terminal device 102a, the positioning information of the terminal device 102b, and the positioning information of the terminal device 102c, the area where the positioning information of each terminal device is located is acquired, if the area where the positioning information is located is not in the statistical area set, the terminal device corresponding to the positioning information is deleted, whether the association relationships exist in the mapping relationship 103 is determined according to the association relationships between the remaining terminal devices and the area where the positioning information is located and the history index 104, and the association relationships that do not exist in the mapping relationship 103 are added to the mapping relationship 103. For example, if the positioning information of the terminal device 102a is acquired in the area 1, the positioning information of the terminal device 102b is acquired in the area 2, the positioning information of the terminal device 102c is acquired in the area 1, a statistical area set is acquired, the statistical area set includes the area 1 and the area 2, the history index 104 is acquired, the association between the terminal device 102a and the area 1, the association between the terminal device 102b and the area 2, and the association between the terminal device 102c and the area 1 are acquired, and if the association between the terminal device 102a and the area 1 and the association between the terminal device 102c and the area 1 exist in the history index 104, the association between the terminal device 102b and the area 2 does not exist in the history index 104 through comparison analysis, the association between the terminal device 102b and the area 2 is added to the mapping relation 103, so that the computer apparatus 101 can update the history index 104 according to the association relationship between the terminal apparatus 102b and the area 2. By using the historical record index, the generation of repeated data in the data processing process is reduced, so that the accuracy of the data statistical result is improved.
It is understood that the method provided by the embodiment of the present application can be executed by a computer device, including but not limited to a terminal device or a server. In the embodiment of the present application, the main execution body for performing data processing may be a computer device. The computer device 101 may be a server or a terminal device, or a system composed of a server and a terminal device, where the above mentioned terminal device may be an electronic device, including but not limited to a mobile phone, a tablet computer, a desktop computer, a notebook computer, a palmtop computer, a Mobile Internet Device (MID), a wearable device (e.g., a smart watch, a smart band, etc.), and the like.
The historical record index comprises a state bit array and a historical device identification index, wherein the state bit array can be a bloom filter and is used for performing coarse filtering on the acquired terminal device, namely detecting whether the terminal device exists in the bloom filter; the historical device identifier index is generated according to the mapping relation between the region and the terminal device, and comprises the identifier mapping relation between the counted terminal device identifier and the region identifier.
The computer device may include at least one processor and a data storage, each processor is configured to perform filtering detection on the acquired terminal device based on the status bit group, perform filtering detection on the acquired terminal device based on the historical device identification index, perform statistics on the mapping relationship, and the like, the data storage is configured to store the mapping relationship, the at least one processor and the data storage may be considered as different hardware in the computer device, and the like, and data interaction may be performed between the hardware.
Referring to fig. 1b specifically, fig. 1b is a schematic diagram illustrating a computer device according to an embodiment of the present disclosure, as shown in fig. 1b, the computer device includes at least one processor, such as a coarse filtering processor 1011, a fine filtering processor 1012, a data statistics processor 1014, and the like, and further includes a data memory 1013, wherein the coarse filtering processor 1011 stores a state bit array, the fine filtering processor 1012 stores a historical device identifier index, and the data memory 1013 stores a mapping relationship between a counted area and a terminal device. The computer equipment acquires the positioning information of the target terminal equipment, if the area where the positioning information is located is in a statistical area set, the area where the positioning information is located is used as a target area, a state bit array is called based on a coarse filtering processor 1011, the target terminal equipment is subjected to coarse filtering through the state bit array, if the state bit array is obtained and comprises the target terminal equipment, the target association relation between the target terminal equipment and the target area is input into a fine filtering processor 1012, a historical equipment identification index is called based on the fine filtering processor 1012, the target association relation is subjected to fine filtering, if the mapping relation is detected to be absent according to the historical equipment identification index, the target association relation is input into a data storage 1013, and the target association relation is added into the mapping relation based on the data storage 1013; if the target association relationship is detected in the mapping relationship according to the historical device identifier index, the target association relationship is input into the data storage 1013, and the mapping relationship is updated in the data storage 1013 based on the target association relationship. Further, the data statistics processor 1014 may obtain the mapping relationship from the data storage 1013, and count the mapping relationship to obtain the device number statistics information of the target area or the device location statistics information of the target terminal device. The computer device comprises a computer device, a data storage device and at least one processor, wherein the computer device comprises the computer device and the data storage device, the processor, the data storage device and the like can perform data interaction, and the at least one processor, the data storage device and the like form the device, namely the computer device.
The computer device may further include at least one processing unit, a data statistics unit, and a device statistics database, where the at least one processing unit performs data processing through a processor included in the corresponding processing unit, the data statistics unit performs data statistics through the processor included in the data statistics unit, and the device statistics database provides a storage space through a memory included in the device statistics database. Referring to fig. 1c specifically, fig. 1c is a schematic diagram of another computer device composition provided in this embodiment of the present application, and as shown in fig. 1c, the computer device may include at least one processing unit, a data statistics unit, and the like, such as a coarse filtering processing unit 1015, a fine filtering processing unit 1016, and a data statistics unit 1018, and further include a device statistics database 1017. The coarse filtering processing unit 1015 may be a server or a terminal device, stores a state bit array, and is configured to perform coarse filtering on the acquired terminal device based on the state bit array; the fine filtering processing unit 1016 may be a server or a terminal device, and store a historical device identifier index, and is configured to perform fine filtering on the obtained identifier mapping relationship between the terminal device and the area where the terminal device is located based on the historical device identifier index; the device statistics database 1017 stores the mapping relationship between the counted region and the terminal device; the data statistics unit 1018 may be a server or a terminal device, and is configured to obtain a mapping relationship in the device statistics database 1017, and perform statistics on the mapping relationship. The data interaction can be carried out among at least one processing unit, the data statistical unit, the equipment statistical database and the like in the computer equipment.
In other words, the execution subject of the embodiment of the present application is a computer device, which may be a device, as shown in fig. 1b, and the device implements the above-mentioned data processing procedure through different processors and the like; the computer device may also be a system in which multiple devices cooperate with each other, as shown in fig. 1c, and the data processing process is implemented by executing different steps by each device.
Further, please refer to fig. 2, fig. 2 is a diagram illustrating a data processing application scenario architecture according to an embodiment of the present application. As shown in fig. 2, a computer device obtains a positioning information set 201, where the positioning information set 201 is related to terminal devices obtained by the computer device in the data processing process, and includes obtained terminal devices and areas where each terminal device is located, that is, association relationships between each terminal device and the area where the terminal device is located, after the computer device obtains the positioning information set 201, the positioning information set 201 is filtered, target positioning information of an area indicated in the positioning information set 201 in a statistical area set is obtained, the area where the target positioning information is located is determined as a target area, the terminal device corresponding to the target positioning information is determined as a target terminal device, a history index 202 is obtained, the target terminal device and the target area are compared with the history index 202, and device statistical data 203 is obtained, the device statistical data 203 is statistical data obtained by comparing the target terminal device and the target area with the history index 202 by the computer device, that is, the association relationship between the terminal device and the area obtained in the comparison process. Optionally, the device statistical data 203 may include only the association relationship that is not present in the history index 202, or may include the association relationship corresponding to all the acquired target location information. It is assumed here that the device statistics 203 includes an association that does not exist in the history index 202, and a recording time of the association, where the recording time is a time of this data processing. The computer device updates the history index 202 according to the device statistical data 203, sorts the device statistical data 203, obtains the number of terminal devices appearing in the target area as the device number statistical information 204, and optionally, may also count the recording time of each terminal device in the target area to obtain the recording time 205 of the corresponding terminal device in the target area.
For example, assuming a statistical time period of one day, the computer device performs the steps of 8: 30 to 8: 31, receiving the positioning information of the terminal device 1 in the area 1, the positioning information of the terminal device 2 in the area 2, and the positioning information of the terminal device 3 in the area 1 in one minute, indicating that there is an association relationship between the terminal device 1 and the area 1, there is an association relationship between the terminal device 2 and the area 2, and there is an association relationship between the terminal device 3 and the area 1, and assuming that the statistical area set includes the area 1 and the area 2, determining that the area 1 where the positioning information of the terminal device 1 is located, the area 2 where the positioning information of the terminal device 2 is located, and the area 1 where the positioning information of the terminal device 3 is located are in the statistical area set. Assuming that the association relationship and the association frequency between the terminal device 1 and the area 1 and the association relationship and the association frequency between the terminal device 3 and the area 1 exist in the history index 202, and the comparison and analysis result shows that the association relationship between the terminal device 2 and the area 2 does not exist in the history index 202, the recording time 8 of the association relationship between the terminal device 2 and the area 2 and the association relationship between the terminal device 2 and the area 2 is: and 30, as the device statistical data 203, and adding the device statistical data 203 to the mapping relationship, wherein the device statistical data 203 further includes the recording time 8 of the terminal device 1 in the area 1: 30, and the recording time 8 of the terminal device 3 in the area 1: 30. the computer device counts the data in the mapping relationship to obtain the number of the terminal devices associated with each region, which is used as the device number statistical information 204 of the corresponding region, and represents the number of the terminal devices (i.e. the human traffic) appearing in the corresponding region. Optionally, the computer device may also count the recording time in the mapping relationship to obtain the recording duration 205 of each terminal device in different areas. In each statistical time period, the positioning information of the terminal equipment can be acquired; alternatively, the computer device receives location information for a terminal device when the terminal device is triggering an Application (APP) based on a location Software Development Kit (SDK).
Further, please refer to fig. 3, fig. 3 is a diagram illustrating a data processing flow according to an embodiment of the present disclosure. As shown in fig. 3, taking a computer device as an example of a system in which multiple devices cooperate with each other, the computer device receives positioning information of a terminal device 301a, a terminal device 301b, and a terminal device 301c, where the positioning information includes a region in which a corresponding terminal device is located, the computer device determines the region in which the corresponding terminal device is located according to the positioning information of each terminal device, and if the region in which the terminal device is located is in a statistical region set, obtains an association relationship between each terminal device in the statistical region set and the region in which the terminal device is located. For example, the computer device receives the positioning information of the terminal device 301a, where the positioning information of the terminal device 301a indicates that the area where the terminal device 301a is located is area 1, and the area 1 obtains the association relationship between the terminal device 301a and the area 1 in the statistical area set; after the computer device receives the positioning information of the terminal device 301b, the positioning information of the terminal device 301b indicates that the area where the terminal device 301b is located is area 2, and the area 2 is in a statistical area set to obtain an association relationship between the terminal device 301b and the area 2; after the computer device receives the positioning information of the terminal device 301c, the positioning information of the terminal device 301c indicates that the area where the terminal device 301c is located is area 1, and the area 1 obtains the association relationship between the terminal device 301c and the area 1 in the statistical area set.
The computer device inputs the association relationship between each terminal device and each region into the coarse filtering processing unit 302, and performs preliminary filtering on the association relationship between each terminal device and each region through the coarse filtering processing unit 302, specifically, a state bit array is stored in the coarse filtering processing unit 302, and after the coarse filtering processing unit 302 acquires the association relationship between each terminal device and each region, the terminal devices are divided into recorded state devices and unrecorded state devices, where the recorded state devices are the terminal devices existing in the state bit array in each terminal device. Assuming that the mapping relationship includes an association relationship between the terminal device 301a and the area 1 and an association relationship between the terminal device 301c and the area 2, and the status bit array is generated according to the terminal device counted in the mapping relationship, at this time, the status bit array includes the terminal device 301a and the terminal device 301c, the computer device performs filtering processing on the acquired terminal device 301a, the terminal device 301b, and the terminal device 301c through the coarse filtering processing unit 302, and takes the terminal device 301a and the terminal device 301c existing in the status bit array as recorded status devices and the terminal device 301b as unrecorded status devices. The state bit array is a binary array, the binary values are respectively a valid state value and a missing state value, the state bit array comprises a plurality of state values, each state value is a valid state value or a missing state value, the valid state values are used for indicating that the corresponding terminal equipment is in the state bit array, the missing state values are used for indicating that the corresponding terminal equipment does not belong to the state bit array, and the state bit array can be a bloom filter and the like.
The coarse filtering processing unit 302 may include at least one coarse filtering processing subunit, such as a coarse filtering processing subunit 3021, a coarse filtering processing subunit 3022, a coarse filtering processing subunit 3023, and a coarse filtering processing subunit 3024, where each coarse filtering processing subunit stores a state bit array, and after the computer device acquires the terminal device, the terminal device may be allocated to different coarse filtering processing subunits, where each coarse filtering processing subunit may be regarded as a server or a terminal device that is independent of each other. Optionally, the computer device may obtain the working conditions of each coarse filtering processing subunit, and allocate the terminal device to the coarse filtering processing subunit with the optimal current processing performance according to the working conditions, where if the working processing memory of the coarse filtering processing subunit 3021 is occupied by 20%, the working processing memory of the coarse filtering processing subunit 3022 is occupied by 2%, the working processing memory of the coarse filtering processing subunit 3023 is occupied by 15%, and the working processing memory of the coarse filtering processing subunit 3024 is occupied by 0%, the obtained terminal device is input to the coarse filtering processing subunit 3024 for performing coarse filtering, or the obtained terminal device is input to the coarse filtering processing subunit 3022 and the coarse filtering processing subunit 3024 for performing coarse filtering, so as to improve the efficiency of coarse filtering.
Optionally, each of the coarse filtering processing sub-units may filter terminal devices in different zone ranges, for example, the coarse filtering processing sub-unit 3021 is configured to filter terminal devices in the zones 1 to 50, the coarse filtering processing sub-unit 3022 is configured to filter terminal devices in the zones 51 to 100, the coarse filtering processing sub-unit 3023 is configured to filter terminal devices in the zones 101 to 150, and the coarse filtering processing sub-unit 3024 is configured to filter terminal devices in the zones 151 to 200. At this time, the computer device acquires that the terminal device 301a is in the area 1, the terminal device 301b is in the area 2, and the terminal device 301c is in the area 1, and the area 1 and the area 2 both belong to the area range from the area 1 to the area 50, and the computer device inputs the terminal device 301a, the terminal device 301b, and the terminal device 301c into the coarse filtering processing subunit 3021 to perform filtering detection on the terminal device 301a, the terminal device 301b, and the terminal device 301c by the coarse filtering processing subunit 3021.
After the computer device obtains the recorded state devices, the recorded state devices and the areas where the recorded state devices are located are input into a fine filtering processing unit 303, and the association relationship between each recorded state device and each area is filtered to obtain the recorded association relationship and the unrecorded association relationship, wherein the fine filtering processing unit 303 includes a historical device identifier index, and the recorded association relationship is the association relationship existing in the historical device identifier index. Specifically, the computer device inputs the association relationship corresponding to the recorded state device into the fine filtering processing unit 303, the association relationship corresponding to the recorded state device includes the association relationship between the terminal device 301a and the area 1 and the association relationship between the terminal device 301c and the area 1, the computer device filters the association relationship corresponding to the recorded state device through the fine filtering processing unit 303, specifically, the fine filtering processing unit 303 filters the association relationship corresponding to the recorded state device through the historical device identifier index, the history device identification index includes the association between the terminal device 301a and the area 1 and the association between the terminal device 301c and the area 2, the association between the terminal device 301a and the area 1 is thus regarded as a recorded association, and the association between the terminal device 301c and the area 1 is regarded as an unrecorded association. The history device identifier index is generated according to an identifier mapping relationship between the identifier of each terminal device and the area identifier of the area, and the identifier mapping relationship corresponds to an association relationship between the terminal device and the area in the mapping relationship, so that the recorded association relationship may be regarded as an association relationship existing in the mapping relationship, in other words, the computer device detects that the history device identifier index includes the association relationship between the terminal device 301a and the area 1, determines that the mapping relationship includes the association relationship between the terminal device 301a and the area 1, and takes the association relationship between the terminal device 301a and the area 1 as the recorded association relationship. For example, the mapping relationship includes an association relationship between the terminal device 1 and the area 1, and the history device identifier index includes an identifier mapping relationship between the terminal device 1 and the area 1, where the history device identifier index may be regarded as a mapping table. The historical device identifier index may be a mapping table in which an area is used as an index, and a terminal device list is used as a value corresponding to the index; or a mapping table with the terminal device as an index and the area list as a corresponding indexed value.
The fine filtering unit 303 may include at least one fine filtering subunit, such as the fine filtering subunit 3031 and the fine filtering subunit 3032, where each fine filtering subunit stores a historical device identifier index, and after the computer device obtains the recorded state device, the recorded state device may be allocated to different fine filtering subunits, where at least one fine filtering subunit may be a server or a terminal device that is independent of each other. Optionally, the computer device may obtain the working conditions of each fine filtering sub-unit, and allocate the recorded state device to the fine filtering sub-unit with the optimal current processing performance or n fine filtering sub-units with the current processing performance arranged in the front row according to the working conditions, specifically referring to the allocation manner of the coarse filtering sub-unit, where n is a positive integer and n is less than or equal to the number of the fine filtering sub-units.
Optionally, each fine filtering processing subunit may filter the association relationship in different region ranges, for example, the fine filtering processing subunit 3031 is configured to filter the association relationship from the region 1 to the region 100, and the fine filtering processing subunit 3032 is configured to filter the association relationship from the region 101 to the region 200. The recorded state devices acquired by the computer device are the terminal device 301a and the terminal device 301c, wherein the terminal device 301a and the area 1 have an association relationship, the terminal device 301c and the area 1 have an association relationship, the area 1 belongs to an area range from the area 1 to the area 100, the computer device inputs the association relationship between the terminal device 301a and the area 1 and the association relationship between the terminal device 301c and the area 1 into the fine filtering processing subunit 3031, and filters the association relationship between the terminal device 301a and the area 1 and the association relationship between the terminal device 301c and the area 1 through the fine filtering processing subunit 3031.
The computer device counts the unrecorded state device, the recorded association relationship and the unrecorded association relationship through a data counting unit 304 to obtain device statistical data; the computer device can also count the mapping relationship through the data counting unit 304 to obtain statistical information, which can be statistical information of device location or statistical information of device quantity; the device location statistical information is used for representing the area information located by the terminal device within the statistical time range, namely the area which the user corresponding to the terminal device has gone within the statistical time range; the device quantity statistical information is used for indicating the quantity of terminal devices appearing in each area within the statistical time range, namely, the users appearing in the areas within the statistical time range can be used for indicating the flow of people in each area. The computer device updates the mapping relation 305 according to the device statistical data, updates the historical record index 306 based on the mapping relation 305, and stores the state bit array and the historical device identifier index included in the updated historical record index 306 into the coarse filtering unit 302 and the fine filtering unit 303, respectively.
Optionally, if the current system time is not in the statistical time range, the data in the mapping relation 305 and the history index 306 are deleted to start a new statistical time range, and the data processing is performed. In other words, when a statistical time range starts, the mapping relation 305 is empty, and correspondingly, the history index 306 generated according to the mapping relation 305 is also empty, the computer device updates the mapping relation 305 through the steps of acquiring the terminal device and filtering the terminal device, and when the statistical time range ends, the data in the mapping relation 305 is sorted to obtain the statistical information. And the computer equipment acquires the terminal equipment and filters the terminal equipment once every time, so that the statistical time period is considered. Wherein one statistical time range comprises a plurality of statistical time periods. The statistical time range may be represented by a timer, and if the statistical time range includes one day, setting the value of the timer to one day when the statistical time range starts, counting down with the timer, and when the obtained value of the timer is 0, considering that the statistical time range ends, and sorting the data in the mapping relation 305 to obtain statistical information; the timer value is set to one day again and a new statistical time range is started. Alternatively, the data in the mapping relation 305 and the history index 306 may be cleared when the value of the timer is set to one day.
The mapping relation 305 is stored in a device statistics database, which may be a Remote Dictionary service (Redis) database, the Redis database is an open-source, network-supporting, memory-based and persistent log-type, Key-Value (Key-Value) database, and the device statistics database may be in an independent Server or terminal device. The mapping relation 305 stores the statistical association relation between all the areas and each terminal device, and the computer device updates the mapping relation 305 once every statistical time period elapses. The key value pairs in the mapping relation 305 are "area identifier" and "terminal device", such as "area 1, terminal device 301 a" and "area 2, terminal device 301 c", and the like, respectively; or, the key-value pairs in the mapping relation 305 are "area id" and "array formed by terminal devices", such as "area 1| [ terminal devices 301a, … ]" and "area 2| [ terminal devices 301c, … ]", respectively; alternatively, the key-value pairs in the mapping relation 305 are "zone id" and "terminal device | latest recording timestamp, an array of two values", such as "zone 1, [ terminal device 301a | 2019-01-0112: 00: 00, … ] "and" area 2, [ terminal device 301c | 2019-01-0212: 00: 00, … ] ", etc. Optionally, the key-value pair may also be formed based on "terminal device" and "area identifier". In the key value pair, the terminal device may be represented by a terminal device identifier or a hash value of the terminal device. The Redis database is a database stored by key value pairs, and the readability of the data in the application can be improved by storing the data to be stored in the key value pair format.
The device statistics database is a device independent of the coarse filtering unit 302, the fine filtering unit 303, and the data statistics unit 304, and the stored mapping relationship is equivalent to a non-memory variable for the coarse filtering unit 302, the fine filtering unit 303, and the data statistics unit 304. The mapping relation can be acquired through the device statistical database or the independent device, the historical record index is generated and distributed to each processing unit, so that each processing unit can acquire the historical record index without storing the mapping relation, the loss of the memory space of each processing unit is reduced, and the resource loss of each processing unit is reduced.
The data statistics unit 304 may include a plurality of data statistics subunits, such as a data statistics subunit 3041, a data statistics subunit 3042, a data statistics subunit 3043, and the like, where each data statistics subunit may be a server or a terminal device that is independent of each other. Optionally, each data statistics subunit is respectively configured to count terminal devices in different area ranges, or count areas where different terminal devices are located, and generate device statistics data according to statistics data obtained by each data statistics subunit. For example, assume that the data statistics subunit 3041 processes data from region 1 to region 100, the data statistics subunit 3042 processes data from region 101 to region 200, and the data statistics subunit 3043 processes data from region p to region q, where p and q are positive integers. After the fine filtering processing subunit 3031 and the fine filtering processing subunit 3032 perform filtering detection on the recorded state device, the association relationship input data statistics subunit 3041 associated with any one of the regions 1 to 100, the association relationship input data statistics subunit 3042 associated with any one of the regions 101 to 200, and the association relationship input data statistics subunit 3043 associated with any one of the regions p to q are input. For example, if the computer device obtains the association relationship between the terminal device 301a and the area 1, the association relationship between the terminal device 301b and the area 2, and the association relationship between the terminal device 301c and the area 1, and the areas 1 and 2 all belong to the area range from the area 1 to the area 100, the association relationship between the terminal device 301a and the area 1, the association relationship between the terminal device 301b and the area 2, and the association relationship between the terminal device 301c and the area 1 are input to the data statistics subunit 3041 to be counted by the data statistics subunit 3041.
Optionally, when each data statistics subunit is used to count terminal devices in different areas, after the fine filtering processing subunit 3031 and the fine filtering processing subunit 3032 perform filtering detection on the recorded state device, each association relationship is input to the corresponding data statistics subunit according to the terminal device in each association relationship. For example, the data statistics subunit 3041 counts data of the areas where the terminal devices 1 to 100 are located, the data statistics subunit 3042 counts data of the areas where the terminal devices 101 to 200 are located, and the data statistics subunit 3043 counts data of the areas where the terminal devices p to q are located. Assuming that the terminal device 301a, the terminal device 301b, and the terminal device 301c acquired by the computer device are terminal devices 1, 104, and 3, the association relationship between the terminal device 301a and the area 1 and the association relationship between the terminal device 301c and the area 1 are input into the data statistics subunit 3041, and the association relationship between the terminal device 301b and the area 2 is input into the data statistics subunit 3042.
In the embodiment of the application, the state bit array is stored in the coarse filtering processing unit, the memory space occupied by the state bit array is small, the obtained terminal equipment is subjected to coarse filtering through the coarse filtering processing unit, a large number of terminal equipment can be filtered, the related data volume of the residual terminal equipment obtained after filtering through the coarse filtering processing unit is small, the incidence relation between the residual terminal equipment and the region where the residual terminal equipment is located is filtered through the fine filtering processing unit, the repeated incidence relation is filtered, the non-counted incidence relation is added into the mapping relation, the data volume needing to be processed is reduced, and the data processing efficiency is improved. Meanwhile, the coarse filtering processing unit and the fine filtering processing unit are combined for use, so that the generation of repeated data is reduced, and the accuracy of a data statistical result is improved.
Further, please refer to fig. 4, where fig. 4 is a flowchart of a data processing method according to an embodiment of the present application. As shown in fig. 4, the data processing process is described with the computer device as an execution subject, and includes the following steps:
step S401, obtaining the positioning information of the target terminal device, and if the area where the positioning information is located is in the statistical area set, taking the area where the positioning information is located as the target area, and obtaining the history index.
Specifically, positioning information of the target terminal device is obtained, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is used as a target area to obtain a history index; the history index is generated according to the mapping relation between the counted area and the terminal equipment. In the mapping relationship, for the same area, the terminal devices associated with the area are different from each other. Specifically, the computer device receives positioning information of a target terminal device, determines the positioning information of the target terminal device according to the positioning information, acquires a region where the positioning information is located, and when the region where the positioning information is located is in a statistical region set, takes the region where the positioning information is located as the target region where the target terminal device is located, and acquires a history index, wherein the history index is generated according to a current mapping relationship, the statistical region set includes at least one statistical region, and the at least one statistical region is a region needing to be counted. In each statistical time period, the computer equipment receives the positioning information of the terminal equipment; or, when the terminal device triggers the APP based on the location SDK, the computer device may obtain location information of the terminal device, and so on.
The areas determined by the computer device through the geofence are an application of Location Based Service (LBS), in other words, the geofence is a virtual geofence which is used as a statistical carrier for "when the terminal device enters, leaves, or is active in a specific geographic area", and can be understood as a customized area on the map. The computer equipment encloses a virtual geographic boundary from an area needing to be counted through a geographic fence, and divides a corresponding area through each virtual geographic boundary, wherein the positions in the virtual geographic boundary all belong to the area corresponding to the virtual geographic boundary. After the computer equipment encloses a plurality of areas through the geographic fence, when the computer equipment counts, the terminal equipment in each area is obtained, and the positioning information of each terminal equipment is obtained. Specifically, referring to fig. 5, fig. 5 is a schematic view of a target area acquisition scene provided in the embodiment of the present application. As shown in fig. 5, when the target terminal device is in "shanghai bank", the computer device receives the positioning information 502 of the target terminal device, obtains an area where the target terminal device is located according to the positioning information of the target terminal device 502 to the positioning information and the virtual geographic boundary determined by the geo-fence, and if the area is in the statistical area set, takes the area as the target area 503 of the target terminal device.
Step S402, acquiring a target association relation between the target terminal equipment and the target area.
Specifically, a target association relationship between the target terminal device and the target area is obtained. Specifically, after the computer device obtains the positioning information of the target terminal device and obtains the target area where the target terminal device is located based on the positioning information, it is determined that a target association relationship exists between the target terminal device and the target area, where the target association relationship is used to indicate that the target terminal device is in the target area, that is, the user corresponding to the target terminal device is in the target area.
Step S403, when the mapping relation is detected to be not existed in the mapping relation according to the historical record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation.
Specifically, when it is detected that no target association relationship exists in the mapping relationship according to the history index, the target association relationship between the target terminal device and the target area is added to the mapping relationship. Specifically, the target association relationship is compared with the history index, when the target association relationship does not exist in the history index, it indicates that the target terminal device does not appear in the target area, the target association relationship is added to the mapping relationship, and the history index is updated according to the mapping relationship, so that the history index is used in the data processing process in the next statistical time period.
And step S404, when the target incidence relation is detected in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area.
Specifically, the target association relationship is compared with the history index, when the target association relationship exists in the history index, it is determined that the target association relationship exists in the mapping relationship detected according to the history index, and the mapping relationship is updated according to the target association relationship between the target terminal device and the target area.
According to the data processing process, the positioning information of the target terminal device is obtained by the computer device, and if the area where the positioning information is located is in the statistic area set, the area where the positioning information is located is used as the target area to obtain the historical record index; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when the mapping relation is detected to be not existed in the history record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area. Through the data processing process, the generation of repeated data is reduced, and the accuracy of the data statistical result is improved.
Further, referring to fig. 6, fig. 6 is a schematic diagram of a specific flow of data processing provided in the embodiment of the present application. Specifically, the computer device performs statistics on data in the mapping relationship once every time a statistical time range passes, where the statistical time range includes a plurality of statistical time periods, and the data in the mapping relationship is obtained based on the association relationship between the terminal device and the area obtained in each statistical time period. In the embodiment of the present application, the association relationship between the terminal device and the area may be obtained by a message channel technology. The message channel technology is a technology for exchanging information among distributed applications, and the message channel technology is to store a positioning information log through a message queue and to reside the message queue on a memory or a disk until the positioning information log in the message queue is read. Specifically, the positioning information log generated by each terminal device is stored in a message queue, and the computer device acquires the positioning information log from the message queue to perform a data processing process on the positioning information log, in other words, the message queue is input data of the computer device.
The embodiment of the present application is described by taking a target terminal device within a statistical time period as an example, and specifically as shown in fig. 6, the method includes the following steps:
step S601, obtaining the positioning information of the target terminal device, and if the area where the positioning information is located is in the statistical area set, taking the area where the positioning information is located as the target area, and obtaining the history index.
Specifically, the computer device obtains the positioning information of the target terminal device, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is taken as the target area to obtain the history index, specifically, the computer device obtains the area where the positioning information of the target terminal device is located based on the area surrounded by the virtual geographic boundary, compares the area where the positioning information is located with the statistical area set, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is taken as the target area to obtain the history index, where reference may be made to specific description shown in step S401 in fig. 4. Optionally, if the area where the positioning information is located is not in the statistical area set, filtering and detecting the target terminal device according to the historical record index; if the filtering detects that the target terminal equipment does not exist in the historical record index, filtering and deleting the positioning information of the target terminal equipment; if the target terminal device exists in the history index, the target terminal device and the area where the target terminal device is located can be cached, and when the target terminal device is counted, the cached target terminal device and the area where the target terminal device is located are obtained, and the device position statistical information of the target terminal device is obtained. Optionally, the computer device may further directly filter and delete the positioning information of the target terminal device when the area where the positioning information is located is not in the statistical area set.
The mapping relation can be acquired through multiple requests, so that the success probability of acquiring the mapping relation is improved. And after the mapping relation is obtained, generating a historical record index according to the mapping relation. Specifically, requesting to acquire a mapping relation between the counted area and the terminal equipment, and recording request times i; i is a positive integer; if the mapping relation is successfully acquired at the ith time, generating a history record index according to the mapping relation; if the mapping relation acquisition at the ith time fails and i is less than N, a time interval t is passediThen, obtaining a mapping relation in the (i + 1) th request; n is a positive integer greater than 1; if the mapping relation is successfully acquired at the (i + 1) th time, generating a history record index according to the mapping relation; if the mapping relation acquisition fails at the (i + 1) th time and i +1 is N, the mapping relation acquisition is suspended. Specifically, a mapping relation is obtained for the first time, and if the first obtaining is successful, a history record index is generated according to the mapping relation; if the first acquisition fails, the time interval t passes1Then, the mapping relation is obtained for the second time, assuming t1If the second acquisition is successful, generating a historical record index according to the mapping relation; if the second acquisition fails, the time interval t passes2Then, the mapping relation is obtained for the third time, assuming t2If the third acquisition is successful, generating a history index according to the mapping relation; …, respectively; if it is the Nth timeAnd if the acquisition fails, the acquisition of the mapping relation is suspended. The success rate of acquiring the first mapping relation is 99%, the success rate of acquiring the second mapping relation is 99.99%, the success rate of acquiring the third mapping relation is 99.9999%, and the like, so that after the mapping relation is acquired through multiple requests, the success rate of acquiring the mapping relation is extremely high, and meanwhile, when the mapping relation is acquired twice, a certain time interval exists, so that data blockage caused by instant data stream peaks due to continuous acquisition of the mapping relation is avoided, and the stability of acquiring the mapping relation is improved.
Optionally, if the obtaining still fails after obtaining the mapping relationship for multiple times, the history index generated in the previous statistical time period may be obtained, and the obtained target terminal device and the target area are filtered and detected. Optionally, if the obtaining still fails after obtaining the mapping relationship for multiple times, sending a mapping relationship obtaining failure message to a manager, so that the manager can obtain the failure message based on the mapping relationship, detect the computer device to obtain abnormal data, and repair the abnormal data.
The mapping relation is stored in the device statistical database, the mapping relation in the device statistical database can be regarded as a global variable of a non-memory, each processing unit or data statistical unit in the computer device needs to acquire the mapping relation from the device statistical database, the mapping relation is acquired through multiple requests in the application, and a time interval exists between two adjacent times of acquisition of the mapping relation, so that the success rate of acquiring the mapping relation is improved, and the stability of acquiring the global variable of the non-memory is improved.
When the last statistical time period is over, acquiring a mapping relation, and generating or updating a historical record index based on the mapping relation; or before the positioning information of the target terminal device is acquired, the mapping relationship is acquired, and the history index is generated or updated according to the mapping relationship.
Further, the history index includes a status bit array and a history device identifier index, and the computer device includes a coarse filtering unit and a fine filtering unit, wherein the status bit array is stored in the coarse filtering unit, and the history device identifier index is stored in the fine filtering unit. Specifically, after the computer device obtains the target terminal device in the target area, the related information of the target terminal device is input into the coarse filtering processing unit, the state bit array is obtained based on the coarse filtering processing unit, and the historical device identification index is obtained based on the fine filtering processing unit. Optionally, before the data processing, the computer device obtains the state bit array based on the coarse filtering unit, and obtains the historical device identifier index based on the fine filtering unit, so that in the data processing process, the state bit array is stored in the coarse filtering unit, and the historical device identifier index is stored in the fine filtering unit. Specifically, when the current statistical time range starts, if the status bit array and the historical device identifier index are empty, it indicates that there is no target association relationship in the historical record index, and step S606 is executed, where in each statistical time period, the computer device updates the historical record index according to the obtained statistical information, in other words, the historical record index obtained in this embodiment is updated in the last statistical time period. The coarse filtering unit may include a plurality of coarse filtering sub-units, and the fine filtering unit may include a plurality of fine filtering sub-units, which may specifically refer to specific descriptions about the coarse filtering unit and the fine filtering unit in fig. 3.
Optionally, when the target terminal device and the target area where the target terminal device is located are obtained through the message queue, the computer device obtains a positioning information log of the target terminal device in the message queue, obtains positioning information in the positioning information log, and determines that the target terminal device is located in the target area based on the positioning information. Or, the positioning information log may include the target terminal device, the positioning information of the target terminal device, and a target area corresponding to the positioning information, and at this time, the computer device may directly analyze the positioning information log to obtain the target terminal device and the target area.
Step S602, a target association relationship between the target terminal device and the target area is obtained.
Specifically, the computer device obtains the positioning information of the target terminal device according to the positioning information of the target terminal device, determines the target area where the target terminal device is located according to the positioning information, and determines that a target association exists between the target terminal device and the target area, which may be referred to specifically as the description in step S402 in fig. 4, and details are not repeated here.
Step S603, detecting whether the target terminal device exists in the status bit array.
Specifically, after obtaining the target association relationship, the computer device detects whether the target association relationship is in the history index according to the history index, and specifically, the computer device detects whether the target terminal device is in the state bit array by first passing through the state bit array. Specifically, the computer device obtains a target state hash value of the target terminal device, and obtains a state bit array, where the state bit array includes a plurality of state values, and the state values are generated according to the state hash value of the terminal device in the mapping relationship; acquiring a target state value of the target terminal equipment from the state bit array according to the target state hash value; if the target state value has at least one missing state value, determining that no target association relationship exists in the mapping relationship, and executing step S605; if the target status values are all valid status values, go to step S604.
When the target state value of the target terminal device is obtained from the state bit array according to the target state hash value, specifically, the state bit array is obtained, and the corresponding target state position of the target state hash value in the state bit array is obtained; and acquiring the state value at the target state position in the state bit array as the target state value of the target terminal equipment. If the mapping relation is null, an initial state bit array is created, the state values of the initial state bit array are set to be missing state values, and a state bit array is obtained.
Specifically, the state bit array is generated based on at least one hash function, the state bit array is a binary array, and the binary array is a valid state value and a missing state value, respectively, it is assumed that the length of the state bit array is m, the number of the at least one hash function is k, and m and k are positive integers. When the mapping relation is null, an initial state bit array is created, and m state values in the initial state bit array are all set as missing state values. When the state bit array is generated according to the mapping relation, the terminal equipment in the mapping relation is obtained, k state hash values of each terminal equipment are obtained according to k hash functions, the k state hash values respectively correspond to k state positions in the state bit array, and the state values at the k state positions are updated to be effective state values so as to generate the updated state bit array. The state position may be a state hash value obtained according to a hash function, or may be obtained by conversion according to the state hash value, such as a conversion method of remainder. After acquiring the target terminal device, acquiring k target state hash values of the target terminal device according to k hash functions, acquiring k target state positions corresponding to the k target state hash values in a state bit array, acquiring k target state values corresponding to the target terminal device according to the k target state positions, and determining that the target terminal device exists in the state bit array if the k target state values are all valid state values; and if the missing state value exists in the k target state values, determining that the target terminal equipment does not exist in the state bit array.
Specifically, referring to fig. 7, fig. 7 is a schematic diagram of a state bit array generation and filtering scenario provided in the embodiment of the present application. As shown in FIG. 7, assume that there are 3 elements { x, y, z } in the mapping, and that k is 3, the valid state value is 1, and the missing state value is 0. When each state value in the state bit array is a missing state value 0, obtaining 3 state hash values of an element x through 3 hash functions, wherein the 3 state hash values respectively correspond to a bit 1, a bit 5 and a bit 13 in the state bit array, and setting the state values at the bit 1, the bit 5 and the bit 13 in the state bit array as 1; obtaining 3 state hash values of the element y through 3 hash functions, wherein the 3 state hash values respectively correspond to a bit 4, a bit 11 and a bit 16 in a state bit array, and the state values at the bit 4, the bit 11 and the bit 16 in the state bit array are set to be 1; obtaining 3 state hash values of the element z through 3 hash functions, where the 3 state hash values respectively correspond to bit 3, bit 5, and bit 11 in the state bit array, and setting the state values at bit 3, bit 5, and bit 11 in the state bit array to 1, thereby obtaining the state bit array shown in fig. 7. Since the subscript of the state bit array starts with 0, the state position is counted from 0, and optionally, the state position may also be counted from 1. Wherein, assuming that the status bit array is recorded as stateBitArray [ ], "stateBitArray [1] ═ 1 is executed according to the element x; statbistarray [5] ═ 1; stattibirelay [13] ═ 1 "; performing "stateBitArray [4] ═ 1 according to element y; statbistarray [11] ═ 1; statebirelay [16] ═ 1 "; performing "stateBitArray [3] ═ 1 according to element z; statbistarray [5] ═ 1; statbistarray [11] ═ 1 ″ to obtain the status bit array shown in fig. 7, where the process may also be implemented based on other programming languages, without limitation.
When the element f is obtained, obtaining 3 state hash values of the element f through the 3 hash functions, wherein the 3 state hash values respectively correspond to bit 3, bit 4 and bit 5 in a state bit array, and when the state values at bit 3, bit 4 and bit 5 in the state bit array are all valid state values 1, determining that the element f exists in the state bit array, and actually, the element f does not exist in the mapping relation; when the element w is obtained, obtaining 3 state hash values of the element w through the 3 hash functions, where the 3 state hash values respectively correspond to bit 4, bit 13, and bit 15 in the state bit array, obtaining that the state values at bit 4 and bit 13 in the state bit array are both valid state values 1, and the state value at bit 15 is a missing state value 0, and determining that the element w does not exist in the state bit array. It can be seen that, after 3 state values are obtained through the 3 hash functions, if there is a missing state value 0 in the 3 state values, it can be determined that the elements corresponding to the 3 state values are not necessarily in the state bit array; when the 3 state values are all valid state values 1, the elements corresponding to the 3 state values are not necessarily in the state bit array, and therefore a certain misjudgment rate may exist, so that when it is determined that the target terminal device exists in the state bit array, the target terminal device does not necessarily exist in the mapping relationship, step S604 is executed, and the target association relationship between the target terminal device and the target area is further filtered and detected through the historical device identification index.
Step S604, detecting whether a target identifier mapping relationship exists in the historical device identifier index.
Specifically, if the target state values are all valid state values, acquiring a target terminal device identifier of the target terminal device, and acquiring a historical device identifier index, wherein the historical device identifier index is generated by an identifier mapping relation between an area and the terminal device identifier; filtering and detecting a target identifier mapping relation between a target terminal device identifier and a target area identifier of a target area according to the historical device identifier index; if filtering detects that no target identifier mapping relation exists in the historical device identifier index, determining that no target association relation exists in the mapping relation, and executing the step S606; if the filtering detects that the target identifier mapping relationship exists in the historical device identifier index, it is determined that the target association relationship exists in the mapping relationship, and step S607 is executed.
Optionally, the terminal device identifier may be a unique identifier such as a factory identification code of the corresponding terminal device, or may be a hash value of the corresponding terminal device, which is not limited herein. When the identifier of the terminal device is the hash value of the terminal device, the computer device obtains the device hash value of the target terminal device, performs filtering detection on the target identifier mapping relationship between the device hash value and the target area identifier of the target area according to the historical device identifier index, determines that no target association relationship exists in the mapping relationship if it is detected that no target identifier mapping relationship exists in the historical device identifier index, and executes step S606, wherein the hash algorithm for generating the device hash value is the same as the hash algorithm for generating the hash value of the terminal device in the historical device identifier index.
Specifically, a target area identifier of the target area is obtained, and an equipment identifier list corresponding to the target area identifier is obtained from a historical equipment identifier index; if the equipment identification list comprises the target terminal equipment identification, determining that a target identification mapping relation exists in the historical equipment identification index; and if the equipment identification list does not comprise the target terminal equipment identification, determining that the target identification mapping relation does not exist in the historical equipment identification index. For example, the history device identifier index is "area 1| [ terminal device 1, terminal device 3, terminal device 5 ]; the method includes the steps of obtaining that a target terminal device is a terminal device 4 and a target area is an area 1, namely, a target association relationship is an association relationship between the terminal device 4 and the area 1, obtaining a device identifier list corresponding to the area 1 from a historical device identifier index according to the area 1, wherein the device identifier list is 'terminal device 1, terminal device 3 and terminal device 5', searching the target terminal device 'terminal device 4' in the device identifier list, determining that the device identifier list does not include a target terminal device identifier, further determining that a target identifier mapping relationship does not exist in the historical device identifier index, namely, the target association relationship does not exist in the mapping relationship, and executing step S606.
In step S605, the status bit array is updated.
Specifically, in the state bit array, the state value at the target state position is updated to the valid state value, and the updated state bit array is obtained. Specifically, when it is determined that the target terminal device does not exist in the state bit array, it indicates that the target terminal device does not exist in the state bit array, and at this time, when k hash functions exist and k target state positions corresponding to the target terminal device are obtained based on the k hash functions, state values at the k target state positions in the state bit array are all updated to valid state values. Optionally, because the status bit array has a certain false positive rate, when the target terminal device does not appear in the mapping relationship, it may be possible to obtain that the target terminal device exists in the status bit array, and further, the target identifier mapping relationship between the target terminal device and the target area identifier of the target area is further filtered and detected according to the historical device identifier index. When the target terminal equipment is determined to exist in the state bit array, the state value representing the corresponding target state position of the target terminal equipment in the state bit array is an effective state value, so that even if the target identification mapping relation does not exist in the historical equipment identification index, the state bit array is updated according to the target terminal equipment, and the state bit array is consistent before and after updating, therefore, the state bit array can be updated according to the target terminal equipment only when the target terminal equipment does not exist in the state bit array, and the workload can be reduced.
For example, referring to fig. 7, the element f does not exist in the mapping relationship, and the state values at the bit 3, the bit 4, and the bit 5 in the state bit array are all set to the valid state value 1 according to the 3 state hash values obtained by the element f, and the state values at the bit 3, the bit 4, and the bit 5 in the state bit array are originally the valid state value 1, so that the invalidation is performed, and therefore, when the element f exists in the state bit array, it is directly detected whether the target identifier mapping relationship exists in the historical apparatus identifier index through step S604 without updating the state bit array. When it is determined that the element w does not exist in the state bit array, the state values at the bit 4, the bit 13 and the bit 15 corresponding to the element w in the state bit array are set as the valid state value 1.
Step S606, when it is detected that no target association relationship exists in the mapping relationship according to the history index, adding the target association relationship between the target terminal device and the target area to the mapping relationship.
Specifically, when it is detected that no target association relationship exists in the mapping relationship according to the history index, the target association relationship between the target terminal device and the target area is added to the mapping relationship. Specifically, reference may be made to the description shown in step S403 in fig. 4. For example, assuming that the mapping relationship includes "area 1| [ terminal device 1, terminal device 3, terminal device 5, … ]" and "area 2| [ terminal device 3, terminal device 4, … ]" and the like, when the target terminal device is terminal device 5 and the target area is area 2, assuming that the device identifier list corresponding to area 2 does not include terminal device 5, adding the target association relationship to the mapping relationship, and obtaining "area 2| [ terminal device 3, terminal device 4, terminal device 5, … ]"; when the target terminal device is the terminal device 5 and the target area is the area 3, and it is assumed that the area 3 is not included in the mapping relationship, the target association relationship is added to the mapping relationship, so as to obtain "area 3| [ terminal device 5 ]".
Optionally, the system network time may also be obtained, and the system network time is added to the mapping relationship, where the system network time corresponds to the target association relationship and is used to indicate the latest time of occurrence of the target association relationship. The system network time may be the current network time, or may be the time carried in the positioning information received by the computer device from the target terminal device, or the start time of the current statistical time period.
And step S607, when the target incidence relation is detected in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area.
Specifically, when a target association relation is detected in a mapping relation according to a historical record index, acquiring first system network time; acquiring historical record information associated with the target area and the target terminal equipment from the mapping relation; and updating the target recording time in the historical recording information to the first system network time. At this time, the target recording time in the history information is used to indicate the time when the target association relationship has appeared most recently, and the history information includes the target association relationship between the target terminal device and the target area, the target recording time, and the like. For example, when the target recording time is "2019-01-0112: 00: 00 ", the time when the target association relation appears latest is" 2019-01-0112: 00: 00 ", i.e. in this" 2019-01-0112: 00: the target association relationship does not appear in the period from the time point 00' to the current network time.
Or when the target incidence relation exists in the mapping relation according to the historical record index, acquiring a target time set corresponding to the target incidence relation from the mapping relation; and acquiring the network time of the second system, and adding the network time of the second system to the target time set. At this time, each recording time included in the target time set represents each time point at which the target association relationship appears. For example, when "2019-01-0112: 00: 00, 2019-01-0112: 01: 00, 2019-01-0112: 30: 00 ", this indicates that the target terminal device is in a state of" 2019-01-0112: 00: 00 "," 2019-01-0112: 01: 00' and "2019-01-0112: 30: 00' at several points in time, they appear in the target region.
The first system network time or the second system network time may be the current network time, or may be the time carried in the positioning information received by the computer device from the target terminal device, or the start time of the current statistical time period.
Step S608, obtaining statistical information according to the mapping relationship.
Specifically, a mapping relation is obtained, and a history area identifier associated with the target terminal device identifier is obtained according to the mapping relation; acquiring the number of historical region identifications as the number of regions; and taking the area number and the historical area identification as the device position statistical information of the target terminal device. The number of the areas is the number of the areas where the target terminal device appears and stored in the mapping relation within the statistical time range; the historical region identifier is used for representing the region where the target terminal device appears in the statistical time range stored in the mapping relation; the device location statistics information is used to indicate that the target terminal device stored in the mapping relationship appears in several regions within a statistical time range, and the several regions are respectively which regions. For example, the history area identifiers associated with the target terminal device in the mapping relationship are acquired as "area 1, area 3 (target area), area 4 and area 8", the number of the history area identifiers is 4, that is, the number of the areas is 4, the device position statistical information of the target terminal device is obtained according to the number of the areas 4 and the history area identifiers, the device position statistical information is used for indicating that the target terminal device appears in 4 areas, and the 4 areas are "area 1, area 3, area 4 and area 8", respectively.
The mapping relationship is stored in the device statistics database, and the mapping relationship is equivalent to a non-memory global variable for a data statistics unit of the statistics mapping relationship, and when the data statistics unit obtains the mapping relationship, the data statistics unit may obtain the mapping relationship through multiple requests to improve stability of obtaining the non-memory global variable, which is specifically described in step S601, and details are not repeated here.
Specifically, the computer device may further obtain a target area identifier of the target area; acquiring a device set associated with the target area identifier according to the mapping relation, wherein the device set comprises historical terminal devices; and acquiring the number of historical terminal devices as the device number statistical information of the target area, wherein the statistical information is used for representing the number of terminal devices appearing in the target area so as to represent the flow of people in the target area. Optionally, the number of historical terminal devices may also be used as the device number; and taking the device number and the historical terminal devices as the device number statistical information of the target area, wherein the device number statistical information can represent the pedestrian volume in the target area and the terminal devices specifically appearing in the target area. The device number is the number of terminal devices appearing in the target area, and the historical terminal devices are the terminal devices appearing in the target area. For example, the association relationship associated with the target area identifier in the mapping relationship is "target area | [ terminal device 1, terminal device 2, terminal device 4 (target terminal device) and terminal device 7 ]", the device set associated with the target area identifier is obtained as "terminal device 1, terminal device 2, terminal device 4 and terminal device 7", the number of historical terminal devices included in the device set is 4, that is, the number of devices is 4, the device number statistical information of the target area is generated according to the device number 4 and the historical terminal devices, the device number statistical information is used to indicate that 4 terminal devices have appeared in the target area, the 4 terminal devices are respectively "terminal device 1, terminal device 2, terminal device 4 and terminal device 7", the device number statistical information can also be used to indicate that the device number is in the statistical time range, the traffic of the target area stored in the mapping relationship is 4, where "terminal device 1, terminal device 2, terminal device 4, and terminal device 7" have appeared in the target area.
Further, if the mapping relationship comprises a target time set, acquiring at least two recording times included in the target time set and a statistical time period of the mapping relationship; acquiring a time interval between adjacent recording times in at least two recording times, determining the time interval which is the same as the statistical time period as an effective time interval, and determining the recording time length of the target terminal device in the target area according to the total value of the effective time interval, wherein the recording time length can be used for representing the stay time length of the target terminal device in the target area. Optionally, in each statistical time period, the positioning information of the terminal device is obtained, and a product of the recording times and the statistical time period is obtained as a recording duration of the target terminal device in the target area according to the recording times of at least two recording times, where the recording times are used to indicate the counted times of the target terminal device in the target area, and each recording time indicates that the statistical time period in which the target terminal device is located in the target area. For example, assuming that the statistical time period is 1 minute, at least two target recording times are "2019-01-0112: 00: 00, 2019-01-0112: 01: 00, 2019-01-0112: 30: 00', according to the statistical time period and at least two target recording times, the recording time of the target terminal equipment in the target area is obtained to be 3 minutes.
Optionally, before executing the step, it may also be detected whether the statistical time range is over, if the statistical time range is over, the step is executed (step S608), and if the statistical time range is not over, the data processing procedure of the next statistical time period is started, that is, the above steps S601 to S607 are executed in the next statistical time period. Assuming a statistical time frame of 1 day, i.e. 0: 00: 00 to 24: 00: 00, then at 24: 00: 00, step S608 is executed, and only steps S601 to S607 are executed at other times.
Optionally, when the target terminal device enters the target area, the recording time of the target terminal device entering the target area may be added to the target time set, and when the target terminal device leaves the target area, the recording time of the target terminal device leaving the target area may be added to the target time set, so that each recording time in the target time set is sequentially two times as a group, and the entering time and the leaving time are respectively represented. At this time, assuming that the target time set includes 6 time points, the sum of the time interval between the first time point and the second time point, the time interval between the third time point and the fourth time point, and the time interval between the fifth time point and the sixth time point is the recording time length of the target terminal device in the target area. When the virtual geographic boundary is constructed through the geographic fence, a trigger mechanism can be added to the virtual geographic boundary, wherein the trigger mechanism comprises an entering trigger mechanism and an exiting trigger mechanism, and when the trigger mechanism is triggered, the time of the target terminal device in the target area is recorded.
Further, referring to fig. 8, fig. 8 is a schematic diagram of a data processing scenario provided in an embodiment of the present application. As shown in fig. 8, the computer device obtains a terminal device set 801, where the terminal device set 801 includes a plurality of terminal devices and an area where each terminal device is located, and performs filtering detection on the terminal device set 801 according to a status bit array 802 to obtain an unrecorded state device 803 and a recorded state device 804, where the filtering detection process refers to step S603 in fig. 6, and records terminal devices in the terminal device set 801 that do not belong to the status bit array 802 as the unrecorded state device 803, and records terminal devices that belong to the status bit array 802 as the recorded state device 804. The recorded status 804 is filtered and detected by the historical device identifier index 805, and specifically, as shown in step S604 in fig. 6, of the association relations corresponding to the recorded status devices 807, the association relation belonging to the historical device identifier index 805 is referred to as a recorded association relation 807, and the association relation not belonging to the historical device identifier index 805 is referred to as an unrecorded association relation 806. The unrecorded state device 803 and the unrecorded association 806 are added to the mapping 808. Optionally, if the mapping relation 808 includes the recording time, the mapping relation 808 is updated according to the recorded association relation 807.
Specifically, assume that the mapping relationship 8081 includes "area 3| [ time 1 of terminal device 2, time 1 of terminal device 4, and time 2 of terminal device 8 ]", where the mapping relationship 8081 includes recording time as an example, the computer device generates the state bit array and the historical device identifier index according to the mapping relationship 8081. Specifically, when the computer device generates the state bit array, it acquires that all the state values in the initial state bit array are missing state values 0, acquires the state hash value of the terminal device 2, determines the state position of the terminal device 2 in the initial state bit array to be 5 according to the state hash value of the terminal device 2, and sets the state value at the state position 5 in the initial state bit array to be an effective state value 1; acquiring a state hash value of the terminal device 4, determining that the state position of the terminal device 4 in an initial state bit array is 2 according to the state hash value of the terminal device 4, and setting the state value at the state position 2 in the initial state bit array as an effective state value 1; acquiring a state hash value of the terminal device 8, determining that the state position of the terminal device 8 in an initial state bit array is 7 according to the state hash value of the terminal device 8, and setting the state value at the state position 7 in the initial state bit array as an effective state value 1; thereby, a state bit array 8021 is obtained, and the state bit array 8021 is "0, 0, 1, 0, 0, 1, 0, 1, 0, 0". The status bit array 8021 is the specific data format and values of the status bit array 802. When the computer device generates the historical device identifier index, it acquires that the initial historical device identifier index is empty, adds the association relationship in the mapping relationship 8081 to the initial historical device identifier index, and generates the historical device identifier index 8051 "area 3| [ terminal device 2, terminal device 4, terminal device 8 ]". The historical device identifier index 8051 is a specific data format and value of the historical device identifier index 805.
When the computer device acquires the terminal device set 801, the terminal device set 801 includes an association relationship between the terminal device 1 and the area 1, an association relationship between the terminal device 2 and the area 3, an association relationship between the terminal device 3 and the area 1, and an association relationship between the terminal device 8 and the area 10. Acquiring a state hash value of the terminal device 1, determining that the state position of the terminal device 1 is 1 according to the state hash value of the terminal device 1, acquiring that the state value at the state position 1 in the state bit array 8021 is a missing state value 0, and determining that the state bit array does not include the terminal device 1; acquiring a state hash value of the terminal device 2, determining that the state position of the terminal device 2 is 5 according to the state hash value of the terminal device 2, acquiring that the state value at the state position 2 in the state bit array 8021 is an effective state value 1, and determining that the state bit array comprises the terminal device 2; acquiring a state hash value of the terminal device 3, determining that the state position of the terminal device 3 is 0 according to the state hash value of the terminal device 3, acquiring that the state value at the state position 0 in the state bit array 8021 is a missing state value 0, and determining that the state bit array does not include the terminal device 3; acquiring a state hash value of the terminal device 8, determining that the state position of the terminal device 8 is 7 according to the state hash value of the terminal device 8, acquiring that the state value at the state position 7 in the state bit array 8021 is an effective state value 1, and determining that the state bit array comprises the terminal device 8. Recording terminal equipment 1 and terminal equipment 3 which do not belong to the state bit array 802 in the terminal equipment set 801 as unrecorded state equipment 803; terminal device 2 and terminal device 8 belonging to the status bit array 802 in the terminal device set 801 are referred to as recorded status devices 804.
The computer device performs filtering detection on recorded state equipment 804 through a historical equipment identification index 805, the recorded state equipment 804 comprises an association relation between a terminal device 2 and an area 3 and an association relation between a terminal device 8 and an area 10, the recorded state equipment 804 is compared with the historical equipment identification index 8051' area 3| [ terminal device 2, terminal device 4, terminal device 8] ", it is determined that the historical equipment identification index 8051 comprises the association relation between the terminal device 2 and the area 3 but does not comprise the association relation between the terminal device 8 and the area 10, the association relation between the terminal device 2 and the area 3 is recorded as an association relation, and the association relation between the terminal device 8 and the area 10 is recorded as an unrecorded association relation 806. The computer device acquires the system network time 809 as time 3, takes the system network time 809 as recording time, and adds the unrecorded state device 803 "association between the terminal device 1 and the area 1, association between the terminal device 3 and the area 1", and the unrecorded association 806 "association between the terminal device 8 and the area 10" and recording time "time 3" to the mapping 808. Optionally, the recording time of the recorded association 807 can also be updated to the system network time 809. Updating the data 8081 "area 3| [ terminal device 2 time 1, terminal device 4 time 1, terminal device 8 time 2 ]" in the mapping relation 808 to data 8082 "area 3| [ terminal device 2 time 3, terminal device 4 time 1, terminal device 8 time 2 ]; region 1| [ terminal device 1 time 3, terminal device 3 time 3 ]; region 10| [ terminal device 8 time 3] ". Optionally, the system network time 809 may also be directly added to the time set in which the association relationship 807 has been recorded, where the data 8082 is "area 3| [ terminal device 2 (time 1, time 3), terminal device 4 time 1, terminal device 8 time 2 ]; region 1| [ terminal device 1 time 3, terminal device 3 time 3 ]; region 10| [ terminal device 8 time 3] ".
The computer device may further update the state bit array 802 according to the unrecorded state device 803, where the obtained unrecorded state device 803 includes the terminal device 1 and the terminal device 3, and the obtained state bit of the terminal device 1 is 1, the state bit of the terminal device 3 is 0, and both the state value at the state position 1 and the state value at the state position 0 in the state bit array 8021 are set to be valid state values 1, so that the state bit array 8021 is updated from "0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0. Similarly, the history device identifier index 805 is updated according to the unrecorded state device 803 and the unrecorded association relation 806, so that the history device identifier index 8051 is updated from "area 3| [ terminal device 2, terminal device 4, terminal device 8 ]" to "area 3| [ terminal device 2, terminal device 4, terminal device 8 ]; region 1| [ terminal device 1, terminal device 3 ]; area 10| [ terminal device 8] ".
According to the embodiment of the application, the positioning information of the target terminal equipment is obtained, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is used as the target area, and the historical record index is obtained; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when the mapping relation is detected to be not existed in the history record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area. Through the data processing process, the generation of repeated data is reduced, and the accuracy of the data statistical result is improved. Meanwhile, the target incidence relation is filtered and detected through the state bit array and the historical equipment identification index, so that the resource consumption is reduced, and the data processing efficiency is improved.
In the data processing process, billions or even more terminal devices may be acquired every day, near ten thousand or more regions to be counted may be acquired, the memory space actually occupied by the index reaches more than 10G, and therefore more processing subunits are allocated for processing, if each processing subunit stores the index, the total occupied memory space may reach 2T or more, so that the processing pressure of the computer device is too large, and even the processing requirement cannot be met, therefore, the size of the index is reduced through the state bit array, more processing subunits (for example, 190 processing subunits) are constructed to perform coarse filtering with the state bit array, and fewer processing subunits (for example, 10) are constructed to perform fine filtering with the historical device identification index. The state bit array maps the terminal equipment to the array with the fixed length, and the memory space occupied by the state bit array is greatly reduced. Although the historical device identification index still occupies a large amount of memory space, a large number of terminal devices can be filtered through coarse filtering, so that the number of the terminal devices for fine filtering through the historical device identification index is small, the data pressure of a processing subunit for fine filtering is reduced, and the loss of resources in the data processing process is reduced.
In brief, more processing subunits are provided for performing coarse filtering on the acquired terminal device, each processing subunit for coarse filtering stores a state bit array, the memory occupied by the state bit array is smaller, the number of the processing subunits for coarse filtering is larger, and the data pressure of each processing subunit for coarse filtering is reduced. After through the coarse filtration, can filter a lot of terminal equipment, carry out the fine filtration through the processing subunit that is used for the fine filtration to remaining terminal equipment after filtering, the quantity that needs to carry out the fine filtration this moment is less, consequently, the data pressure that is used for the processing subunit of fine filtration is also less. By combining coarse filtering and fine filtering, the memory space occupied by the historical record index is reduced, and the resource loss is reduced.
Further, referring to fig. 9, fig. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. The data processing means may be a computer program (including program code) running on a computer device, for example, an application software; the apparatus may be used to perform the corresponding steps in the methods provided by the embodiments of the present application. As shown in fig. 9, the data processing apparatus 90 may be used in the computer device in the embodiment corresponding to fig. 4 or fig. 6, specifically, the data processing apparatus 90 may include: a first obtaining module 11, a second obtaining module 12, a relationship adding module 13 and a relationship updating module 14.
A first obtaining module 11, configured to obtain location information of a target terminal device, and if an area where the location information is located is in a statistical area set, take the area where the location information is located as a target area, and obtain a history index; the historical record index is generated according to the mapping relation between the counted area and the terminal equipment;
a second obtaining module 12, configured to obtain a target association relationship between the target terminal device and the target area;
a relation adding module 13, configured to add, when it is detected that the target association relation does not exist in the mapping relation according to the history index, a target association relation between the target terminal device and the target area to the mapping relation;
a relationship updating module 14, configured to update the mapping relationship according to a target association relationship between the target terminal device and the target area when it is detected that the target association relationship exists in the mapping relationship according to the history index.
Wherein, the history index comprises a state bit array;
the above apparatus 90 further comprises:
a third obtaining module 15, configured to obtain a target state hash value of the target terminal device, and obtain the state bit array, where the state bit array includes a plurality of state values, and the state values are generated according to the state hash value of the terminal device in the mapping relationship;
a state obtaining module 16, configured to obtain a target state value of the target terminal device from the state bit array according to the target state hash value;
a first determining module 17, configured to determine that the target association relationship does not exist in the mapping relationship if at least one missing state value exists in the target state value.
Wherein, the history index also comprises a history equipment identification index; the above apparatus 90 further comprises:
a fourth obtaining module 18, configured to obtain a target terminal device identifier of the target terminal device and obtain the historical device identifier index if the target state values are all valid state values; the historical equipment identifier index is generated by an identifier mapping relation between the area identifier and the terminal equipment identifier;
a first filtering module 19, configured to perform filtering detection on a target identifier mapping relationship between the target terminal device identifier and the target area identifier of the target area according to the historical device identifier index;
a second determining module 20, configured to determine that the target association relationship does not exist in the mapping relationship if it is detected by filtering that the target identifier mapping relationship does not exist in the historical device identifier index;
the second determining module 20 is further configured to determine that the target association relationship exists in the mapping relationship if the filtering detects that the target identifier mapping relationship exists in the historical device identifier index.
Wherein, the state acquiring module 16 includes:
an array obtaining unit 161, configured to obtain a corresponding target state position of the target state hash value in the state bit array;
a state obtaining unit 162, configured to obtain, in the state bit array, a state value at the target state position as a target state value of the target terminal device.
Wherein, the first filtering module 19 includes:
a hash obtaining unit 191 configured to obtain a target area identifier of the target area, and obtain an equipment identifier list corresponding to the target area identifier from the historical equipment identifier index;
a first determining unit 192, configured to determine that a target identifier mapping relationship between the target terminal device identifier and a target area identifier of the target area exists in the historical device identifier index if the device identifier list includes the target terminal device identifier;
a second determining unit 193, configured to determine that a target identifier mapping relationship between the target terminal device identifier and the target area identifier of the target area does not exist in the historical device identifier index if the device identifier list does not include the target terminal device identifier.
Wherein the apparatus 90 further comprises:
an index updating module 21, configured to update the state value at the target state position to a valid state value in the state bit array.
Wherein the apparatus 90 further comprises:
a creating module 22, configured to create an initial state bit array if the mapping relationship is null, and set a state value of the initial state bit array to a missing state value to obtain a state bit array.
Wherein, the target terminal device carries a target terminal device identifier, and the apparatus 90 further includes:
a history region obtaining module 23, configured to obtain the mapping relationship, and obtain a history region identifier associated with the target terminal device identifier according to the mapping relationship;
a first statistical module 24, configured to obtain the number of the historical region identifiers as a region number;
a first generating module 25, configured to use the number of areas and the history area identifier as the device location statistics information of the target terminal device.
Wherein the apparatus 90 further comprises:
a target area obtaining module 26, configured to obtain a target area identifier of the target area;
an equipment obtaining module 27, configured to obtain, according to the mapping relationship, an equipment set associated with the target area identifier, where the equipment set includes historical terminal equipment;
a second statistical module 28, configured to obtain the number of the historical terminal devices as device number statistical information of the target area.
Wherein, the relationship updating module 14 includes:
a time obtaining unit 141, configured to obtain a first system network time if it is detected that the target association relationship exists in the mapping relationship according to the history index;
a record obtaining unit 142, configured to obtain history information associated with the target area and the target terminal device from the mapping relationship;
a time updating unit 143, configured to update the target recording time in the history information to the first system network time.
The relationship updating module 14 further includes:
a set obtaining unit 144, configured to, if it is detected that the target association relationship exists in the mapping relationship according to the history index, obtain a target time set corresponding to the target association relationship from the mapping relationship;
the time adding unit 145 is configured to obtain a second system network time, and add the second system network time to the target time set.
Wherein the apparatus 90 further comprises:
a set obtaining module 29, configured to obtain at least two recording times included in the target time set and a statistical time period of the mapping relationship;
an interval obtaining module 30, configured to obtain a time interval between adjacent recording times of the at least two recording times, and determine a time interval that is the same as the statistical time period as an effective time interval;
a duration determining module 31, configured to determine the total value of the valid time intervals as a recording duration of the target terminal device in the target area.
Wherein the apparatus 90 further comprises:
the first obtaining module 11 is further configured to, if the area where the positioning information is located is not in the statistical area set, perform filtering detection on the target terminal device according to the history index;
an information deleting module 32, configured to filter and delete the positioning information of the target terminal device if it is detected by filtering that the target terminal device does not exist in the history index.
Wherein the apparatus 90 further comprises:
a relation obtaining module 33, configured to request to obtain a mapping relation between the counted area and the terminal device, and record a request number i; i is a positive integer;
an index generating module 34, configured to generate the history index according to the mapping relationship if the mapping relationship is successfully obtained at the ith time;
the relationship obtaining module 33 is further configured to obtain the relationship if the ith time is reachedIf the mapping fails and i is less than N, the time interval t passesiThen, the mapping relation is requested to be obtained in the (i + 1) th time; n is a positive integer greater than 1;
the index generating module 34 is further configured to generate the history index according to the mapping relationship if the mapping relationship is successfully obtained in the (i + 1) th time;
the exception handling module 35 is configured to suspend obtaining the mapping relationship if obtaining the mapping relationship at the (i + 1) th time fails and i +1 is N.
The embodiment of the application describes a data processing device, which acquires historical record indexes by acquiring positioning information of target terminal equipment and taking an area where the positioning information is located as a target area if the area where the positioning information is located is in a statistical area set; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when the mapping relation is detected to be not existed in the history record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area. Through the data processing process, the generation of repeated data is reduced, the accuracy of a data statistical result is improved, and the data processing efficiency is improved.
Referring to fig. 10, fig. 10 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 10, the computer device 1000 in the embodiment of the present application may include: one or more processors 1001, memory 1002, and input-output interface 1003. The processor 1001, the memory 1002, and the input/output interface 1003 are connected by a bus 1004. The memory 1002 is used for storing a computer program including program instructions, and the input/output interface 1003 is used for inputting and outputting data, including data interaction between each communication client and the event server, and data interaction between a user and each communication client; the processor 1001 is configured to execute program instructions stored in the memory 1002 to perform the following operations:
acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in a statistical area set, taking the area where the positioning information is located as a target area to acquire a historical record index; the historical record index is generated according to the mapping relation between the counted area and the terminal equipment;
acquiring a target association relation between the target terminal equipment and the target area;
when the mapping relation is detected to be not existed in the mapping relation according to the historical record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation;
and updating the mapping relation according to the target association relation between the target terminal equipment and the target area when the target association relation is detected to exist in the mapping relation according to the historical record index.
In some possible embodiments, the processor 1001 may be a Central Processing Unit (CPU), and the processor may be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 1002 may include both read-only memory and random-access memory, and provides instructions and data to the processor 1001 and the input/output interface 1003. A portion of the memory 1002 may also include non-volatile random access memory. For example, the memory 1002 may also store device type information.
In a specific implementation, the computer may execute, through each built-in functional module, the implementation manner provided in each step in fig. 4 or fig. 6, which may be specifically referred to the implementation manner provided in each step in fig. 4 or fig. 6, and is not described herein again.
The embodiment of the present application provides a computer device, including: the processor, the input/output interface and the memory, and the processor acquires the computer instructions in the memory, and executes the steps of the method shown in fig. 4 or fig. 6 to perform data processing operation. With computer instructions in the memory, the processor performs the steps of: acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in the statistical area set, taking the area where the positioning information is located as a target area and acquiring a historical record index; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment; acquiring a target association relation between target terminal equipment and a target area; when the mapping relation is detected to be not existed in the history record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation; and when the target incidence relation exists in the mapping relation according to the historical record index, updating the mapping relation according to the target incidence relation between the target terminal equipment and the target area. Through the data processing process, the generation of repeated data is reduced, the accuracy of a data statistical result is improved, and the data processing efficiency is improved.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by the processor, the data processing method provided in each step in fig. 4 or fig. 6 may be implemented, for specific reference, the implementation manner provided in each step in fig. 4 or fig. 6 may be referred to, and details of this implementation manner are not described herein again. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application. By way of example, program instructions may be deployed to be executed on one computer device or on multiple computer devices at one site or distributed across multiple sites and interconnected by a communication network.
The computer-readable storage medium may be the data processing apparatus provided in any of the foregoing embodiments or an internal storage unit of the computer, such as a hard disk or a memory of the computer. The computer readable storage medium may also be an external storage device of the computer, such as a plug-in hard disk, a smart card (SMC), a Secure Digital (SD) card, a flash card (flash card), and the like, which are provided on the computer. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the computer. The computer-readable storage medium is used for storing the computer program and other programs and data required by the computer. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
The term "comprises" and any variations thereof in the description and claims of the embodiments of the present application and in the drawings is intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or apparatus that comprises a list of steps or elements is not limited to the listed steps or modules, but may alternatively include other steps or modules not listed or inherent to such process, method, apparatus, product, or apparatus.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The method and the related apparatus provided by the embodiments of the present application are described with reference to the flowchart and/or the structural diagram of the method provided by the embodiments of the present application, and each flow and/or block of the flowchart and/or the structural diagram of the method, and the combination of the flow and/or block in the flowchart and/or the block diagram can be specifically implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block or blocks.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the scope of the present application, so that the present application is not limited thereto, and all equivalent variations and modifications can be made to the present application.

Claims (15)

1. A method of data processing, the method comprising:
acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in a statistical area set, taking the area where the positioning information is located as a target area to acquire a historical record index; the historical record index is generated according to the mapping relation between the counted area and the terminal equipment;
acquiring a target association relation between the target terminal equipment and the target area;
when the mapping relation is detected to be not existed in the mapping relation according to the historical record index, adding the target association relation between the target terminal equipment and the target area into the mapping relation;
and when the target association relationship exists in the mapping relationship according to the historical record index, updating the mapping relationship according to the target association relationship between the target terminal equipment and the target area.
2. The method of claim 1, wherein the history index comprises an array of state bits;
the method further comprises the following steps:
acquiring a target state hash value of the target terminal equipment, and acquiring a state bit array, wherein the state bit array comprises a plurality of state values, and the state values are generated according to the state hash value of the terminal equipment in the mapping relation;
acquiring a target state value of the target terminal equipment from the state bit array according to the target state hash value;
and if at least one missing state value exists in the target state value, determining that the target association relation does not exist in the mapping relation.
3. The method of claim 2, wherein the history index further comprises a history device identification index;
the method further comprises the following steps:
if the target state values are all valid state values, acquiring a target terminal device identifier of the target terminal device, and acquiring the historical device identifier index; the historical equipment identification index is generated by an identification mapping relation between the area identification and the terminal equipment identification;
according to the historical equipment identification index, filtering and detecting a target identification mapping relation between the target terminal equipment identification and the target area identification of the target area;
if filtering detects that the target identifier mapping relation does not exist in the historical equipment identifier index, determining that the target association relation does not exist in the mapping relation;
and if the target identifier mapping relation exists in the historical equipment identifier index, the target association relation exists in the mapping relation.
4. The method of claim 2, wherein said obtaining the target state value of the target terminal device from the state bit array according to the target state hash value comprises:
acquiring a corresponding target state position of the target state hash value in the state bit array;
and acquiring the state value at the target state position in the state bit array as the target state value of the target terminal equipment.
5. The method as claimed in claim 3, wherein said filtering and detecting a target identifier mapping relationship between the target terminal device identifier and the target area identifier of the target area according to the historical device identifier index comprises:
acquiring a target area identifier of the target area, and acquiring an equipment identifier list corresponding to the target area identifier from the historical equipment identifier index;
if the device identifier list comprises the target terminal device identifier, determining that a target identifier mapping relation between the target terminal device identifier and the target area identifier exists in the historical device identifier index;
and if the equipment identification list does not comprise the target terminal equipment identification, determining that a target identification mapping relation between the target terminal equipment identification and the target area identification does not exist in the historical equipment identification index.
6. The method of claim 1, wherein the target terminal device carries a target terminal device identity, the method further comprising:
acquiring the mapping relation, and acquiring a history area identifier associated with the target terminal equipment identifier according to the mapping relation;
acquiring the number of the historical region identifications as the number of regions;
and taking the area number and the historical area identification as the device position statistical information of the target terminal device.
7. The method of claim 1, wherein the method further comprises:
acquiring a target area identifier of the target area;
acquiring a device set associated with the target area identifier according to the mapping relation, wherein the device set comprises historical terminal devices;
and acquiring the number of the historical terminal devices as the device number statistical information of the target area.
8. The method of claim 1, wherein when it is detected that the target association exists in the mapping relationship according to the history index, updating the mapping relationship according to the target association relationship between the target terminal device and the target area comprises:
if the target association relation exists in the mapping relation according to the historical record index, acquiring first system network time;
acquiring historical record information associated with the target area and the target terminal equipment from the mapping relation;
and updating the target recording time in the historical recording information to the first system network time.
9. The method of claim 1, wherein when it is detected that the target association exists in the mapping relationship according to the history index, updating the mapping relationship according to the target association relationship between the target terminal device and the target area comprises:
if the target association relationship exists in the mapping relationship according to the historical record index, acquiring a target time set corresponding to the target association relationship from the mapping relationship;
and acquiring second system network time, and adding the second system network time to the target time set.
10. The method of claim 9, wherein the method further comprises:
acquiring at least two recording times included in the target time set and a statistical time period of the mapping relation;
acquiring a time interval between adjacent recording times in the at least two recording times, and determining a time interval which is the same as the statistical time period as an effective time interval;
and determining the total value of the effective time interval as the recording duration of the target terminal equipment in the target area.
11. The method of claim 1, wherein the method further comprises:
if the area where the positioning information is located is not in the statistical area set, filtering and detecting the target terminal equipment according to the historical record index;
and if the filtering detects that the target terminal equipment does not exist in the historical record index, filtering and deleting the positioning information of the target terminal equipment.
12. The method of claim 1, wherein prior to the step of obtaining location information of a target terminal device, the method further comprises:
requesting to acquire the mapping relation between the counted area and the terminal equipment, and recording the request times i; i is a positive integer;
if the mapping relation is successfully acquired at the ith time, generating the historical record index according to the mapping relation;
if the mapping relation acquisition at the ith time fails and i is less than N, a time interval t is passediThen, the mapping relation is requested to be obtained in the (i + 1) th time; n is a positive integer greater than 1;
if the mapping relation is successfully acquired at the (i + 1) th time, generating the historical record index according to the mapping relation;
and if the mapping relation is failed to be acquired at the (i + 1) th time and i +1 is N, the acquisition of the mapping relation is suspended.
13. A data processing apparatus, characterized in that the apparatus comprises:
the first acquisition module is used for acquiring positioning information of target terminal equipment, and if the area where the positioning information is located is in the statistical area set, the area where the positioning information is located is used as a target area to acquire a historical record index; the historical record index is generated according to the mapping relation between the statistical area and the terminal equipment;
the second acquisition module is used for acquiring a target association relation between the target terminal equipment and the target area;
a relation adding module, configured to add a target association relation between the target terminal device and the target area to the mapping relation when it is detected that the target association relation does not exist in the mapping relation according to the history index;
and the relation updating module is used for updating the mapping relation according to the target association relation between the target terminal equipment and the target area when the target association relation is detected to exist in the mapping relation according to the historical record index.
14. A computer device comprising a processor, a memory, an input output interface;
the processor is connected to the memory and the input/output interface, respectively, wherein the input/output interface is used for inputting and outputting data, the memory is used for storing program codes, and the processor is used for calling the program codes to execute the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions which, when executed by a processor, perform the method according to any one of claims 1-12.
CN202010166886.2A 2020-03-11 2020-03-11 Data processing method and device, computer and readable storage medium Active CN111382334B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010166886.2A CN111382334B (en) 2020-03-11 2020-03-11 Data processing method and device, computer and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010166886.2A CN111382334B (en) 2020-03-11 2020-03-11 Data processing method and device, computer and readable storage medium

Publications (2)

Publication Number Publication Date
CN111382334A true CN111382334A (en) 2020-07-07
CN111382334B CN111382334B (en) 2022-05-13

Family

ID=71215335

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010166886.2A Active CN111382334B (en) 2020-03-11 2020-03-11 Data processing method and device, computer and readable storage medium

Country Status (1)

Country Link
CN (1) CN111382334B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112948389A (en) * 2021-03-05 2021-06-11 上海上讯信息技术股份有限公司 MD 5-based database table data comparison method and equipment
CN113194039A (en) * 2021-04-23 2021-07-30 京东数科海益信息科技有限公司 Method and device for segmenting system data flow, electronic equipment and storage medium
CN113807260A (en) * 2021-09-17 2021-12-17 北京百度网讯科技有限公司 Data processing method and device, electronic equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120078922A1 (en) * 2010-09-29 2012-03-29 International Business Machines Corporation Data reorganization
US20140032271A1 (en) * 2012-07-20 2014-01-30 Environmental Systems Research Institute (ESRI) System and method for processing demographic data
CN104732756A (en) * 2013-12-24 2015-06-24 中兴通讯股份有限公司 Method for conducting public transportation planning by utilizing mobile communication data mining
CN106991146A (en) * 2017-03-24 2017-07-28 京信通信技术(广州)有限公司 People information statistical method and system
CN107105055A (en) * 2017-05-23 2017-08-29 上海双微导航技术有限公司 A kind of application and geography information matching system based on positional information
CN110809236A (en) * 2019-09-11 2020-02-18 腾讯科技（深圳）有限公司 Area positioning method, area positioning device, computer equipment and computer readable storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120078922A1 (en) * 2010-09-29 2012-03-29 International Business Machines Corporation Data reorganization
US20140032271A1 (en) * 2012-07-20 2014-01-30 Environmental Systems Research Institute (ESRI) System and method for processing demographic data
CN104732756A (en) * 2013-12-24 2015-06-24 中兴通讯股份有限公司 Method for conducting public transportation planning by utilizing mobile communication data mining
CN106991146A (en) * 2017-03-24 2017-07-28 京信通信技术(广州)有限公司 People information statistical method and system
CN107105055A (en) * 2017-05-23 2017-08-29 上海双微导航技术有限公司 A kind of application and geography information matching system based on positional information
CN110809236A (en) * 2019-09-11 2020-02-18 腾讯科技（深圳）有限公司 Area positioning method, area positioning device, computer equipment and computer readable storage medium

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112948389A (en) * 2021-03-05 2021-06-11 上海上讯信息技术股份有限公司 MD 5-based database table data comparison method and equipment
CN113194039A (en) * 2021-04-23 2021-07-30 京东数科海益信息科技有限公司 Method and device for segmenting system data flow, electronic equipment and storage medium
CN113807260A (en) * 2021-09-17 2021-12-17 北京百度网讯科技有限公司 Data processing method and device, electronic equipment and storage medium
CN113807260B (en) * 2021-09-17 2022-07-12 北京百度网讯科技有限公司 Data processing method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN111382334B (en) 2022-05-13

Similar Documents

Publication Publication Date Title
CN111382334B (en) Data processing method and device, computer and readable storage medium
CN112650762B (en) Data quality monitoring method and device, electronic equipment and storage medium
CN108776934B (en) Distributed data calculation method and device, computer equipment and readable storage medium
CN106953740B (en) Processing method, client, server and system for page access data in application
CN111181799B (en) Network traffic monitoring method and equipment
CN107038620B (en) Information pushing and device based on taxi taking preference of user
CN111597388B (en) Sample collection method, device, equipment and medium based on distributed system
CN112434039A (en) Data storage method, device, storage medium and electronic device
Yazji et al. Efficient location aware intrusion detection to protect mobile devices
EP2897088A1 (en) Method and apparatus for optimum spatial clustering
CN111339183A (en) Data processing method, edge node, data center and storage medium
CN111651170B (en) Instance dynamic adjustment method and device and related equipment
CN111680104B (en) Data synchronization method, device, computer equipment and readable storage medium
CN113342603A (en) Alarm data processing method and device, computer equipment and storage medium
CN110807050B (en) Performance analysis method, device, computer equipment and storage medium
CN111666298A (en) Method and device for detecting user service class based on flink, and computer equipment
CN115174233A (en) Network security analysis method, device, system and medium based on big data
CN113987492A (en) Method and device for determining alarm event
CN110191097A (en) Detection method, system, equipment and the storage medium of login page safety
CN112506926A (en) Monitoring data storage and query method and corresponding device, equipment and medium
CN112561593A (en) Point distribution control method, device, equipment and medium
CN111061719A (en) Data collection method, device, equipment and storage medium
CN117857465B (en) Data processing method, device, equipment, storage medium and program product
CN115941209B (en) Block chain decentralization trusted data acquisition method and system
CN113722194B (en) Log statistics method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40025774

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant