CN112749166A - Service data processing method, device, equipment and storage medium - Google Patents
Service data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112749166A CN112749166A CN202110058355.6A CN202110058355A CN112749166A CN 112749166 A CN112749166 A CN 112749166A CN 202110058355 A CN202110058355 A CN 202110058355A CN 112749166 A CN112749166 A CN 112749166A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- hash
- stored
- fragment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims description 12
- 239000012634 fragment Substances 0.000 claims abstract description 51
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000004590 computer program Methods 0.000 claims description 7
- 238000013467 fragmentation Methods 0.000 claims description 3
- 238000006062 fragmentation reaction Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000005192 partition Methods 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a method, a device, equipment and a storage medium for processing service data, wherein the method comprises the following steps: acquiring service data from an original server; storing data meeting the service conditions in the service data to a new server according to a mode of combining the service attribute hash and the data component hash; acquiring stored data from a new server in a multi-fragment concurrent traversal mode; and under the condition that the acquired stored data meet the sending requirement, sending the stored data to a user account corresponding to the stored data. The service data can be subjected to hash storage in a hash combination routing mode, hash collision of the data is avoided, the stored data is traversed in a multi-fragment concurrent traversal mode, the data with the same service attribute can be rapidly traversed, and traversal performance is improved.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a method, an apparatus, a device, and a storage medium for processing service data.
Background
The data traversal refers to scanning all or part of the existing data, and transmitting the obtained data to the next node for processing, and the existing data traversal scheme generally includes traditional MYSQL data traversal and data traversal based on a memory carrier.
In the process of implementing the invention, the following technical problems are found in the prior art: in a scene of mass data, a traditional MYSQL data traversal mode has the defect that the performance does not reach the standard, and even the traversal of the mass data cannot be realized, for example, when the data volume of a single table reaches 500 ten thousand, the performance is sharply reduced, so that the complete traversal cannot be realized. In a traversal mode based on a memory carrier, under the condition of thirty million business data, 8GB memory space is occupied, 800GB memory space is needed for trillion business data, and backup or state recording is needed at a hardware level, which requires additional overhead and results in higher cost.
Disclosure of Invention
In order to solve at least one of the above technical problems, embodiments of the present application provide the following solutions.
In a first aspect, an embodiment of the present application further provides a method for processing service data, where the method includes:
acquiring service data from an original server;
storing data meeting the service conditions in the service data to a new server according to a mode of combining the service attribute hash and the data component hash;
acquiring stored data from a new server in a multi-fragment concurrent traversal mode;
and under the condition that the acquired stored data meet the sending requirement, sending the stored data to a user account corresponding to the stored data.
In a second aspect, an embodiment of the present application further provides a service data processing apparatus, where the apparatus includes:
the acquisition module is used for acquiring the service data from the original server;
the storage module is used for storing the data meeting the service conditions in the service data to the new server according to the mode of combining the service attribute hash and the data component hash;
the acquisition module is also used for acquiring the stored data from the new server in a multi-fragment concurrent traversal mode;
and the sending module is used for sending the stored data to the user account corresponding to the stored data under the condition that the acquired stored data meets the sending requirement.
In a third aspect, an embodiment of the present application further provides an electronic device, including: the system comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein when the processor executes the computer program, the business data processing method provided by any embodiment of the application is realized.
In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the service data processing method provided in any embodiment of the present application.
The embodiment of the application provides a method, a device, equipment and a storage medium for processing service data, wherein the method comprises the following steps: acquiring service data from an original server; storing data meeting the service conditions in the service data to a new server according to a mode of combining the service attribute hash and the data component hash; acquiring stored data from a new server in a multi-fragment concurrent traversal mode; and under the condition that the acquired stored data meet the sending requirement, sending the stored data to a user account corresponding to the stored data. The service data can be subjected to hash storage in a hash combination routing mode, hash collision of the data is avoided, the stored data is traversed in a multi-fragment concurrent traversal mode, the data with the same service attribute can be rapidly traversed, and traversal performance is improved.
Drawings
Fig. 1 is a flowchart of a service data processing method in an embodiment of the present application;
fig. 2 is a flowchart of a method for storing data meeting a service condition in service data to a new server according to a combination of a service attribute hash and a data component hash in an embodiment of the present application;
FIG. 3 is a flow diagram of a method for combining a service attribute hash with a data component hash in an embodiment of the present application;
FIG. 4 is a flowchart of a method for obtaining stored data from a new server in a multi-segment concurrent traversal manner according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a service data processing apparatus in an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
In addition, in the embodiments of the present application, the words "optionally" or "exemplarily" are used for indicating as examples, illustrations or explanations. Any embodiment or design described herein as "optionally" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the words "optionally" or "exemplarily" etc. is intended to present the relevant concepts in a concrete fashion.
In order to facilitate a clearer understanding of the solutions provided in the embodiments of the present application, related concepts related to the embodiments of the present application are further described herein, specifically as follows:
elastic search: the system is a search server based on Lucene, can provide a full-text search engine with distributed multi-user capability, and realizes data transmission based on a RESTful web interface. The elastic search is used in cloud computing, can realize real-time search, and is stable, reliable and convenient to install and use. Clients are available in Java,. NET (C #), PHP, Python, Apache Groovy, Ruby and many other languages.
Fig. 1 is a flowchart of a service data processing method provided in an embodiment of the present application, where the method may be applied to a database storing mass data, as shown in fig. 1, the method may include, but is not limited to, the following steps:
s101, acquiring service data from an original server.
Exemplarily, an original server in the embodiment of the present application may adopt an elastic search, the elastic search may retain a recent history record to provide a function of tracing traversal, and a hard disk space of the elastic search server is relatively large compared with a memory space, so that a risk that data cannot be stored due to business change is not generated. In the embodiment of the present application, the number of the origin servers may be one or more, for example, it is assumed that there are three elastic search servers as the origin servers for storing the service data. Alternatively, the business data can be obtained from the elastic search cluster by a cursor traversal (scroll) manner in the prior art.
The service data in this step may include user-related data in various network platform scenarios, for example, data information such as user coupons, user orders, and commodities in the orders in the e-commerce platform.
And S102, storing the data meeting the service conditions in the service data to a new server according to the mode of combining the service attribute hash and the data component hash.
The service condition in this step may be a condition for screening service data that is adaptively set according to a scene use requirement. Taking the e-commerce platform as an example, if the service data is a coupon of the user, the remaining time of the coupon of the user from the expiration date is less than or equal to the early warning period, which can be understood as the advance time of sending a reminding message to the user account, as a service condition for screening the data. For example, assuming that the early warning period is 10 days, relevant data with a remaining effective time of 10 days or less from the coupon expiration date can be screened out from massive user coupon data according to a mode of combining the service attribute hash and the data component hash, and the screened data is stored in a new server.
It should be noted that the number of the early warning periods may be one or more, and different early warning periods may be set for different types of coupons, for example, the early warning period of a coupon for a washing product is 7 days, the early warning device of a coupon for a book is 10 days, and the coupon for a daily product is 20 days, and any type of coupon may filter data according to the service conditions.
Alternatively, the new server may also adopt an elastic search, that is, a new elastic search except the original elastic search cluster for storing the business data may be used as the new server to store the data meeting the business condition. When the data meeting the service condition is stored in the new server, the msgpack technology can be used for compressing the data so as to save the storage resource of the server.
S103, acquiring the stored data from the new server in a multi-fragment concurrent traversal mode.
In this step, a multi-slice concurrent traversal (multiple slice) may be understood as performing a traversal for multiple slices (slices) in the server in parallel, and performing a traversal for one slice in series. Through the mode of multi-fragment concurrent traversal, rapid traversal can be realized to acquire data stored on the multi-fragments in a new server, so that the traversal performance is improved.
And S104, transmitting the stored data to the user account corresponding to the stored data under the condition that the acquired stored data meets the transmission requirement.
The sending requirement in this step may be a trigger condition set by the user according to the scene requirement. Similarly, taking the coupon of the e-commerce platform as an example, the accumulated amount of the coupon can be used as a transmission requirement of transmission data, that is, the accumulated amount of the coupon reaches a certain amount, and then the data can be transmitted to the corresponding user account; or, the number of the coupons may be used as a transmission requirement for transmitting data, for example, when the number of the coupons reaches a certain number, the data may be transmitted to the corresponding user account; or, the accumulated amount may be combined with the filtering condition of the service data to serve as a sending requirement, for example, if the remaining time of any coupon from the expiration date is equal to the warning period, and the accumulated amount of all coupons including any coupon reaches a certain amount, the relevant data message may be sent to the corresponding user account at one time.
Of course, the above sending requirement is only an exemplary description, and a person skilled in the art may adaptively set a relevant condition according to a network platform requirement or a requirement of a different scenario, which is not limited in the embodiment of the present application.
The embodiment of the application provides a service data processing method, which can comprise the following steps: acquiring service data from an original server; storing data meeting the service conditions in the service data to a new server according to a mode of combining the service attribute hash and the data component hash; acquiring stored data from a new server in a multi-fragment concurrent traversal mode; and under the condition that the acquired stored data meet the sending requirement, sending the stored data to a user account corresponding to the stored data. The service data can be subjected to hash storage in a hash combination routing mode, hash collision of the data is avoided, the stored data is traversed in a multi-fragment concurrent traversal mode, the data with the same service attribute can be rapidly traversed, and traversal performance is improved.
As shown in fig. 2, in an example, the implementation manner of the step S102 may include, but is not limited to, the following steps:
s201, scanning the data meeting the service conditions in a multi-fragment concurrent traversal mode according to the service attributes of the service data, and taking the data meeting the service conditions as target data.
Illustratively, the service attribute of the service data may include a user unique identifier pin, that is, the user identifier pin is used as an object, and a multi-segment concurrent traversal manner is adopted to perform data traversal, so that all data corresponding to the user identifier pin and meeting the service condition may be scanned.
For example, taking the user coupon in the e-commerce platform scenario as an example, assume that the set service conditions include: the coupon due date is the time of day plus the early warning period. If a user has 10 coupons, wherein the expiration date of 5 coupons is less than the time of the day + the early warning period, and the expiration date of 1 coupon is less than the time of the day + the early warning period, that is, the user has 6 coupons meeting the set service conditions, then the data information of the 6 coupons of the user can be quickly scanned in a multi-segment traversal manner, and the data information of the 6 coupons is the target data corresponding to the service attributes in this step.
S202, determining the fragments of the target data according to the mode of combining the service attribute hash and the data component hash.
In this embodiment, the service attribute hash may include a hash value of the same service attribute in each target data stored in the new server. Illustratively, the implementation of this step may include: determining the hash value of the same service attribute in each target data, combining the hash value of the data component of the current target data with the hash value of the same service attribute for each target data, and taking the balance of the number of fragments according to the combined hash value to obtain the fragment identifier corresponding to the current target data. And further, determining the fragment corresponding to the fragment identifier as the fragment to which the current target data belongs.
The hash value combination in the implementation process may be understood as hash value addition. The data components may be for scanning data, a piece of data corresponding to a data component of the scanned data.
By the method for routing the combined hash value, the data with the same hash value of the service attribute can be ensured to be hashed and stored, so that the condition that the data are unevenly distributed due to hash collision is avoided.
And S203, storing each target data into the corresponding fragment in the new server.
After the fragments corresponding to the target data are determined through the process, the target data can be stored in the corresponding fragments in the new server, and the purpose of hash storage of the target data is achieved.
As shown in fig. 3, in an example, the implementation manner of combining the service attribute hash and the data component hash in step S202 may include, but is not limited to, the following implementation manners:
s301, the hash value of the data component of the current target data is left according to the size of the routing hash value.
The size of the routing hash value in this step may be understood as a routing range, which is a range obtained by taking a group of fragments as an object and taking the hash value of the data component of the current target data as a remainder. Since each data corresponds to a respective data component, hash storage of the data can be achieved by taking into account the factors of the data components in the routing process.
And S302, combining the remainder result with the hash value with the same service attribute.
If the size of the routing hash value is partition _ rout _ size, the hash value of the data component corresponding to the data is hash (docid), and the hash value of the same service attribute is hash (routing), the hash value of the service attribute hash and the hash value of the data component may be represented as hash (routing) + hash (docid)% partition _ rout _ size.
Further, the step S202 of determining the representation manner of the fragment identifier corresponding to the current target data may include: shardNo ═ (hash) (routing) + hash (docid)% partition _ rout _ size%
primary_shard_nums
Wherein shardNo represents a fragment identifier, and primary _ shard _ nums represents the number of fragments.
As shown in fig. 4, in an example, the implementation manner of step S103 may include:
s401, traversing the service attributes of the data in the single fragment in a cursor traversal mode, and determining the data stored in the single fragment of the new server.
In the process of data cleaning, data screening and storage are performed by adopting a mode of combining the service attribute hash and the data component hash, so that all data with the same service attribute are ensured to be on the same fragment. In this step, the data may be scanned in a cursor traversal manner according to the service attribute of the data, and the data corresponding to the same service attribute in the single segment is determined by taking the service attribute as a unit, so as to implement compact traversal of the data with the same attribute.
S402, traversing in the multi-segment in a multi-segment concurrent traversing mode, and acquiring stored data from a new server.
In the embodiment of the application, the target data corresponding to the service attribute can be quickly traversed with better traversal performance by combining the multi-fragment concurrent traversal and the vernier traversal, and the method has better traversal advantages in a mass data scene. After data is obtained through traversal, relevant service processing can be performed after data is decompressed through the msgpack technology, for example, a reminding message of service data is sent to a user account.
In an example, an implementation manner of sending the stored data to the user account corresponding to the stored data in step S104 may include: and sending the stored data to the user account corresponding to the stored data at the appointed time.
The designated time in the implementation process may be a trigger time for sending data, which is set according to the scene needs, and the setting of the designated time is used for flexibly triggering data sending. That is, in the case where the stored data satisfies the transmission requirement, the stored data may be transmitted to the user account corresponding to the stored data at a designated time. For example, after traversing the stored data, if the transmission requirements are met, for example, if there are 5 coupons with due date < time of day + pre-warning period, and there are 1 coupon with due date ═ time of day + pre-warning period, and the accumulated amount of the 6 coupons is greater than the set amount threshold, the data may be immediately transmitted to the user account corresponding to the data, or the traversed data may be transmitted to the corresponding user account after a preset time interval. If data cleaning is carried out at 3 am, the service data meeting the service conditions are obtained and stored in the new server, and if the related service data meet the sending requirements, the data obtained by traversing from the new server can be sent to the user account at 9 am.
Fig. 5 is a service data processing apparatus according to an embodiment of the present application, and as shown in fig. 5, the apparatus may include an obtaining module 501, a storing module 502, and a sending module 503;
the acquisition module is used for acquiring service data from an original server;
the storage module is used for storing the data meeting the service conditions in the service data to the new server according to the mode of combining the service attribute hash and the data component hash;
the acquisition module is also used for acquiring the stored data from the new server in a multi-fragment concurrent traversal mode;
and the sending module is used for sending the stored data to the user account corresponding to the stored data under the condition that the acquired stored data meets the sending requirement.
In one example, the storage module may include a scanning unit, a determining unit, and a storage unit;
the scanning unit is used for scanning data meeting the service conditions in a multi-fragment concurrent traversal mode according to the service attributes of the service data, and taking the data meeting the service conditions as target data;
the determining unit is used for determining the fragments of each target data according to the mode of combining the service attribute hash and the data component hash;
and the storage unit is used for storing each target data into the corresponding fragment in the new server.
For example, the determining unit may be configured to perform the following processes to determine the fragmentation of each target data, for example, to determine a hash value of the same service attribute in each target data; for each target data, combining the hash value of the data component of the current target data with the hash value of the same service attribute, and taking the surplus of the number of fragments according to the combined hash value to obtain a fragment identifier corresponding to the current target data; and determining the fragment corresponding to the fragment identifier as the fragment to which the current target data belongs.
Further, the determining unit may be further configured to perform remainder on the hash value of the data component of the current target data according to the size of the route hash value, and combine a remainder result with the hash value of the same service attribute.
In one example, the obtaining module is configured to traverse the service attribute of the data in a single segment in a cursor traversal manner, and determine data stored in the single segment of the new server; and traversing in the multi-segment in a multi-segment concurrent traversal mode, and acquiring the stored data from the new server.
Furthermore, the acquisition module can be used for scanning the data in a cursor traversal mode according to the service attribute of the data; and determining data corresponding to the same service attribute in the single fragment of the new server by taking the service attribute as a unit.
In one example, the sending module is further configured to send the stored data to a user account corresponding to the stored data at a specified time.
The service data processing device can execute the service data processing method provided by fig. 1-4, and has corresponding devices and beneficial effects in the method.
Fig. 6 is a schematic structural diagram of an electronic apparatus according to embodiment 6 of the present invention, as shown in fig. 6, the electronic apparatus includes a controller 601, a memory 602, an input device 603, and an output device 606; the number of the controllers 601 in the electronic device may be one or more, and one controller 601 is taken as an example in fig. 6; the controller 601, the memory 602, the input device 603, and the output device 606 in the cloud management platform may be connected by a bus or other means, and fig. 6 illustrates an example of connection by a bus.
The memory 602 is used as a computer-readable storage medium and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules (for example, the obtaining module 501, the storing module 502, and the sending module 503 in the service data processing apparatus) corresponding to the service data processing method in the embodiments of fig. 1 to 4. The controller 601 executes various functional applications and data processing of the electronic device by running software programs, instructions and modules stored in the memory 602, that is, implements the service data processing method described above.
The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 602 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 602 may further include memory remotely located from the controller 601, which may be connected to a terminal/server through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 603 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the electronic apparatus. The output device 606 may include a display device such as a display screen.
Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer controller, are used to perform a business data processing method, including the steps shown in fig. 1-4.
From the above description of the embodiments, it is obvious for those skilled in the art that the present application can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods described in the embodiments of the present application.
It should be noted that the modules included in the service data processing apparatus are only divided according to functional logic, but are not limited to the above division manner, as long as the corresponding functions can be implemented, and are not used to limit the scope of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (10)
1. A method for processing service data is characterized by comprising the following steps:
acquiring service data from an original server;
storing data meeting service conditions in the service data to a new server according to a mode of combining service attribute hash and data component hash;
acquiring stored data from the new server in a multi-fragment concurrent traversal mode;
and under the condition that the acquired stored data meet the sending requirement, sending the stored data to a user account corresponding to the stored data.
2. The method of claim 1, wherein storing data meeting a service condition in the service data to a new server according to a combination of a service attribute hash and a data component hash comprises:
scanning data meeting the service conditions in a multi-fragment concurrent traversal mode according to the service attributes of the service data, and taking the data meeting the service conditions as target data;
determining the fragmentation of each target data according to a mode of combining the service attribute hash and the data component hash;
and storing each target data into a corresponding fragment in a new server.
3. The method of claim 2, wherein determining the fragmentation of each target data according to a combination of a service attribute hash and a data component hash comprises:
determining hash values of the same service attribute in each target data;
for each target data, combining the hash value of the data component of the current target data with the hash value of the same service attribute, and taking the surplus of the number of fragments according to the combined hash value to obtain a fragment identifier corresponding to the current target data;
and determining the fragment corresponding to the fragment identifier as the fragment to which the current target data belongs.
4. The method of claim 3, wherein combining the hash value of the data component of the current target data with the hash value of the same business attribute comprises:
the hash value of the data component of the current target data is left according to the size of the route hash value;
and combining the remainder result with the hash value with the same service attribute.
5. The method of claim 1, wherein obtaining the stored data from the new server by means of multi-shard concurrent traversal comprises:
traversing the service attributes of the data in the single fragment in a cursor traversal mode, and determining the data stored in the single fragment of the new server;
and traversing in the multi-segment in a multi-segment concurrent traversal mode, and acquiring the stored data from the new server.
6. The method of claim 5, wherein traversing in a single-shard in a vernier traversal manner for the service attributes of the data, determining the data stored in the single-shard of the new server, comprises:
scanning data in a cursor traversal mode according to the service attribute of the data;
and determining data corresponding to the same service attribute in the single fragment of the new server by taking the service attribute as a unit.
7. The method of claim 1, wherein sending the stored data to a user account to which the stored data corresponds comprises:
and sending the stored data to a user account corresponding to the stored data at a specified time.
8. A service data processing apparatus, comprising:
the acquisition module is used for acquiring the service data from the original server;
the storage module is used for storing the data meeting the service conditions in the service data to a new server according to a mode of combining the service attribute hash and the data component hash;
the acquisition module is further used for acquiring the stored data from the new server in a multi-fragment concurrent traversal mode;
and the sending module is used for sending the stored data to the user account corresponding to the stored data under the condition that the obtained stored data meets the sending requirement.
9. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the business data processing method according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method for processing traffic data according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110058355.6A CN112749166A (en) | 2021-01-15 | 2021-01-15 | Service data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110058355.6A CN112749166A (en) | 2021-01-15 | 2021-01-15 | Service data processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112749166A true CN112749166A (en) | 2021-05-04 |
Family
ID=75652267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110058355.6A Pending CN112749166A (en) | 2021-01-15 | 2021-01-15 | Service data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112749166A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220644A (en) * | 2021-05-28 | 2021-08-06 | 北京微纳星空科技有限公司 | File processing method, device, equipment and storage medium |
| CN113238845A (en) * | 2021-05-17 | 2021-08-10 | 北京沃东天骏信息技术有限公司 | Delayed settlement processing method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040162815A1 (en) * | 2003-02-13 | 2004-08-19 | Klaus Irle | Adapting business objects for searches and searching adapted business objects |
| CN102402617A (en) * | 2011-12-23 | 2012-04-04 | 天津神舟通用数据技术有限公司 | Easily compressed database index storage system using fragments and sparse bitmap, and corresponding construction, scheduling and query processing methods |
| CN104243462A (en) * | 2014-09-05 | 2014-12-24 | 中国科学院计算机网络信息中心 | User identity authentication method and system for discovery service |
| US20160321317A1 (en) * | 2012-03-29 | 2016-11-03 | Tracelink, Inc. | Methods and systems for managing distributed concurrent data updates of business objects |
| US20190228086A1 (en) * | 2018-01-25 | 2019-07-25 | Merck Sharp & Dohme Corp. | Verification of Data Provenance for Existing Computer Systems |
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
-
2021
- 2021-01-15 CN CN202110058355.6A patent/CN112749166A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040162815A1 (en) * | 2003-02-13 | 2004-08-19 | Klaus Irle | Adapting business objects for searches and searching adapted business objects |
| CN102402617A (en) * | 2011-12-23 | 2012-04-04 | 天津神舟通用数据技术有限公司 | Easily compressed database index storage system using fragments and sparse bitmap, and corresponding construction, scheduling and query processing methods |
| US20160321317A1 (en) * | 2012-03-29 | 2016-11-03 | Tracelink, Inc. | Methods and systems for managing distributed concurrent data updates of business objects |
| CN104243462A (en) * | 2014-09-05 | 2014-12-24 | 中国科学院计算机网络信息中心 | User identity authentication method and system for discovery service |
| US20190228086A1 (en) * | 2018-01-25 | 2019-07-25 | Merck Sharp & Dohme Corp. | Verification of Data Provenance for Existing Computer Systems |
| CN112181993A (en) * | 2020-10-27 | 2021-01-05 | 广州市网星信息技术有限公司 | Service data query method, device, server and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 黄永轩: "公众号思维", 31 January 2017, 广东经济出版社 , pages: 105 - 112 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113238845A (en) * | 2021-05-17 | 2021-08-10 | 北京沃东天骏信息技术有限公司 | Delayed settlement processing method and device |
| CN113220644A (en) * | 2021-05-28 | 2021-08-06 | 北京微纳星空科技有限公司 | File processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108737473B (en) | Data processing method, device and system | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| CN107391770B (en) | Method, device and equipment for processing data and storage medium | |
| CN112749166A (en) | Service data processing method, device, equipment and storage medium | |
| CN110932933B (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN109981702B (en) | File storage method and system | |
| CN111782692A (en) | Frequency control method and device | |
| CN104462500A (en) | Method for determining activeness of controls and control processing method and device | |
| CN113645287A (en) | Automobile message storage method and device and automobile message storage system | |
| CN110413846B (en) | Data processing method and device for webpage mirror image and computer readable storage medium | |
| CN110460668B (en) | File uploading method and device, computer equipment and storage medium | |
| CN110609924A (en) | Method, device and equipment for calculating total quantity relation based on graph data and storage medium | |
| CN108696559B (en) | Stream processing method and device | |
| CN113434796A (en) | Page cache operation method and device, storage medium and electronic device | |
| CN112860421B (en) | Method, apparatus and computer program product for job processing | |
| CN108960378B (en) | Data downloading method, system, device and storage medium | |
| CN110888840A (en) | File query method, device, equipment and medium in distributed file system | |
| CN109558403B (en) | Data aggregation method and device, computer device and computer readable storage medium | |
| KR20210000041A (en) | Method and apparatus for analyzing log data in real time | |
| CN115396494A (en) | Real-time monitoring method and system based on stream computing | |
| CN112311823B (en) | Flow control method and device of auditing system and server | |
| CN114564249A (en) | Recommendation scheduling engine, recommendation scheduling method, and computer-readable storage medium | |
| US20170139969A1 (en) | Method for filtering and analyzing big data, electronic device, and non-transitory computer-readable storage medium | |
| CN109828968B (en) | Data deduplication processing method, device, equipment, cluster and storage medium | |
| CN107103079B (en) | Live broadcast method and system for dynamic website |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant after: Jingdong Technology Holding Co.,Ltd. Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176 Applicant before: Jingdong Digital Technology Holding Co., Ltd |