CN110602159A - Data processing method and system - Google Patents
Data processing method and system Download PDFInfo
- Publication number
- CN110602159A CN110602159A CN201910696454.XA CN201910696454A CN110602159A CN 110602159 A CN110602159 A CN 110602159A CN 201910696454 A CN201910696454 A CN 201910696454A CN 110602159 A CN110602159 A CN 110602159A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- server
- target data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 146
- 239000000284 extract Substances 0.000 claims abstract description 9
- 238000004891 communication Methods 0.000 claims description 15
- 238000005538 encapsulation Methods 0.000 claims description 9
- 238000012546 transfer Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 2
- 238000004140 cleaning Methods 0.000 claims 2
- 238000000034 method Methods 0.000 description 19
- 230000008569 process Effects 0.000 description 14
- 230000003993 interaction Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
- G06Q30/0645—Rental transactions; Leasing transactions
-
- G06Q50/40—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Abstract
The application provides a data processing method and a data processing system, and relates to the field of data processing. Sending data requests to a plurality of supplier terminals through a processing server, wherein the data requests comprise a plurality of target parameters, then responding the plurality of target parameters by the plurality of supplier terminals to generate a plurality of pieces of target data, sending the plurality of pieces of target data to the processing server, and sending the plurality of pieces of target data to a plurality of storage servers for storage according to a preset rule by the processing server; and finally, the processing server extracts the target data from the plurality of storage servers one by one, encapsulates the target data and sends the encapsulated target data to the client. The data processing method and the data processing system have the advantages of being high in storage speed and high in concurrency bearing capacity.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data processing method and system.
Background
With the improvement of living standard of people, more and more users choose to rent cars for tourism when playing outside, so that the online car renting platform is rapidly developed.
For the online car renting platform, after a user sends a search request, the data needs to be acquired from a plurality of suppliers, so that the data volume is huge.
Therefore, for a single storage server, the processing server can only store data into the storage server in a queue manner, and the storage and calling of the data are slow, and the high concurrency tolerance is poor.
Disclosure of Invention
The application aims to provide a data processing method to solve the problem that in the prior art, high concurrency bearing capacity is poor when data is processed.
Another objective of the present application is to provide a data processing system to solve the problem of poor high concurrency tolerance for data processing in the prior art.
In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present application are as follows:
in one aspect, an embodiment of the present invention provides a data processing method applied to a data processing system, where the data processing system includes a processing server and a storage server cluster, the storage server cluster includes a plurality of storage servers, the processing server is in communication connection with the storage server cluster, and the processing server is also in communication connection with a client and a plurality of provider terminals; the processing method comprises the following steps:
the processing server sends a data request to the plurality of supplier terminals, wherein the data request comprises a plurality of target parameters;
the supplier terminals respond to the target parameters to generate a plurality of pieces of target data and send the target data to the processing server;
the processing server sends the target data to the storage servers for storage according to a preset rule;
and the processing server extracts the target data from the plurality of storage servers one by one, packages the target data and sends the packaged target data to the client.
On the other hand, the embodiment of the invention also provides a data processing system, which comprises a processing server and a storage server cluster, wherein the storage server cluster comprises a plurality of storage servers, the processing server is in communication connection with the storage server cluster, and the processing server is also in communication connection with a client and a plurality of supplier terminals;
the processing server is used for sending data requests to the plurality of supplier terminals, wherein the data requests comprise a plurality of target parameters;
the supplier terminals are used for responding to the target parameters to generate a plurality of pieces of target data and sending the target data to the processing server;
the processing server is used for sending the target data to the storage servers for storage according to a preset rule;
the processing server is further used for extracting the target data from the storage servers one by one, packaging the target data and sending the packaged target data to the client.
Compared with the prior art, the application has the following beneficial effects:
the embodiment of the application provides a data processing method and a data processing system, wherein a processing server sends data requests to a plurality of supplier terminals, the data requests comprise a plurality of target parameters, then the plurality of supplier terminals respond to the plurality of target parameters to generate a plurality of pieces of target data, the plurality of pieces of target data are sent to the processing server, and then the processing server sends the plurality of pieces of target data to a plurality of storage servers for storage according to preset rules; and finally, the processing server extracts the target data from the plurality of storage servers one by one, encapsulates the target data and sends the encapsulated target data to the client. Due to the fact that the storage server cluster is adopted in the data processing system, after the processing server receives the target data fed back by the supplier, the target data can be sent to the plurality of storage servers for storage according to the rule, the data do not need to be stored in the storage servers in a queue mode, the storage rate is high, and the high concurrency bearing capacity is high.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and it will be apparent to those skilled in the art that other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is an interaction diagram of a data processing system according to an embodiment of the present application.
Fig. 2 is a flowchart of a data processing method according to an embodiment of the present application.
Fig. 3 is an interaction diagram of a storage server cluster according to an embodiment of the present application.
In the figure: 100-a data processing system; 110-a processing server; 120-storage server cluster; 121-a storage server; 130-cold standby server.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", and the like indicate orientations or positional relationships based on orientations or positional relationships shown in the drawings or orientations or positional relationships conventionally found in use of products of the application, and are used only for convenience in describing the present application and for simplification of description, but do not indicate or imply that the referred devices or elements must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present application.
In the description of the present application, it is also to be noted that, unless otherwise explicitly specified or limited, the terms "disposed" and "connected" are to be interpreted broadly, e.g., as being either fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
First embodiment
An embodiment of the present invention provides a data processing method, which is applied to a data processing system 100, please refer to fig. 1, where the data processing system 100 includes a processing server 110 and a storage server cluster 120, the storage server cluster 120 includes a plurality of storage servers 121, the processing server 110 is in communication connection with the storage server cluster 120, that is, the processing server 110 is in communication connection with each storage server 121, the processing server 110 can implement data interaction with each storage server 121, and the processing server 110 is also in communication connection with a client and a plurality of providers. The data processing system 100 provided by the embodiment of the invention has a faster data processing speed and a stronger capability of bearing high concurrent events.
Specifically, referring to fig. 2, the data processing method includes:
s101, a client sends a data request to a processing server, wherein the data request comprises a plurality of target parameters.
In this embodiment, the client may be an intelligent terminal used by a user, such as a computer or a mobile phone, and when the user needs to rent a car, the client may send a data request to the processing server 110.
The data request sent by the user to the processing server 110 by using the mobile phone is taken as an example for explanation, a plurality of APPs are installed in the mobile phone of the user, and when the user opens the car rental APP, the communication connection can be established between the client and the processing server 110. Meanwhile, the user can select to input own requirement information on the APP, for example, the user needs to rent a car in guangzhou in number 20, that is, the own requirement information is input on the APP, so that the client generates a data request and sends the data request to the processing server 110.
The data request includes car rental location information and car rental time information, and the target parameter described in this embodiment is demand information input by a user, which includes car rental location information and car rental time information. Certainly, in some other embodiments, the target parameter may also be other parameters, for example, the target parameter may also include vehicle type information, and the like, which is not limited in this embodiment.
S102, the processing server sends data requests to a plurality of supplier terminals, wherein the data requests comprise a plurality of target parameters.
When the processing server 110 receives a data request sent by a client, the data request is forwarded to a plurality of provider terminals. For example, if the online car rental platform agrees with 200 suppliers, the processing server 110 will send the data request to the 200 suppliers at the same time. Of course, in some other embodiments, the processing server 110 may not forward the data directly, but needs to process the data request and send the processed data request to the provider side, for example, modify the data format of the data request into a uniform format.
S103, the plurality of supplier terminals respond to the plurality of target parameters to generate a plurality of pieces of target data, and send the plurality of pieces of target data to the processing server.
After receiving the data request sent by the processing server 110, the supplier end responds to the multiple target parameters to generate multiple pieces of target data, that is, the vehicle information meeting the requirements is searched in the database of the supplier end according to the target parameters, and the vehicle information is sent to the processing server 110, so that the server can receive the multiple pieces of target data.
It can be understood that the target data may be partitioned according to different suppliers, that is, all data fed back by each supplier is a piece of target data, and the entry mark data may include a plurality of pieces of vehicle information meeting the demand. For example, for 200 suppliers, the processing server 110 can receive up to 200 pieces of target data.
Of course, the target data may be classified according to different pieces of vehicle information, that is, each piece of vehicle information meeting the requirement is one piece of target data, and for the processing server 110, it may receive more than 200 pieces of target data.
And S104, the processing server sends the plurality of pieces of target data to a plurality of storage servers for storage according to a preset rule.
After the processing server 110 receives a plurality of pieces of target data, because the data volume itself is large, the processing server 110 needs to store the target data in the storage server 121 and then process the data, that is, the processing server 110 sends the target data to the storage server cluster 120 according to a preset rule.
Specifically, as an implementation manner of the present application, a specific scheme for the processing server 110 to send a plurality of pieces of target data to a plurality of storage servers 121 for storage according to a preset rule is as follows:
the processing server 110 sends the plurality of pieces of target data to the plurality of storage servers 121 in turn according to a preset ratio for storage.
That is, after the processing server 110 receives the object data, the object data are stored in the storage servers 121 in proportion, for example, the number of the storage servers 121 is 3, and the object data is 100, then the server may store the object data in the three storage servers 121 in proportion of 3:3:4, that is, the three storage servers 121 store 30, and 40 object data, respectively.
Meanwhile, in the storage process, the processing server 110 may send the target data to the storage servers 121 in turn, for example, the three storage servers 121 are A, B, C respectively, the processing server 110 may store the received first piece of target data in a, store the second piece of target data in B, store the third piece of target data in C, store the fourth piece of target data in a, and cycle sequentially. Compared with the traditional mode of adopting a single storage server 121, the storage server cluster 120 provided by the embodiment can store data in a plurality of storage servers 121 in a round mode when storing data, so that a large amount of data is not required to be stored in the storage servers 121 in a queue mode in the storage process, the storage rate is faster, the processing server 110 is not easy to accumulate a large amount of data to cause breakdown, and the capacity of processing high concurrency conditions is stronger, that is, the capacity of processing a large amount of data sent by a plurality of provider terminals at the same time is stronger.
It should be noted that the storage server 121 not only stores the target data fed back by the provider, but also processes the data request sent by the server, that is, the storage server cluster 120 stores the relevant data of the provider and the client at the same time.
S105, the storage server cluster 120 automatically cleans up the target data stored therein.
In an embodiment, the data of the storage server cluster 120 is stored in the memory first and is copied to the hard disk at the same time, it can be understood that the storage space of the hard disk is higher than that of the memory, for example, the storage space of the hard disk is 500G, and the storage space of the memory is 10G. Meanwhile, for the existing method of storing data by using a single storage server 121, after the user searches, the supplier end feeds back a large amount of data, and different users access the processing server 110 by using the clients every day, so that a large amount of data is generated every day, that is, the data keeps an infinitely increased state, and the feedback rate of the user during query is slow.
In view of this, the storage server cluster 120 provided in the present application will automatically clean the data stored therein, so as to clean the invalid data, and ensure the space utilization and query rate.
As an implementation manner of the present application, when the capacity of the plurality of storage servers 121 reaches the upper limit, the storage server cluster 120 overwrites the stored data with the target data after receiving the target data.
That is, each storage server 121 has an upper limit on the capacity, and when the data storage reaches the upper limit of the storage server 121, and the processing server 110 resumes the data transmission from the storage server 121, the storage server 121 overwrites the target data with the previously stored data after receiving the target data. Further, the storage server 121 may cover the data with the longest storage time by using the target data, for example, if a piece of data with the longest storage time in the storage server 121 is data before 2 days, the piece of data between 2 days may be automatically covered after the target data enters, so that the storage amount of the storage server 121 is stabilized at a value, the data stored in the storage server 121 is not infinitely increased, and the search service of the user is more stable.
As another implementation manner of the present application, each storage server 121 deletes target data whose storage time reaches a threshold value.
That is, for each storage server 121, it will clean up each piece of data by time. For example, if the threshold of the storage time is set to 1 day, the storage server 121 will automatically clean up the data stored for one day. For example, when one piece of data is stored in the storage server 121 at 6 am of today, the storage server 121 deletes the piece of data by 6 am of the next day, so as to ensure that the data stored in the storage server 121 does not increase infinitely and the search service of the user is more stable. It is understood that in implementing this implementation, there is no precedence between S105 and S101-S104.
Meanwhile, for the existing storage server 121, a failure may occur in the operation process, and once the failure occurs, the stored data cannot be called, so that the stability of the storage server 121 needs to be improved.
As an implementation manner of the present application, after receiving the target data, each storage server 121 performs data sharing on the target data and other storage servers 121 in the storage server cluster 120, so that the target data stored in each storage server 121 is the same.
That is, after each storage server 121 receives the data information, data sharing is implemented in the storage server 121 group, as can be understood, referring to fig. 3, in the storage server cluster 120, two storage servers 121 are connected in a pairwise communication manner, so as to achieve an effect of data synchronization, so that in the storage server cluster 120, each storage server 121 stores the same data, and when any one of the storage servers 121 fails, the processor may also call the data in the other storage servers 121, without affecting the normal operation of the processing server 110.
As another implementation manner of the present application, the data processing system 100 further includes a plurality of mapping storage servers 121, each storage server 121 is communicatively connected to at least one mapping storage server 121, and each storage server 121, after receiving the target data, copies the target data to the mapping storage server 121 communicatively connected thereto for storage.
That is, the data in each storage server 121 will be backed up to the mapping storage server 121, and when the storage server 121 fails, the processing server 110 can call the data in the mapping storage server 121 to process the data, so that the normal operation of the processing server 110 is not affected.
S106, the client sends a request instruction to the processing server after preset time.
Different from S101, in this step, a request instruction is sent to the processing server, and is triggered by the client, that is, the user triggers the request instruction by the front end itself without being aware of the request instruction. For example, after the user initiates the data request in S101 in the form of a web page, the browser may trigger the operation of sending the request instruction to the processing server after a preset time. Or, after the user initiates the data request in S101 through the APP, the APP may trigger an operation of sending a request instruction to the processing server after a preset time.
Meanwhile, the preset time described in the present application may be set to 1-2S, for example, the preset time is set to 1S or 2S, which is not specifically limited in the present application.
And S107, the processing server extracts the target data from the plurality of storage servers 121 one by one, encapsulates the target data and sends the encapsulated target data to the client.
Since the data acquired by the processing server 110 from the provider side is the original data, the original data needs to be encapsulated before being sent to the client side for viewing. The processing server 110 extracts the target data from the storage server 121 one by one, and performs encapsulation processing. It is understood that, for the extraction of data, the processing server 110 can also extract and package data in the plurality of storage servers 121 in turn.
Meanwhile, the encapsulation processing described in the present application refers to adding complete parameters to the target data extracted from the storage server 121, for example, the target data provided by the supplier end is actually vehicle information screened according to the request of the client end, and the encapsulation processing of the processing server 110 is data such as adding price to the target data according to the length of the user renting time. Certainly, the encapsulation process can also perform standardization processing on data provided by different supplier terminals, so that the data sent to the client terminal are standard data in the same format, and the user can watch the data more conveniently. Of course, the encapsulation process may also include more data processing procedures, which is not limited in this application.
It should be further noted that, for different supplier terminals, the rates of the feedback data are different, therefore, after receiving the target data sent by each supplier terminal, the processing server 110 handles the storage server cluster 120, then extracts the target data from the storage server 121 for encapsulation, and completes encapsulation of one data, that is, sends the data to the client for display, that is, for each data fed back by the supplier terminal, it performs asynchronous processing substantially, and it is faster to respond to the search request of the user.
And S108, when the processing server receives the effective information fed back by the client, sending a data transfer instruction to the storage server cluster, wherein the effective information is one or more target data after encapsulation processing.
After the packaged target data are sent to the client, the user can see the information of all vehicles which can be rented by the user, and the user can select whether to rent the vehicle or not. The effective information described in the embodiment is that the user selects to rent one or more target data or the user collects the target data through the client, and when the user performs a corresponding operation, the client sends the effective information to the processing server 110, for example, the information that the user selects to rent a vehicle. For example, the car rental platform can analyze different preferences of the customer under the current situation according to the data, and if the analysis shows that most of the customers prefer the cars of the type a, the information can be fed back to the supplier, so that the supplier can prepare more cars of the type a for rental.
For the storage server cluster 120, the data is automatically cleaned up to ensure that the data is not in an infinite growth. Therefore, when the processing server 110 receives the valid information fed back by the client, it will send a data transfer instruction to the storage server cluster 120 to transfer the user valid information to other storage devices, and no data loss will be caused by the update of the storage server cluster 120.
And S109, after receiving the data transfer instruction, the storage server cluster sends the target data corresponding to the effective information to the cold standby server for storage.
In this embodiment, the data processing system 100 further includes a cold-standby server 130, and the cold-standby server 130 is communicatively connected to the storage server cluster 120. After receiving the data transfer instruction, the storage server 121 transfers the valid information to the cold standby server 130 for storage, so that the valid information is not cleared in the clearing process of the storage server cluster 120.
Moreover, for the cold standby server 130 provided in this embodiment, it can only be accessed by the staff of the rental car platform, and the user cannot access the cold standby server 130, so that the data of the user and the staff is isolated, and the port of the staff and the port of the user are not affected. In the prior art, only the storage server 121 is arranged, so that both the port accessed by the staff and the port accessed by the user can only access the storage server 121, the stability of the system is low, and the stability of the system can be improved by arranging the cold standby server 130.
S110, when the storage time of the target data stored in the cold standby server reaches a threshold value, the target data of which the storage time reaches the threshold value is transferred to a target container.
It is understood that since there is an upper limit to the capacity of the cold standby server 130, the storage effective information in the cold standby server 130 is also cleared.
In this embodiment, the data stored in the cold standby server 130 is cleared according to the storage time, and specifically, when the storage time of the target data stored in the cold standby server 130 reaches a threshold value, the cold standby server 130 forwards the target data to the target container. The target container may be a storage-capable medium such as a hard disk. For example, if the storage time threshold is set to 2 years, the data stored in the cold standby server 130 can only be stored in the cold standby server 130 for 2 years, and after 2 years, the cold standby server 130 may transfer the data to a hard disk for archiving, so that the staff can analyze the data.
It is understood that S110 is not in sequence with S101-S109.
Second embodiment
Referring to fig. 1 again, an embodiment of the present application further provides a data processing system 100, where the data processing system 100 includes a processing server 110 and a storage server cluster 120, the storage server cluster 120 includes a plurality of storage servers 121, the processing server 110 is communicatively connected to the storage server cluster 120, and the processing server 110 is further communicatively connected to a client and a plurality of providers.
Specifically, the processing server 110 is configured to send a data request to a plurality of provider terminals, where the data request includes a plurality of target parameters, the plurality of provider terminals are configured to generate a plurality of pieces of target data in response to the plurality of target parameters, and send the plurality of pieces of target data to the processing server 110, the processing server 110 is configured to send the plurality of pieces of target data to the plurality of storage servers 121 for storage according to a preset rule, and the processing server 110 is further configured to extract the target data from the plurality of storage servers 121 one by one, encapsulate the target data, and send the encapsulated target data to the client.
Meanwhile, the data processing system 100 further includes a cold standby server 130, and the cold standby server 130 is in communication connection with the storage server cluster 120, so as to perform data interaction.
Since the specific process of the data processing system 100 provided in this application for data processing is the same as the data processing method described in the first embodiment, this embodiment will not be described again.
To sum up, the embodiment of the present application provides a data processing method and system, where a processing server sends a data request to multiple provider terminals, where the data request includes multiple target parameters, the multiple provider terminals respond to the multiple target parameters to generate multiple pieces of target data, and send the multiple pieces of target data to the processing server, and the processing server sends the multiple pieces of target data to multiple storage servers according to a preset rule for storage; and finally, the processing server extracts the target data from the plurality of storage servers one by one, encapsulates the target data and sends the encapsulated target data to the client. Due to the fact that the storage server cluster is adopted in the data processing system, after the processing server receives the target data fed back by the supplier, the target data can be sent to the plurality of storage servers for storage according to the rule, the data do not need to be stored in the storage servers in a queue mode, the storage rate is high, and the high concurrency bearing capacity is high.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. The data processing method is characterized by being applied to a data processing system, wherein the data processing system comprises a processing server and a storage server cluster, the storage server cluster comprises a plurality of storage servers, the processing server is in communication connection with the storage server cluster, and the processing server is also in communication connection with a client and a plurality of supplier terminals; the processing method comprises the following steps:
the processing server sends a data request to the plurality of supplier terminals, wherein the data request comprises a plurality of target parameters;
the supplier terminals respond to the target parameters to generate a plurality of pieces of target data and send the target data to the processing server;
the processing server sends the target data to the storage servers for storage according to a preset rule;
the client sends a request instruction to the processing server after a preset time
And the processing server extracts the target data from the plurality of storage servers one by one, packages the target data and sends the packaged target data to the client.
2. The data processing method of claim 1, wherein the step of sending the plurality of pieces of target data to the plurality of storage servers for storage according to a preset rule by the processing server comprises:
and the processing server sends the target data to the storage servers in turn according to a preset proportion for storage.
3. The data processing method of claim 1, wherein the data processing system further comprises a cold-standby server communicatively coupled to the cluster of storage servers; after the step of extracting the target data from the plurality of storage servers one by the processing server, encapsulating the target data, and sending the encapsulated target data to the client, the data processing method further includes:
when the processing server receives effective information fed back by the client, sending a data transfer instruction to the storage server cluster, wherein the effective information is one or more target data after encapsulation processing;
and after receiving the data transfer instruction, the storage server cluster sends the target data corresponding to the effective information to the cold standby server for storage.
4. The data processing method of claim 3, wherein the data processing method further comprises:
and when the storage time of the target data stored in the cold standby server reaches a threshold value, transferring the target data of which the storage time reaches the threshold value into a target container.
5. The data processing method of claim 1, wherein the data processing method further comprises:
and the storage server cluster automatically cleans the target data stored by the storage server cluster.
6. The data processing method of claim 5, wherein the step of the storage server cluster automatically cleaning the target data stored by the storage server cluster comprises:
when the capacity of the plurality of storage servers reaches the upper limit, the storage server cluster overwrites the stored data with the target data after receiving the target data.
7. The data processing method of claim 5, wherein the step of the storage server cluster automatically cleaning the target data stored by the storage server cluster comprises:
and each storage server deletes the target data with the storage time reaching the threshold value.
8. The data processing method according to claim 1, wherein after the step of the processing server sending the plurality of pieces of target data to the plurality of storage servers for storage according to a preset rule, the data processing method further comprises:
after each storage server receives the target data, the target data and other storage servers in the storage server cluster are subjected to data sharing, so that the target data stored in each storage server are the same.
9. The data processing method according to claim 1, wherein the data processing system further comprises a plurality of mapping storage servers, each of the storage servers is communicatively connected to at least one mapping storage server, and after the step of sending the plurality of pieces of target data to the plurality of storage servers for storage according to the preset rule by the processing server, the data processing method further comprises:
after each storage server receives the target data, copying the target data to a mapping storage server in communication connection with the storage server for storage.
10. A data processing system is characterized in that the data processing system comprises a processing server and a storage server cluster, the storage server cluster comprises a plurality of storage servers, the processing server is in communication connection with the storage server cluster, and the processing server is also in communication connection with a client and a plurality of supplier terminals;
the processing server is used for sending data requests to the plurality of supplier terminals, wherein the data requests comprise a plurality of target parameters;
the supplier terminals are used for responding to the target parameters to generate a plurality of pieces of target data and sending the target data to the processing server;
the processing server is used for sending the target data to the storage servers for storage according to a preset rule;
the processing server is further used for extracting the target data from the storage servers one by one, packaging the target data and sending the packaged target data to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910696454.XA CN110602159A (en) | 2019-07-30 | 2019-07-30 | Data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910696454.XA CN110602159A (en) | 2019-07-30 | 2019-07-30 | Data processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110602159A true CN110602159A (en) | 2019-12-20 |
Family
ID=68853074
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910696454.XA Pending CN110602159A (en) | 2019-07-30 | 2019-07-30 | Data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110602159A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112698175A (en) * | 2020-12-18 | 2021-04-23 | 武汉衍熙微器件有限公司 | Radio frequency device measuring system and method |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064890A (en) * | 2012-12-11 | 2013-04-24 | 泉州豪杰信息科技发展有限公司 | Global position system (GPS) mass data processing method |
| CN103327084A (en) * | 2013-06-08 | 2013-09-25 | 北京古盘创世科技发展有限公司 | Public and private hybrid distributed cloud storage system and cloud storage method |
| CN104331505A (en) * | 2014-11-20 | 2015-02-04 | 合一网络技术(北京)有限公司 | Distributed acquisition and storage-based monitoring system |
| CN104639661A (en) * | 2015-03-13 | 2015-05-20 | 华存数据信息技术有限公司 | Distributed storage system and storing and reading method for files |
| CN105491149A (en) * | 2015-12-26 | 2016-04-13 | 深圳市金立通信设备有限公司 | Data storage method and terminal |
| CN107436725A (en) * | 2016-05-25 | 2017-12-05 | 杭州海康威视数字技术股份有限公司 | A kind of data are write, read method, apparatus and distributed objects storage cluster |
| CN107566498A (en) * | 2017-09-08 | 2018-01-09 | 深圳市智物联网络有限公司 | A kind of method for monitoring numerical control machine and system based on Internet of Things |
| CN108494826A (en) * | 2018-02-25 | 2018-09-04 | 心触动（武汉）科技有限公司 | A kind of distribution cloud storage method and system |
| CN108737473A (en) * | 2017-04-20 | 2018-11-02 | 贵州白山云科技有限公司 | A kind of data processing method, apparatus and system |
| CN108881230A (en) * | 2018-06-21 | 2018-11-23 | 佛山科学技术学院 | A kind of safe transmission method and device of government affairs big data |
| CN109040173A (en) * | 2018-06-21 | 2018-12-18 | 佛山科学技术学院 | A kind of reliable storage method and device of government affairs big data |
| CN109104433A (en) * | 2018-09-28 | 2018-12-28 | 方信息科技（上海）有限公司 | A kind of distributed cryptographic storage system |
| US20190079832A1 (en) * | 2016-09-28 | 2019-03-14 | Hanwha Techwin Co., Ltd. | Data distribution storing method and system thereof |
| CN109547574A (en) * | 2019-01-04 | 2019-03-29 | 平安科技（深圳）有限公司 | A kind of data transmission method and relevant apparatus |
| CN109788051A (en) * | 2019-01-02 | 2019-05-21 | 深圳市网心科技有限公司 | Network resource transmission method and system based on distributed file system |
-
2019
- 2019-07-30 CN CN201910696454.XA patent/CN110602159A/en active Pending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064890A (en) * | 2012-12-11 | 2013-04-24 | 泉州豪杰信息科技发展有限公司 | Global position system (GPS) mass data processing method |
| CN103327084A (en) * | 2013-06-08 | 2013-09-25 | 北京古盘创世科技发展有限公司 | Public and private hybrid distributed cloud storage system and cloud storage method |
| CN104331505A (en) * | 2014-11-20 | 2015-02-04 | 合一网络技术(北京)有限公司 | Distributed acquisition and storage-based monitoring system |
| CN104639661A (en) * | 2015-03-13 | 2015-05-20 | 华存数据信息技术有限公司 | Distributed storage system and storing and reading method for files |
| CN105491149A (en) * | 2015-12-26 | 2016-04-13 | 深圳市金立通信设备有限公司 | Data storage method and terminal |
| CN107436725A (en) * | 2016-05-25 | 2017-12-05 | 杭州海康威视数字技术股份有限公司 | A kind of data are write, read method, apparatus and distributed objects storage cluster |
| US20190079832A1 (en) * | 2016-09-28 | 2019-03-14 | Hanwha Techwin Co., Ltd. | Data distribution storing method and system thereof |
| CN108737473A (en) * | 2017-04-20 | 2018-11-02 | 贵州白山云科技有限公司 | A kind of data processing method, apparatus and system |
| CN107566498A (en) * | 2017-09-08 | 2018-01-09 | 深圳市智物联网络有限公司 | A kind of method for monitoring numerical control machine and system based on Internet of Things |
| CN108494826A (en) * | 2018-02-25 | 2018-09-04 | 心触动（武汉）科技有限公司 | A kind of distribution cloud storage method and system |
| CN108881230A (en) * | 2018-06-21 | 2018-11-23 | 佛山科学技术学院 | A kind of safe transmission method and device of government affairs big data |
| CN109040173A (en) * | 2018-06-21 | 2018-12-18 | 佛山科学技术学院 | A kind of reliable storage method and device of government affairs big data |
| CN109104433A (en) * | 2018-09-28 | 2018-12-28 | 方信息科技（上海）有限公司 | A kind of distributed cryptographic storage system |
| CN109788051A (en) * | 2019-01-02 | 2019-05-21 | 深圳市网心科技有限公司 | Network resource transmission method and system based on distributed file system |
| CN109547574A (en) * | 2019-01-04 | 2019-03-29 | 平安科技（深圳）有限公司 | A kind of data transmission method and relevant apparatus |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112698175A (en) * | 2020-12-18 | 2021-04-23 | 武汉衍熙微器件有限公司 | Radio frequency device measuring system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107943841A (en) | Stream data processing method, system and computer-readable recording medium | |
| US20010047363A1 (en) | Apparatus and methods for providing personalized application search results for wireless devices based on user profiles | |
| CN106059825A (en) | Distributed system and configuration method | |
| US20080243789A1 (en) | Groupware system with improved contact data handling | |
| CN104346345B (en) | The storage method and device of data | |
| CN105868231A (en) | Cache data updating method and device | |
| CN107888666A (en) | A kind of cross-region data-storage system and method for data synchronization and device | |
| CN102355374A (en) | Data acquisition method and equipment | |
| CN106933548A (en) | Global information is obtained, processes and updated, methods, devices and systems | |
| CN109120461A (en) | A kind of service feature end-to-end monitoring method, system and device | |
| CN109298957A (en) | A kind of information management method with high reliability for supporting to regularly send | |
| CN112150203A (en) | Real estate client visit identification method and device, electronic equipment and storage medium | |
| CN112711612A (en) | Data processing method, data processing device, storage medium and electronic device | |
| US20180018363A1 (en) | Time series data processing method and apparatus | |
| JP2002203057A (en) | Added value data warehouse system | |
| CN107870982A (en) | Data processing method, system and computer-readable recording medium | |
| CN101673217A (en) | Method for realizing remote program call and system thereof | |
| CN110602159A (en) | Data processing method and system | |
| CN102831179B (en) | web page processing method and device | |
| CN108268497A (en) | The method of data synchronization and device of relevant database | |
| CN103825922B (en) | A kind of data-updating method and web server | |
| CN110929130B (en) | Public security level audit data query method based on distributed scheduling | |
| CN102170476B (en) | Cloud computing method and device based on cloud node autonomic learning | |
| CN111859127A (en) | Subscription method and device of consumption data and storage medium | |
| CN112000618B (en) | File change management method, device, equipment and storage medium for cluster nodes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191220 |
|
| RJ01 | Rejection of invention patent application after publication |