CN107766463A - A kind of storage processing method and device of the middleware system of multi-service driving - Google Patents
A kind of storage processing method and device of the middleware system of multi-service driving Download PDFInfo
- Publication number
- CN107766463A CN107766463A CN201710900534.3A CN201710900534A CN107766463A CN 107766463 A CN107766463 A CN 107766463A CN 201710900534 A CN201710900534 A CN 201710900534A CN 107766463 A CN107766463 A CN 107766463A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- stored
- terminal
- local disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses the storage processing method and device of a kind of middleware system of multi-service driving, method includes：Data to be stored are added in target partition file according to data storage algorithm；If judging to know, the size of all partitioned files reaches preset value, or current time reaches predetermined period, then by the local disk of the data storage in all partitioned files to middleware system；If receiving the data sharing request of first terminal transmission, sent according to the target data in data sharing acquisition request local disk, and by target data to first terminal.Data to be stored are added in the target partition file of buffer area by the embodiment of the present invention by existing, ensure that the adding speed of data；Size or predetermined period further according to partitioned file are by all data storages in local disk so that middleware system can store to magnanimity and complicated data；And by being sent from local disk by target data to first terminal, facilitate data management.
Description
Technical field
The present embodiments relate to communication technical field, and in particular to a kind of storage of the middleware system of multi-service driving
Processing method and processing device.
Background technology
In recent years, as the fast development of internet industry, global computer data total amount also show explosive increase.
There are increasing scientific research institution and producer to appreciate the immense value contained in these data, big data, distribution
Formula calculates, PC cluster, cloud computing, and many concepts related to large-scale data such as deep learning all become hot issue.
Traditional middleware main function is the otherness and operating system and procotol for shielding network hardware platform
Isomerism, connect two independent systems either two independent application programs, enable connected both sides relatively more flat
Sliding exchange information.Traditional middleware is divided into five classes, is database middleware, Oriented Transaction-Processing Middleware, remote process respectively
Call middleware, Object Request Broker's middleware and message-oriented middleware.Wherein message-oriented middleware is that most study use is most wide.
But existing middleware can not meet some applications of big data, such as the increase meeting pair of system and data scale
Performance and system storage tape necessarily influence, and its retractility can not meet the system scale constantly increased；Line number is entered using internal memory
According to storage and forwarding, there is no persistence mechanism, be distributed storage and message system event that system can not realize message data
The functions such as barrier recovery.In recent years, flow data and operation data analysis have become the application such as real-time monitoring, on-line analysis with processing
Important part, this just needs a set of somewhat more complicated message system to provide support to it.
During the embodiment of the present invention is realized, inventor has found that existing middleware system can not be to magnanimity and complexity
Data carry out distributed caching and management.
The content of the invention
Because existing method has above mentioned problem, the embodiment of the present invention proposes a kind of middleware system of multi-service driving
Storage processing method and device.
In a first aspect, the embodiment of the present invention proposes a kind of storage processing method of the middleware system of multi-service driving, bag
Include：
Data to be stored are received, the data to be stored are added in target partition file according to data storage algorithm；
If judging to know, the size of all partitioned files reaches preset value, or current time reaches predetermined period, then will
Data storage in all partitioned files and empties the data of all partitioned files to the local disk of middleware system；
It is local according to the data sharing acquisition request if receiving the data sharing request of first terminal transmission
Target data in disk, and the target data is sent to the first terminal, so that the first terminal is according to
Target data generation consumer record is stored.
Alternatively, if the data sharing request for receiving first terminal transmission, is asked according to the data sharing
The target data in the local disk is obtained, is specifically included：
If receiving the data sharing request of first terminal transmission, target is obtained according to the data sharing request analysis
Data determine the target data in described earth magnetism in the logical offsets of the local disk according to the logical offsets
Position in disk, and the target data according to the position acquisition.
Alternatively, it is described to receive data to be stored, the data to be stored are added to by target according to data storage algorithm
In partitioned file, specifically include：
Data to be stored are received, are determined to store the mesh of the data to be stored in several subregions according to data storage algorithm
Subregion is marked, and the data to be stored are added in target partition file corresponding to the target partition.
Alternatively, if the data sharing request for receiving first terminal transmission, is asked according to the data sharing
The target data in the local disk is obtained, and the target data is sent to the first terminal, so that described first
Terminal generates consumer record according to the target data and stored, and specifically includes：
It is local according to the data sharing acquisition request if receiving the data sharing request of first terminal transmission
Target data in disk, and the target data is sent to the first terminal and second terminal, so that described first is whole
End and the second terminal generate consumer record according to the target data and stored；
Wherein, the second terminal is the terminal relevant with the first terminal.
Alternatively, the middleware system includes several proxy servers；
Correspondingly, it is described to receive data to be stored, the data to be stored are added to by target according to data storage algorithm
In partitioned file, specifically include：
Data to be stored are received, the target proxy in several proxy servers according to determining server-assignment rule
Server, and the target partition text according to data storage algorithm by the data to be stored added to the target proxy server
In part.
Second aspect, the embodiment of the present invention also propose a kind of storage processing unit of the middleware system of multi-service driving,
Including：
Data add module, for receiving data to be stored, the data to be stored are added according to data storage algorithm
Into target partition file；
Data memory module, if for judging to know that the size of all partitioned files reaches preset value, or current time
Predetermined period is reached, then by the local disk of the data storage in all partitioned files to middleware system, and by all subregions
The data of file empty；
Data transmission blocks, if the data sharing request for receiving first terminal transmission, is total to according to the data
The target data in local disk described in acquisition request is enjoyed, and the target data is sent to the first terminal, so that institute
First terminal is stated to be stored according to target data generation consumer record.
Alternatively, if the data transmission blocks are specifically used for the data sharing request for receiving first terminal transmission,
Logical offsets of the target data in the local disk are obtained according to the data sharing request analysis, it is inclined according to the logic
Shifting amount determines position of the target data in the local disk, and the target data according to the position acquisition.
Alternatively, the data add module is specifically used for receiving data to be stored, if being determined according to data storage algorithm
The target partition of the data to be stored is stored in dry subregion, and the data to be stored are added to the target partition pair
In the target partition file answered.
The third aspect, the embodiment of the present invention also propose a kind of electronic equipment, including：
At least one processor；And
At least one memory being connected with the processor communication, wherein：
The memory storage has and by the programmed instruction of the computing device, the processor described program can be called to refer to
Order is able to carry out the above method.
Fourth aspect, the embodiment of the present invention also propose a kind of non-transient computer readable storage medium storing program for executing, the non-transient meter
Calculation machine readable storage medium storing program for executing stores computer program, and the computer program makes the computer perform the above method.
As shown from the above technical solution, the embodiment of the present invention passes through the existing target point by data to be stored added to buffer area
In area file, the adding speed of data ensure that；Size or predetermined period further according to partitioned file exist all data storages
Local disk so that middleware system can store to magnanimity and complicated data；And by from local disk by target
Data are sent to the first terminal, facilitate data management.
Brief description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
There is the required accompanying drawing used in technology description to be briefly described, it should be apparent that, drawings in the following description are only this
Some embodiments of invention, for those of ordinary skill in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these figures.
Fig. 1 is a kind of stream of the storage processing method of the middleware system for multi-service driving that one embodiment of the invention provides
Journey schematic diagram；
Fig. 2 is that a kind of proxy server that one embodiment of the invention provides and the model of consumer, producer's relation are illustrated
Figure；
Fig. 3 is the structural representation for the partitioned storage that one embodiment of the invention provides；
Fig. 4 is a kind of knot of the storage processing unit of the middleware system for multi-service driving that one embodiment of the invention provides
Structure schematic diagram；
Fig. 5 is the logic diagram for the electronic equipment that one embodiment of the invention provides.
Embodiment
Below in conjunction with the accompanying drawings, the embodiment of the present invention is further described.Following examples are only used for more
Clearly demonstrate technical scheme, and can not be limited the scope of the invention with this.
Fig. 1 shows a kind of flow of the storage processing method of the middleware system for multi-service driving that the present embodiment provides
Schematic diagram, including：
S101, data to be stored are received, it is according to data storage algorithm that the data to be stored are literary added to target partition
In part.
Wherein, the data to be stored are the data that needs that middleware system receives are stored.
The pending data is stored in the algorithm of which partitioned file for determination by the data storage algorithm.For example,
Hash algorithm can be used.
The target partition file is the partitioned file for storing the data to be stored.
The middleware system of multi-service driving is the middleware system that can handle multiple business simultaneously.
In proxy server as shown in Figure 2, producers and consumers' model, model consumption (is pushed away) with traditional Push
Unlike pattern, the present embodiment carries out " consumption " using Pull (drawing) model, i.e., data are pulled to agency service from the producer
Device, then consumer is from proxy server pulling data, and consumer is as the masters consumed, triggering process of consumption.Wherein, it is raw
Production person is provides the terminal of data, and consumer is the terminal of request data, and proxy server is the important composition of middleware system
Part, the middleware system that the present embodiment provides include multiple proxy servers.
If S102, judgement know that the size of all partitioned files reaches preset value, or current time reaches predetermined period,
Then by the local disk of the data storage in all partitioned files to middleware system, and the data of all partitioned files are clear
It is empty.
Wherein, the preset value is the size threshold value of partitioned file set in advance, when more than the preset value, illustrates this
Empty data, it is therefore desirable to all data are transferred to local disk and stored.
The predetermined period is data transfer cycles set in advance, when reaching the predetermined period, illustrates that this empties number
According to, it is therefore desirable to the required business datum emptied is transferred to local recycle bin.
If S103, the data sharing request for receiving first terminal transmission, according to the data sharing acquisition request institute
The target data in local disk is stated, and the target data is sent to the first terminal, so that the first terminal root
Stored according to target data generation consumer record.
Wherein, the first terminal is consumer.
Consumer record is stored by first terminal, without being preserved by proxy server, significantly reduces answering for agent side
Miscellaneous degree and load.
The middleware system that the present embodiment provides can carry out distributed caching to the data of magnanimity and complexity, have efficient
Persistence ability.Hard disk can also be read and write in constant time complexity to data more than TB levels by reaching.Gulped down with height
Rate is told, the message trunking system for forming the personal computer by low cost can also support the energy of handling up of 100K bar message above
Power.
Data to be stored are added in the target partition file of buffer area by the present embodiment by existing, ensure that adding for data
Acceleration；Size or predetermined period further according to partitioned file are by all data storages in local disk so that middleware system
Magnanimity and complicated data can be stored；And by being sent from local disk by target data to the first terminal,
Facilitate data management.
Further, on the basis of above method embodiment, S103 is specifically included：
If receiving the data sharing request of first terminal transmission, target is obtained according to the data sharing request analysis
Data determine the target data in described earth magnetism in the logical offsets of the local disk according to the logical offsets
Position in disk, and the target data according to the position acquisition.
Wherein, the logical offsets are that the space of local disk is divided into several fixed-size units, each
Sequence of unit is numbered, and the size and location of data is determined by start element numbering and end unit numbering.It is convenient to calculate data
Life cycle and the predetermined period.
Further, on the basis of above method embodiment, S101 is specifically included：
Data to be stored are received, are determined to store the mesh of the data to be stored in several subregions according to data storage algorithm
Subregion is marked, and the data to be stored are added in target partition file corresponding to the target partition.
Specifically, the present embodiment uses the lightweight storage organization based on subregion：Based on the light of partitioned storage unit mechanism
Magnitude storage organization.The corresponding logic log of each subregion of one service conversation theme (topic), a logic log by
What one group of an equal amount of partitioned file (Segment file) split formed.When publisher (producer) is by message or number
During according to being sent to subregion, middleware server is added it in last partitioned file in a daily record, such as Fig. 3 institutes
Show.Fig. 3 includes 3 subregions, and respectively subregion 0 (Partition0), subregion 1 (Partition1) arrives subregion n
In multiple subregions such as (Partition n), when writing message, it can write in last partitioned file of subregion 0, can also
Write in last partitioned file of subregion 1, can also write in subregion n last partitioned file.When quantity reaches
Certain value or to when determining the time, then partitioned file is stored in disk.Message could be by consumer after being only stored in disk
Consumed.Different from typical message system, the data in this middleware system do not have clear and definite message id, all data
Or partitioned file is positioned by logical offsets.Avoid with the true address of ID mapping data and caused dimension
The excess overhead of shield, search and random access.
Further, on the basis of above method embodiment, S103 is specifically included：
It is local according to the data sharing acquisition request if receiving the data sharing request of first terminal transmission
Target data in disk, and the target data is sent to the first terminal and second terminal, so that described first is whole
End and the second terminal generate consumer record according to the target data and stored；
Wherein, the second terminal is the terminal relevant with the first terminal.
For example, first terminal is mobile phone, and second terminal is notebook computer, and user is in first terminal and second terminal
It is upper to have logged in wechat simultaneously；When user asks A files on first terminal to middleware system, then middleware system is by A files
Send simultaneously to the wechat of first terminal and the wechat of second terminal, with synchronous two terminals, realize more preferable Consumer's Experience.
Further, on the basis of above method embodiment, the middleware system includes several proxy servers；
Correspondingly, S101 is specifically included：
Data to be stored are received, the target proxy in several proxy servers according to determining server-assignment rule
Server, and the target partition text according to data storage algorithm by the data to be stored added to the target proxy server
In part.
Specifically, the middleware system that the present embodiment provides uses scheduling and management based on Zookeeper application programs.
Zookeeper application programs can detect addition and the message proxy server deleted and consumer, can balance each consumption
Situation of the person when having Message Agent and consumer is added and deleted, can safeguard Consumption relation, additionally it is possible to each point of tracking
The offset consumed in area.
The middleware system that the present embodiment provides supports the message partition between more message servers, and ensures message in subregion
The order of reading, there is online horizontal extension ability；This middleware system can be collected with relatively low delay and distributed big
Log Data File is measured, message queue is capable of the flow pressure of buffer system, makes system from being failed under the pressure of big data；
Message server node can be extended according to flow, and without modification code and configuration file.
The access data type that middleware system is supported includes：The operation of database, log event, internet of things equipment data
With web page event etc., the integrated type of outbound data includes：The operation systems such as hadoop, database, data warehouse and CRM.Branch
Holding application in real time includes：Data conversion, custom application program, data analysis and monitoring etc..
The present embodiment uses persistent storage, i.e., different from traditional cache way by the way of innovation, including being not meant to
Middle preservation data as much as possible are deposited, and these data are written in file system when needed, but take complete phase
Anti- thinking, all data will write in persistence log system immediately, not carry out the calling of data immediately.
Middleware system that the present embodiment provides when it is implemented, using Avro coded systems, Snappy compression algorithms and
CA/SSL transmits secured fashion；Mainly uploaded including end message, information download, information download management, Broker management and letter
The partial function of transmitting, monitoring five is ceased, specifically：
A1, end message upload function
End message upload function includes reading parameter configuration files, including acquisition parameter and upload parameters.
Wherein, upload parameters include following three class：
1) Broker IP/ port lists
2) frequency is submitted：M seconds, n bars record (reach be defined at first)
3) summary information frequency acquisition is added up
Wherein, adding up summary information includes summary information acquisition time, end message submission time, submits successfully/failure
Mark, daily classify bulk registration number and submission failure error code and information.
Per Terminal Type, collection information is realized as a theme (topic), theme registration by broker ends.Joined according to collection
Number, acquisition terminal information (having been carried out) and accumulative summary information, local file is arrived in the end message storage collected；According to
Upload parameters, end message is sent to Broker servers；When data upload fails, log information, and handle are uploaded unsuccessfully in generation
The information and log information storage for uploading failure arrive local file, wait until upper load period next time, upload this information first.
A2, information download function
Information download function includes reading and information is downloaded.
Wherein, parameter configuration files include connection name, Broker ip/port lists, number of concurrent, theme, store path
Etc. content.
Information is downloaded i.e. by processes such as decoding, decryption, from broker download informations, with avro files or is organized into Hive
Object approach, by region, information type classified and stored to HDFS.
A3, information download management function
Information download management function presses multi-step grouping management.
Specifically, reading, editor, the storage (including local and remote 2 parts) of configuration file, point of configuration file are included
Hair, the installation of download module, and download module start, closed.
A4, Broker management function
Broker management functions press multi-step grouping management.
Specifically, include reading, editor, the storage (including local and remote 2 parts) of configuration file, Ip, port, agreement,
Cipher key store/password, the management for trusting the objects such as storehouse/password, the distribution of configuration file, the installation of Broker modules, Broker moulds
Block starts, closed, registration, deletion, the modification of message subject, and CA key managements.
A5, information transfer monitoring function
Information transfer monitoring function is monitored by multi-step grouping.
Specifically, including receive message number, forward message number, time delay, average received message quantity per second/gulp down
The amount of telling, average forwarding message number/handling capacity per second, Broker process status, Broker processes error code and content, information
Download module process status, information downloading process error code and content, and alarm threshold management.
The middleware system that the present embodiment provides with reference to existing result collection system and message queue, so may be used simultaneously
To adapt to online and offline processing two ways, i.e. real-time Communication for Power and non-realtime traffic two ways.In handling capacity and scalable
Property aspect, the middleware system has more preferable performance, for example, distributed structure/architecture, partitioned storage, order disk read-write etc. side
Formula.
Fig. 4 shows a kind of structure of the storage processing unit of the middleware system for multi-service driving that the present embodiment provides
Schematic diagram, described device include：Data add module 401, data memory module 402 and data transmission blocks 403, wherein：
The data add module 401 is used to receive data to be stored, according to data storage algorithm by the number to be stored
According to added in target partition file；
If the size that the data memory module 402 is used to judge to know all partitioned files reaches preset value, or works as
Preceding time reaches predetermined period, then by the local disk of the data storage in all partitioned files to middleware system, and by institute
The data for having partitioned file empty；
If the data transmission blocks 403 are used for the data sharing request for receiving first terminal transmission, according to
Target data in local disk described in data sharing acquisition request, and the target data is sent to the first terminal,
Stored so that the first terminal generates consumer record according to the target data.
Specifically, the data add module 401 receives data to be stored, will be described to be stored according to data storage algorithm
Data are added in target partition file；If the data memory module 402 judges to know that the size of all partitioned files reaches
Preset value, or current time reach predetermined period, then the data in all subregion scratch files are transferred into home server
Recycle bin,；If the data transmission blocks 403 receive the data sharing request of first terminal transmission, according to the number
The target data in the local disk is obtained according to sharing request, and the target data is sent to the first terminal, with
Make the first terminal generate consumer record according to the target data to be stored.
Data to be stored are added in the target partition file of buffer area by the present embodiment by existing, ensure that adding for data
Acceleration；Size or predetermined period further according to partitioned file are by all data storages in local disk so that middleware system
Magnanimity and complicated data can be stored；And by being sent from local disk by target data to the first terminal,
Facilitate data management.
Further, on the basis of said apparatus embodiment, if the data transmission blocks 403 are specifically used for receiving
The data sharing request that first terminal is sent, then obtain target data in described earth magnetism according to the data sharing request analysis
The logical offsets of disk, position of the target data in the local disk, and root are determined according to the logical offsets
According to target data described in the position acquisition.
Further, on the basis of said apparatus embodiment, the data add module 401 is specifically used for receiving waiting to deposit
Data are stored up, are determined to store the target partition of the data to be stored in several subregions according to data storage algorithm, and by described in
Data to be stored are added in target partition file corresponding to the target partition.
Further, on the basis of said apparatus embodiment, if the data transmission blocks 403 are specifically used for receiving
The data sharing that first terminal is sent is asked, then the number of targets in the local disk according to the data sharing acquisition request
According to, and the target data is sent to the first terminal and second terminal, so that the first terminal and described second is eventually
End generates consumer record according to the target data and stored；
Wherein, the second terminal is the terminal relevant with the first terminal.
Further, on the basis of said apparatus embodiment, the middleware system includes several proxy servers；
Correspondingly, the data add module 401 is specifically used for receiving data to be stored, according to server-assignment rule really
Target proxy server in several fixed described proxy servers, and added the data to be stored according to data storage algorithm
Add in the target partition file of the target proxy server.
The storage processing unit of the middleware system of multi-service driving described in the present embodiment can be used for performing above-mentioned side
Method embodiment, its principle is similar with technique effect, and here is omitted.
Reference picture 5, the electronic equipment, including：Processor (processor) 501, memory (memory) 502 and total
Line 503；
Wherein,
The processor 501 and memory 502 complete mutual communication by the bus 503；
The processor 501 is used to call the programmed instruction in the memory 502, to perform above-mentioned each method embodiment
The method provided.
The present embodiment discloses a kind of computer program product, and the computer program product includes being stored in non-transient calculating
Computer program on machine readable storage medium storing program for executing, the computer program include programmed instruction, when described program instruction is calculated
When machine performs, computer is able to carry out the method that above-mentioned each method embodiment is provided.
The present embodiment provides a kind of non-transient computer readable storage medium storing program for executing, the non-transient computer readable storage medium storing program for executing
Computer instruction is stored, the computer instruction makes the computer perform the method that above-mentioned each method embodiment is provided.
Device embodiment described above is only schematical, wherein the unit illustrated as separating component can
To be or may not be physically separate, it can be as the part that unit is shown or may not be physics list
Member, you can with positioned at a place, or can also be distributed on multiple NEs.It can be selected according to the actual needs
In some or all of module realize the purpose of this embodiment scheme.Those of ordinary skill in the art are not paying creativeness
Work in the case of, you can to understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
Realized by the mode of software plus required general hardware platform, naturally it is also possible to pass through hardware.Based on such understanding, on
The part that technical scheme substantially in other words contributes to prior art is stated to embody in the form of software product, should
Computer software product can store in a computer-readable storage medium, such as ROM/RAM, magnetic disc, CD, including some fingers
Make to cause a computer equipment (can be personal computer, server, or network equipment etc.) to perform each implementation
Method described in some parts of example or embodiment.
It should be noted that：The above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although reference
The present invention is described in detail previous embodiment, it will be understood by those within the art that：It still can be right
Technical scheme described in foregoing embodiments is modified, or carries out equivalent substitution to which part technical characteristic；And this
A little modifications are replaced, and the essence of appropriate technical solution is departed from the spirit and model of various embodiments of the present invention technical scheme
Enclose.
Claims (10)
- A kind of 1. storage processing method of the middleware system of multi-service driving, it is characterised in that including：Data to be stored are received, the data to be stored are added in target partition file according to data storage algorithm；If judging to know, the size of all partitioned files reaches preset value, or current time reaches predetermined period, then will be all Data storage in partitioned file and empties the data of all partitioned files to the local disk of middleware system；If the data sharing request of first terminal transmission is received, according to local disk described in the data sharing acquisition request In target data, and the target data is sent to the first terminal, so that the first terminal is according to the target Data generation consumer record is stored.
- 2. according to the method for claim 1, it is characterised in that if the data sharing for receiving first terminal transmission please Ask, then the target data in the local disk according to the data sharing acquisition request, is specifically included：If receiving the data sharing request of first terminal transmission, target data is obtained according to the data sharing request analysis In the logical offsets of the local disk, determine the target data in the local disk according to the logical offsets Position, and the target data according to the position acquisition.
- 3. according to the method for claim 1, it is characterised in that it is described to receive data to be stored, according to data storage algorithm The data to be stored are added in target partition file, specifically included：Data to be stored are received, are determined to store the target point of the data to be stored in several subregions according to data storage algorithm Area, and the data to be stored are added in target partition file corresponding to the target partition.
- 4. according to the method for claim 1, it is characterised in that if the data sharing for receiving first terminal transmission please Ask, then the target data in the local disk according to the data sharing acquisition request, and by the target data send to The first terminal, stored, specifically included so that the first terminal generates consumer record according to the target data：If the data sharing request of first terminal transmission is received, according to local disk described in the data sharing acquisition request In target data, and the target data is sent to the first terminal and second terminal so that the first terminal and The second terminal generates consumer record according to the target data and stored；Wherein, the second terminal is the terminal relevant with the first terminal.
- 5. according to the method for claim 1, it is characterised in that the middleware system includes several proxy servers；Correspondingly, it is described to receive data to be stored, the data to be stored are added to by target partition according to data storage algorithm In file, specifically include：Data to be stored are received, the target proxy service in several proxy servers according to determining server-assignment rule Device, and the target partition file according to data storage algorithm by the data to be stored added to the target proxy server In.
- A kind of 6. storage processing unit of the middleware system of multi-service driving, it is characterised in that including：Data add module, for receiving data to be stored, the data to be stored are added to by mesh according to data storage algorithm Mark in partitioned file；Data memory module, if for judging to know that the size of all partitioned files reaches preset value, or current time reaches Predetermined period, then by the local disk of the data storage in all partitioned files to middleware system, and by all partitioned files Data empty；Data transmission blocks, please according to the data sharing if the data sharing request for receiving first terminal transmission The target data obtained in the local disk is sought, and the target data is sent to the first terminal, so that described the One terminal generates consumer record according to the target data and stored.
- 7. according to the method for claim 6, it is characterised in that if the data transmission blocks are specifically used for receiving first The data sharing request that terminal is sent, then obtain target data in the local disk according to the data sharing request analysis Logical offsets, position of the target data in the local disk is determined according to the logical offsets, and according to institute State target data described in position acquisition.
- 8. according to the method for claim 6, it is characterised in that the data add module is specifically used for receiving number to be stored According to being determined to store the target partition of the data to be stored in several subregions according to data storage algorithm, and wait to deposit by described Data are stored up to be added in target partition file corresponding to the target partition.
- 9. a kind of electronic equipment, it is characterised in that including：At least one processor；AndAt least one memory being connected with the processor communication, wherein：The memory storage has can be by the programmed instruction of the computing device, and the processor calls described program instruction energy Enough perform the method as described in claim 1 to 5 is any.
- 10. a kind of non-transient computer readable storage medium storing program for executing, it is characterised in that the non-transient computer readable storage medium storing program for executing is deposited Computer program is stored up, the computer program makes the computer perform the method as described in claim 1 to 5 is any.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710900534.3A CN107766463A (en) | 2017-09-28 | 2017-09-28 | A kind of storage processing method and device of the middleware system of multi-service driving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710900534.3A CN107766463A (en) | 2017-09-28 | 2017-09-28 | A kind of storage processing method and device of the middleware system of multi-service driving |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107766463A true CN107766463A (en) | 2018-03-06 |
Family
ID=61266360
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710900534.3A Pending CN107766463A (en) | 2017-09-28 | 2017-09-28 | A kind of storage processing method and device of the middleware system of multi-service driving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107766463A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108646987A (en) * | 2018-05-21 | 2018-10-12 | 北京五八信息技术有限公司 | A kind of management method of file volume, device, storage medium and terminal |
| CN109669941A (en) * | 2018-11-16 | 2019-04-23 | 深圳证券交易所 | The asynchronous persistence method of data, device and computer readable storage medium |
| CN109815292A (en) * | 2019-01-03 | 2019-05-28 | 广州中软信息技术有限公司 | A kind of concerning taxes data collection system based on asynchronous message mechanism |
| CN110851077A (en) * | 2019-10-25 | 2020-02-28 | 中盈优创资讯科技有限公司 | Logstash data processing device and method |
| CN111641592A (en) * | 2020-05-08 | 2020-09-08 | 卓尔智联(武汉)研究院有限公司 | Data transmission method and device based on middleware and computer equipment |
| CN111741041A (en) * | 2019-07-10 | 2020-10-02 | 北京京东尚科信息技术有限公司 | Message processing method and device, electronic equipment and computer readable medium |
| CN116527691A (en) * | 2023-06-27 | 2023-08-01 | 天津中远海运散运数字科技有限公司 | Method, device, equipment and medium for synchronizing ship-shore data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103645859A (en) * | 2013-11-19 | 2014-03-19 | 华中科技大学 | Disk array caching method for virtual SSD and SSD isomerous mirror image |
| CN103973791A (en) * | 2014-05-08 | 2014-08-06 | 江苏物联网研究发展中心 | Rapid uniform storage system of internet of things |
| CN104113599A (en) * | 2014-07-22 | 2014-10-22 | 北京星网锐捷网络技术有限公司 | Caching method and device and proxy server |
-
2017
- 2017-09-28 CN CN201710900534.3A patent/CN107766463A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103645859A (en) * | 2013-11-19 | 2014-03-19 | 华中科技大学 | Disk array caching method for virtual SSD and SSD isomerous mirror image |
| CN103973791A (en) * | 2014-05-08 | 2014-08-06 | 江苏物联网研究发展中心 | Rapid uniform storage system of internet of things |
| CN104113599A (en) * | 2014-07-22 | 2014-10-22 | 北京星网锐捷网络技术有限公司 | Caching method and device and proxy server |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108646987A (en) * | 2018-05-21 | 2018-10-12 | 北京五八信息技术有限公司 | A kind of management method of file volume, device, storage medium and terminal |
| CN109669941A (en) * | 2018-11-16 | 2019-04-23 | 深圳证券交易所 | The asynchronous persistence method of data, device and computer readable storage medium |
| CN109815292A (en) * | 2019-01-03 | 2019-05-28 | 广州中软信息技术有限公司 | A kind of concerning taxes data collection system based on asynchronous message mechanism |
| CN111741041A (en) * | 2019-07-10 | 2020-10-02 | 北京京东尚科信息技术有限公司 | Message processing method and device, electronic equipment and computer readable medium |
| CN111741041B (en) * | 2019-07-10 | 2023-05-12 | 北京京东尚科信息技术有限公司 | Message processing method and device, electronic equipment and computer readable medium |
| CN110851077A (en) * | 2019-10-25 | 2020-02-28 | 中盈优创资讯科技有限公司 | Logstash data processing device and method |
| CN111641592A (en) * | 2020-05-08 | 2020-09-08 | 卓尔智联(武汉)研究院有限公司 | Data transmission method and device based on middleware and computer equipment |
| CN111641592B (en) * | 2020-05-08 | 2022-07-29 | 卓尔智联(武汉)研究院有限公司 | Data transmission method and device based on middleware and computer equipment |
| CN116527691A (en) * | 2023-06-27 | 2023-08-01 | 天津中远海运散运数字科技有限公司 | Method, device, equipment and medium for synchronizing ship-shore data |
| CN116527691B (en) * | 2023-06-27 | 2023-11-03 | 天津中远海运散运数字科技有限公司 | Method, device, equipment and medium for synchronizing ship-shore data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107766463A (en) | A kind of storage processing method and device of the middleware system of multi-service driving | |
| US11902092B2 (en) | Systems and methods for latency-aware edge computing | |
| US20170083579A1 (en) | Distributed data processing method and system | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN105659209B (en) | The cloud service of trustship on a client device | |
| CN106254461B (en) | A kind of method of data synchronization of intelligent vehicle sensing capability test platform | |
| CN105468442A (en) | Application and information movement in a cloud environment | |
| WO2016090421A1 (en) | Notifications on mobile devices | |
| CN105577772A (en) | Material receiving method, material uploading method and device | |
| WO2018049563A1 (en) | Systems and methods for caching | |
| CN109800005A (en) | A kind of hot update method of client and device | |
| CN110058989A (en) | User behavior Intention Anticipation method and apparatus | |
| CN107168802A (en) | The merging method and device of a kind of cloud storage small file | |
| CN107370684A (en) | Business current-limiting method and business current-limiting apparatus | |
| CN110493302A (en) | A kind of document transmission method, equipment and computer readable storage medium | |
| CN105589658A (en) | Resource processing method and system, server as well as warehouse management method and device | |
| US20150058438A1 (en) | System and method providing hierarchical cache for big data applications | |
| CN110309215A (en) | A kind of data processing method, system and metadata updates method, system | |
| CN109947729A (en) | A kind of real-time data analysis method and device | |
| Perez et al. | A monitoring framework for multi-site 5G platforms | |
| CN110321225A (en) | Load-balancing method, meta data server and computer readable storage medium | |
| WO2021137768A1 (en) | Method, apparatus and device for consuming messages based on message queue | |
| US8990301B2 (en) | Broker designation and selection in a publish-subscription environment | |
| CN109491807A (en) | Data exchange method, device and system | |
| CN113822454B (en) | Method and device for predicting slice complaint processing resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180306 |
|
| RJ01 | Rejection of invention patent application after publication |