CN117319474A - Multi-line program columnar data transmission processing method and device - Google Patents
Multi-line program columnar data transmission processing method and device Download PDFInfo
- Publication number
- CN117319474A CN117319474A CN202311385860.7A CN202311385860A CN117319474A CN 117319474 A CN117319474 A CN 117319474A CN 202311385860 A CN202311385860 A CN 202311385860A CN 117319474 A CN117319474 A CN 117319474A
- Authority
- CN
- China
- Prior art keywords
- sub
- positioning information
- objects
- splitting
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 title claims abstract description 60
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 71
- 238000000034 method Methods 0.000 claims abstract description 49
- 230000015654 memory Effects 0.000 claims description 37
- 238000004590 computer program Methods 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 description 17
- 238000010586 diagram Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 230000007246 mechanism Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 239000012634 fragment Substances 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 230000033228 biological regulation Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 229910002056 binary alloy Inorganic materials 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/133—Protocols for remote procedure calls [RPC]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The specification relates to the technical field of data transmission, and provides a multithreading serialized data transmission processing method and device. The method comprises the following steps: acquiring a target object; splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object; adding respective positioning information into each first sub-object to obtain a corresponding second sub-object; carrying out parallel serialization processing on the second sub-object by utilizing multiple threads; and transmitting the serialization processing result to the target receiving end. By the embodiment of the invention, the transmission efficiency of the structural object can be increased, and the performance of remote calling service and application can be improved.
Description
Technical Field
The present disclosure relates to the field of data transmission technologies, and in particular, to a method and an apparatus for processing multi-line program columnar data transmission.
Background
Communication between distributed systems may be implemented using remote call-based approaches, remote procedure calls (Remote Procedure Call, RPC) refer to intercommunication between processes running on different machines, unlike local calls in which:
(1) The local call belongs to in-process call, the program directly calls the function through the function name, the function name can be regarded as the address of the function in the memory, for example, calling the function f (), and the compiler can help us find the memory address corresponding to the function f (); however, in RPC, since address spaces in different processes are different, it is not feasible to call a corresponding service only by using a function name, so that all functions need to have a call ID to implement unique identification, a machine must attach the call ID when a remote procedure is called, and a mapping table of functions and call IDs needs to be built and maintained.
(2) The local call is to transfer parameters through the memory; when the caller calls the RPC, the caller needs to transmit the call ID and the corresponding function parameters (collectively referred to as the structure object) to the callee, the process is distributed on different machines, and the memory of the different machines is used, so that the parameters cannot be transmitted through the memory, the content transmitted by the network protocol is a binary stream, and the type of the parameters cannot be directly transmitted, so that the caller needs to convert the parameters into a binary stream first, and after the parameters are transmitted to the callee, the callee converts the binary stream into a format which can be read by the caller. The caller converts the parameters into a binary stream, commonly referred to as serialization, and the conversion of the binary by the callee is commonly referred to as deserialization. Similarly, the called party returns the result to the calling party, and a serialization and deserialization process is also required. That is, the passing of parameters by the RPC requires serialization and deserialization operations as compared to local calls. In the prior art, one or more structural objects are subjected to overall serialization and deserialization, and serialization and deserialization are performed on one thread, so that the efficiency of converting the objects into a binary mode capable of being stored and transmitted is not high enough, the resource utilization rate is not sufficient, and when the objects are too large, the time consumption is relatively long, so that the performance of remote calling service and application is not high enough.
Disclosure of Invention
In view of the fact that the whole serialization and deserialization of one or more structure objects are performed in the current use, and the serialization and deserialization are performed on one thread, the efficiency of converting the objects into a binary system capable of being stored and transmitted is not high enough, the resource utilization rate is not sufficient, when the objects are too large, the time consumption is long, the performance of remote calling service and application is not high enough, and the scheme is proposed so as to overcome the problems or at least partially solve the problems.
In one aspect, some embodiments of the present disclosure provide a method for processing multithreaded serialized data transmission, applied to a caller, the method comprising:
acquiring a target object;
splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object;
adding respective positioning information into each first sub-object to obtain a corresponding second sub-object;
carrying out parallel serialization processing on the second sub-object by utilizing multiple threads;
and transmitting the serialization processing result to the target receiving end.
Further, the splitting the target object into a plurality of first sub-objects and determining the positioning information of each first sub-object includes:
judging whether the size of the target object is larger than a first occupied space or not;
if yes, splitting the target object by using the first occupied space to obtain a plurality of first sub-objects;
recording the splitting ordinal number of each first sub-object and the splitting total number of the first sub-objects;
and matching the split ordinal number of each first sub-object with the positioning information corresponding to the split total number from a preset positioning information table, or inputting the split ordinal number of each first sub-object and the split total number into a positioning information generation algorithm to obtain the corresponding positioning information.
Further, the adding the respective positioning information in each first sub-object includes adding a first positioning identifier and a second positioning identifier in the respective positioning information in the first position and the second position of each first sub-object.
Further, after the serialization processing result is transmitted to the target receiving end, the method further includes:
intercepting the serialized data with the length of a preset value from the same initial position of each serialized processing result;
calculating a transmission state identification value corresponding to each serialized data;
storing the transmission state identification value into a cache;
judging whether the serialization processing result is transmitted or not through the transmission state identification value in the buffer memory, and obtaining a retransmission judgment result;
and executing corresponding continuous transmission operation according to the retransmission judgment result.
In another aspect, some embodiments of the present specification further provide a multithreaded serialized data transmission processing apparatus for use by a caller, the apparatus comprising:
the receiving module is used for acquiring a target object;
the splitting module is used for splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object;
the adding module is used for adding respective positioning information into each first sub-object to obtain a corresponding second sub-object;
the processing module is used for carrying out parallel serialization processing on the second sub-object by utilizing multiple threads;
and the transmission module is used for transmitting the serialization processing result to the target receiving end.
Based on the same inventive concept, embodiments herein also provide a multi-threaded program-listed data transmission processing method, applied to a called party, the method comprising,
receiving a plurality of serialization information sent by a calling party;
performing parallel deserialization processing on the serialized information by utilizing multiple threads to obtain a second sub-object;
extracting positioning information from each second sub-object, and deleting the positioning information in each second sub-object to obtain a first sub-object;
determining the splicing sequence of the first sub-objects by using the positioning information extracted from the second sub-objects;
and merging the first sub-objects according to the splicing sequence, so as to obtain a target object.
In another aspect, embodiments herein further provide a multithreaded serialized data transmission processing apparatus for application to a callee, the apparatus comprising:
the receiving module is used for receiving a plurality of serialization information sent by the calling party;
the processing module is used for carrying out parallel deserialization processing on the serialized information by utilizing multiple threads to obtain a second sub-object;
the deleting module is used for extracting positioning information from each second sub-object and deleting the positioning information in each second sub-object to obtain a first sub-object;
the determining module is used for determining the splicing sequence of the first sub-objects by utilizing the positioning information extracted from the second sub-objects;
and the merging module is used for merging the first sub-objects according to the splicing sequence so as to obtain a target object.
In another aspect, some embodiments of the present description also provide a computer device including a memory, a processor, and a computer program stored on the memory, which when executed by the processor, performs the instructions of the above method.
In another aspect, some embodiments of the present description also provide a computer storage medium having stored thereon a computer program which, when executed by a processor of a computer device, performs instructions of the above method.
In another aspect, some embodiments of the present description also provide a computer program product comprising a computer program which, when executed by a processor of a computer device, performs instructions of the above method.
One or more technical solutions provided in some embodiments of the present disclosure at least have the following technical effects:
according to the embodiment of the specification, after the target object is automatically acquired, the target object is split into a plurality of first sub-objects, the problem of transmitting one target object is converted into the problem of transmitting a plurality of smaller sub-objects, and corresponding positioning information is added to the first sub-objects to obtain corresponding second sub-objects so as to determine the splicing sequence of the sub-objects after subsequent merging and splitting, so that the transmission safety can be improved, the plurality of second sub-objects are processed in a multithreading parallel serialization manner, the efficiency of transmitting the serialization processing result to the target receiving end is improved, and the performance of remote calling service and application is guaranteed.
The foregoing description is merely an overview of some embodiments of the present disclosure, which may be practiced in accordance with the disclosure of the present disclosure, for the purpose of making the foregoing and other objects, features, and advantages of some embodiments of the present disclosure more readily apparent, and for the purpose of providing a more complete understanding of the present disclosure's technical means.
Drawings
In order to more clearly illustrate some embodiments of the present description or technical solutions in the prior art, the following description will briefly explain the embodiments or drawings needed in the description of the prior art, and it is obvious that the drawings in the following description are only some embodiments described in the present description, and other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art. In the drawings:
FIG. 1 is a schematic diagram of an implementation system of a multithreaded serialized data transmission processing method in some embodiments of the present disclosure;
FIG. 2 illustrates a flow chart of a multi-threaded, programmed data transmission processing method in some embodiments of the present disclosure;
FIG. 3 is a schematic diagram illustrating steps for determining positioning information according to some embodiments of the present disclosure;
fig. 4 is a schematic diagram illustrating a retransmission determination step in some embodiments of the present disclosure;
fig. 5 is a schematic structural diagram of a multi-line programmed data transmission processing device according to some embodiments of the present disclosure;
FIG. 6 is a flow chart of a method of multi-threaded programmed data transmission processing in some embodiments of the present disclosure;
fig. 7 is a schematic structural diagram of a multi-line programmed data transmission processing device according to some embodiments of the present disclosure;
fig. 8 is a schematic diagram of a computer device provided in some embodiments of the present disclosure.
[ reference numerals description ]
1. A multithreaded serialized data transmission processing system;
101. a terminal;
102. a server;
501. a receiving module;
502. splitting the module;
503. adding a module;
504. a processing module;
505. a transmission module;
701. a receiving module;
702. a processing module;
703. deleting the module;
704. a determining module;
705. a merging module;
802. a computer device;
804. a processor;
806. a memory;
808. a driving mechanism;
810. an input/output interface;
812. an input device;
814. an output device;
816. a presentation device;
818. a graphical user interface;
820. a network interface;
822. a communication link;
824. a communication bus.
Detailed Description
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in some embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure based on some embodiments in the present disclosure.
It should be noted that the terms "first," "second," and the like in the description and claims herein and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or device. It should be noted that, in the technical scheme of the application, the acquisition, storage, use, processing and the like of the data all conform to the relevant regulations of national laws and regulations.
Fig. 1 is a schematic diagram of an implementation system of a multithreaded serialized data transmission processing method according to an embodiment of the invention, which may include: the terminal 101 and the server 102 communicate with each other through a network, which may include a local area network (Local Area Network, abbreviated as LAN), a wide area network (Wide Area Network, abbreviated as WAN), the internet, or a combination thereof, and are connected to a website, user equipment (e.g., a computing device), and a back-end system. The staff can send the multithreaded serial data transmission processing request to the server 102 through the terminal 101, after the server 102 receives the multithreaded serial data transmission processing request, the target object in the database is called for calculation processing to obtain a serial processing result, and the serial processing result is sent to the terminal 101, so that the staff can perform subsequent processing work according to the serial processing result.
In this embodiment of the present disclosure, the server 102 may be an independent physical server, or may be a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN, content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.
In an alternative embodiment, terminal 101 may include, but is not limited to, a self-service terminal device, a desktop computer, a tablet computer, a notebook computer, a smart wearable device, and the like. Alternatively, the operating system running on the electronic device may include, but is not limited to, an android system, an IOS system, linux, windows, and the like. Of course, the terminal 101 is not limited to the above-mentioned electronic device having a certain entity, and may be software running in the above-mentioned electronic device.
In addition, it should be noted that, fig. 1 is only an application environment provided by the present disclosure, and in practical application, a plurality of terminals 101 may also be included, which is not limited in this specification.
Fig. 2 is a flowchart of a multi-threaded, programmed data transmission processing method (for use by a caller) according to an embodiment of the invention, the present disclosure provides the method steps as described in the examples or flowcharts, but may include more or fewer steps based on conventional or non-inventive labor. The order of steps recited in the embodiments is merely one way of performing the order of steps and does not represent a unique order of execution. When a system or apparatus product in practice is executed, it may be executed sequentially or in parallel according to the method shown in the embodiments or the drawings. As shown in fig. 2, the method may be applied to the server side, and the method may include:
s201: acquiring a target object;
s202: splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object;
s203: adding respective positioning information into each first sub-object to obtain a corresponding second sub-object;
s204: carrying out parallel serialization processing on the second sub-object by utilizing multiple threads;
s205: and transmitting the serialization processing result to the target receiving end.
According to the embodiment of the specification, after the target object is automatically acquired, the target object is split into a plurality of first sub-objects, the problem of transmitting one target object is converted into the problem of transmitting a plurality of smaller sub-objects, and corresponding positioning information is added to the first sub-objects to obtain corresponding second sub-objects so as to determine the splicing sequence of the sub-objects after subsequent merging and splitting, so that the transmission safety can be improved, the plurality of second sub-objects are processed in a multithreading parallel serialization manner, the efficiency of transmitting the serialization processing result to the target receiving end is improved, and the performance of remote calling service and application is guaranteed.
Specifically, in some embodiments, the target object is data to be transmitted in the remote calling process, the first sub-object is a fragment of the target object obtained by splitting the target object, each first sub-object has corresponding positioning information, the positioning information can be represented by a preset special character string (for example, a character string formed by a plurality of symbols with a low frequency of use), the positioning information can be used for representing the position of the corresponding first sub-object in the target object, and can also be understood to represent the splitting sequence number of the first sub-object, the second sub-object is a fragment with positioning information, the positioning information can be located at the beginning end and the tail end of the second sub-object, correspondingly, the positioning information located at the beginning end and the tail end can be the same or different, the positioning information can also be located at the preset position of the second sub-object, the multi-thread is a plurality of threads for parallel processing, each thread can be regarded as an execution scene or an execution unit, the serialization process of converting the object into a transmissible byte sequence is called serialization, the byte sequence is binary data formed by 0 and 1, the second sub-object is a binary data sequence formed by the two or 1, and the second sub-object is a plurality of parallel processing target objects are simultaneously processed, and the two or more than two binary data are simultaneously, and the two binary data sequences are simultaneously processed, and the target sequences are simultaneously, and the two binary data are processed.
Referring to fig. 3, in some embodiments, splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object may include:
s301: judging whether the size of the target object is larger than a first occupied space or not;
s302: if yes, splitting the target object by using the first occupied space to obtain a plurality of first sub-objects;
s303: recording the splitting ordinal number of each first sub-object and the splitting total number of the first sub-objects;
s304: and matching the split ordinal number of each first sub-object with the positioning information corresponding to the split total number from a preset positioning information table, or inputting the split ordinal number of each first sub-object and the split total number into a positioning information generation algorithm to obtain the corresponding positioning information.
Specifically, in some embodiments, the first occupied space may be understood as a threshold value for measuring whether the target object needs to be split, when the target object is smaller than the first occupied space, the target object may be directly subjected to serialization processing without splitting the target object and performing subsequent processing, in some embodiments, in order to improve security, a preset special character string may be added to the target object, then the serialization processing may be performed, when the target object is not smaller than the first occupied space, the target object may be split based on the first occupied space, for example, the target object may be split into a plurality of first sub-objects each having a size smaller than the first occupied space, or may be split into a plurality of first sub-objects each having a size smaller than the first occupied space, which is not limited herein, recording the splitting ordinal number of each first sub-object and the splitting total number of the first sub-object, wherein the splitting ordinal number represents the sequence number of all the first sub-objects obtained by splitting, the splitting total number is the total number of the first sub-objects split by the target object, positioning information corresponding to the splitting ordinal number and the splitting total number is stored in a preset positioning information table, when one target object is transmitted, the corresponding positioning information can be matched according to the splitting ordinal number and the splitting total number of the target object, or in some embodiments, the splitting ordinal number and the splitting total number of each first sub-object are input into a positioning information generating algorithm, corresponding positioning information can be obtained, for example, a positioning information generating algorithm can generate a hash value with a corresponding fixed length according to the splitting ordinal number and the splitting total number, and a character string representing the splitting ordinal number is added at the fixed position of the hash value, thereby generating positioning information.
In some embodiments, the adding the respective positioning information in each first sub-object may include adding a first positioning identifier and a second positioning identifier in the respective positioning information at a first location and a second location of each first sub-object, respectively.
It will be appreciated that in some embodiments, the first positioning identifier and the second positioning identifier are special strings that represent the splitting ordinal number and the splitting total number, further in some embodiments, the first positioning identifier may be used to represent the splitting ordinal number, the second positioning identifier may be used to represent the splitting total number, or the second positioning identifier may be used to represent the splitting ordinal number, the first positioning identifier may be used to represent the splitting total number, or the first positioning identifier and the second positioning identifier may both represent the splitting ordinal number and the splitting total number, etc., when the respective positioning information is added to each first sub-object, the first positioning identifier and the second positioning identifier are added to each first sub-object, by adding the first positioning identifier and the second positioning identifier, the security of the transmission process can be improved, and the correct splicing total number and the correct splicing order can be obtained when the target object fragments are spliced subsequently, the first sub-object before adding the positioning information can be easily determined according to the second sub-object, for example, if the positioning information is regarded as a whole to be added to the fixed position of the first sub-object, the obtained second sub-object is stored in the corresponding database, and when the sizes of the second sub-objects are not completely consistent and the second sub-objects are stored in the database in a stack manner, the position of one end of each second sub-object can only be determined when the second sub-objects are spliced, but the position of the other end can not be determined, so that the positioning information can not be directly added to the first sub-object, and in some embodiments, the adding of more than two positioning identifiers to the first sub-object can cause the waste of computing resources, in an exemplary embodiment, a first location identifier is added to the head of a first sub-object as a start location identifier and a second location identifier is added to the tail of the first sub-object as an end location identifier, resulting in a corresponding second sub-object.
Further, referring to fig. 4, in some embodiments, after transmitting the serialized processing result to the target receiving end, the method may further include:
s401: intercepting the serialized data with the length of a preset value from the same initial position of each serialized processing result;
s402: calculating a transmission state identification value corresponding to each serialized data;
s403: storing the transmission state identification value into a cache;
s404: judging whether the serialization processing result is transmitted or not through the transmission state identification value in the buffer memory, and obtaining a retransmission judgment result;
s405: and executing corresponding continuous transmission operation according to the retransmission judgment result.
Specifically, in some embodiments, the serialized data is a part of the serialized processing result, the serialized data is generally selected from the beginning of the serialized processing result, the transmission state identification value is data representing the transmission state of the serialized processing result, in some embodiments, the transmission state identification value corresponding to each serialized data may be calculated by using the Message-Digest Algorithm 5 (md5), it needs to be noted that under normal transmission conditions, the transmission state identification value does not need to be queried in a cache, however, under conditions such as network abnormality or outage, the transmission of the target object is interrupted halfway, at this time, the whole retransmission of the target object takes a long time, and resource waste is caused, and possibly the subsequent second sub-object splicing process is influenced, so that by querying whether the transmission state identification value exists in the cache, whether the corresponding serialized result is transmitted or not can be determined, and a retransmission determination result is obtained: which second sub-objects have been uploaded and which have not been uploaded, so that corresponding continuous uploading operations are performed, and the second sub-objects which have not been uploaded continue to be processed and transmitted.
It should be noted that although the operations of the method of the present invention are described in a particular order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the particular order or that all of the illustrated operations be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
Corresponding to the above-mentioned multithreaded program-serialized data transmission processing method applied to the caller, some embodiments of the present disclosure further provide a multithreaded serialized data transmission processing apparatus applied to the caller, as shown in fig. 5, in some embodiments, the apparatus may include:
a receiving module 501, configured to obtain a target object;
a splitting module 502, configured to split the target object into a plurality of first sub-objects and determine positioning information of each first sub-object;
an adding module 503, configured to add respective positioning information to each first sub-object to obtain a corresponding second sub-object;
a processing module 504, configured to perform parallel serialization processing on the second sub-object by using multiple threads;
the transmission module 505 is configured to transmit the serialization processing result to the target receiving end.
Based on the same inventive concept, referring to fig. 6, corresponding to the above-mentioned multi-threaded program columnar data transmission processing method applied to the called party, in some embodiments, a multi-threaded program columnar data transmission processing method is applied to the called party, where the method may include:
s601: receiving a plurality of serialization information sent by a calling party;
s602: performing parallel deserialization processing on the serialized information by utilizing multiple threads to obtain a second sub-object;
s603: extracting positioning information from each second sub-object, and deleting the positioning information in each second sub-object to obtain a first sub-object;
s604: determining the splicing sequence of the first sub-objects by using the positioning information extracted from the second sub-objects;
s605: and merging the first sub-objects according to the splicing sequence, so as to obtain a target object.
It may be understood that, in some embodiments, the serialization information is a byte sequence formed by 0 and 1 generated by serializing the second sub-object on the side of the caller, and the inverse serialization is a process of restoring the byte sequence into an object, in some embodiments, the corresponding second sub-object can be obtained after performing the inverse serialization processing on the plurality of serialization information, in order to avoid the waste of computing resources, and improve the accuracy of the merging result, after the positioning information is extracted from the second sub-object, it is further required to determine whether the number of sub-results of the inverse serialization processing result is equal to the total number of splits in the positioning information, if yes, the positioning information in each second sub-object is continuously deleted to obtain the first sub-object, and because the positioning information includes the split ordinal number of each first sub-object, the splicing order of each first sub-object can be determined through the positioning information, and finally, each first sub-object is merged according to the splicing order, so as to obtain the target object, and complete the processing and transmission of the target object.
Corresponding to the above-mentioned multithreaded program-serialized data transmission processing method applied to the called party, some embodiments of the present disclosure further provide a multithreaded serialized data transmission processing apparatus applied to the called party, as shown in fig. 7, and in some embodiments, the apparatus may include:
a receiving module 701, configured to receive a plurality of serialization information sent by a calling party;
a processing module 702, configured to perform parallel deserialization processing on the serialized information by using multiple threads, so as to obtain a second sub-object;
a deleting module 703, configured to extract positioning information from each second sub-object, and delete the positioning information in each second sub-object to obtain a first sub-object;
a determining module 704, configured to determine a stitching order of the first sub-objects by using the positioning information extracted from the second sub-objects;
and a merging module 705, configured to merge the first sub-objects according to the stitching order, so as to obtain a target object.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.
In the embodiments of the present disclosure, the user information (including, but not limited to, user device information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) are information and data that are authorized by the user and are sufficiently authorized by each party.
Embodiments of the present description also provide a computer device. As shown in fig. 8, in some embodiments of the present description, the computer device 802 may include one or more processors 804, such as one or more Central Processing Units (CPUs) or Graphics Processors (GPUs), each of which may implement one or more hardware threads. The computer device 802 may also include any memory 806 for storing any kind of information, such as code, settings, data, etc., in a particular embodiment a computer program on the memory 806 and executable on the processor 804, which when executed by the processor 804, may perform the instructions of the method of any of the embodiments described above. For example, and without limitation, memory 806 may include any one or more of the following combinations: any type of RAM, any type of ROM, flash memory devices, hard disks, optical disks, etc. More generally, any memory may store information using any technique. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent fixed or removable components of computer device 802. In one case, the computer device 802 may perform any of the operations of the associated instructions when the processor 804 executes the associated instructions stored in any memory or combination of memories. The computer device 802 also includes one or more drive mechanisms 808, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like, for interacting with any memory.
The computer device 802 may also include an input/output interface 810 (I/O) for receiving various inputs (via an input device 812) and for providing various outputs (via an output device 814). One particular output mechanism may include a presentation device 816 and an associated graphical user interface 818 (GUI). In other embodiments, input/output interface 810 (I/O), input device 812, and output device 814 may not be included, but merely as a computer device in a network. The computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communications buses 824 couple the above-described components together.
The communication link 822 may be implemented in any manner, such as, for example, through a local area network, a wide area network (e.g., the internet), a point-to-point connection, etc., or any combination thereof. Communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processor to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computer device. Computer readable media, as defined in the specification, does not include transitory computer readable media (transmission media), such as modulated data signals and carrier waves.
It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description embodiments may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present embodiments may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The embodiments of the specification may also be practiced in distributed computing environments where tasks are performed by remote processors that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
It should also be understood that, in the embodiments of the present specification, the term "and/or" is merely one association relationship describing the association object, meaning that three relationships may exist. For example, a and/or B may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the embodiments of the present specification. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. A multi-threaded, programmed data transmission processing method, for use with a caller, the method comprising:
acquiring a target object;
splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object;
adding respective positioning information into each first sub-object to obtain a corresponding second sub-object;
carrying out parallel serialization processing on the second sub-object by utilizing multiple threads;
and transmitting the serialization processing result to the target receiving end.
2. The method of claim 1, wherein splitting the target object into a plurality of first sub-objects and determining location information for each first sub-object comprises:
judging whether the size of the target object is larger than a first occupied space or not;
if yes, splitting the target object by using the first occupied space to obtain a plurality of first sub-objects;
recording the splitting ordinal number of each first sub-object and the splitting total number of the first sub-objects;
and matching the split ordinal number of each first sub-object with the positioning information corresponding to the split total number from a preset positioning information table, or inputting the split ordinal number of each first sub-object and the split total number into a positioning information generation algorithm to obtain the corresponding positioning information.
3. The method of claim 1, wherein adding the respective positioning information in each of the first sub-objects includes adding a first positioning identifier and a second positioning identifier in the respective positioning information at a first location and a second location of each of the first sub-objects, respectively.
4. The method of claim 1, wherein after transmitting the serialized processing result to the target receiving end, further comprises:
intercepting the serialized data with the length of a preset value from the same initial position of each serialized processing result;
calculating a transmission state identification value corresponding to each serialized data;
storing the transmission state identification value into a cache;
judging whether the serialization processing result is transmitted or not through the transmission state identification value in the buffer memory, and obtaining a retransmission judgment result;
and executing corresponding continuous transmission operation according to the retransmission judgment result.
5. A multi-threaded, programmed data transmission processing method for use with a called party, the method comprising:
receiving a plurality of serialization information sent by a calling party;
performing parallel deserialization processing on the serialized information by utilizing multiple threads to obtain a second sub-object;
extracting positioning information from each second sub-object, and deleting the positioning information in each second sub-object to obtain a first sub-object;
determining the splicing sequence of the first sub-objects by using the positioning information extracted from the second sub-objects;
and merging the first sub-objects according to the splicing sequence, so as to obtain a target object.
6. A multi-threaded, programmed data transmission processing apparatus for use with a caller, the apparatus comprising:
the receiving module is used for acquiring a target object;
the splitting module is used for splitting the target object into a plurality of first sub-objects and determining positioning information of each first sub-object;
the adding module is used for adding respective positioning information into each first sub-object to obtain a corresponding second sub-object;
the processing module is used for carrying out parallel serialization processing on the second sub-object by utilizing multiple threads;
and the transmission module is used for transmitting the serialization processing result to the target receiving end.
7. A multi-threaded, programmed data transmission processing apparatus for use with a called party, the apparatus comprising:
the receiving module is used for receiving a plurality of serialization information sent by the calling party;
the processing module is used for carrying out parallel deserialization processing on the serialized information by utilizing multiple threads to obtain a second sub-object;
the deleting module is used for extracting positioning information from each second sub-object and deleting the positioning information in each second sub-object to obtain a first sub-object;
the determining module is used for determining the splicing sequence of the first sub-objects by utilizing the positioning information extracted from the second sub-objects;
and the merging module is used for merging the first sub-objects according to the splicing sequence so as to obtain a target object.
8. A computer device comprising a memory, a processor, and a computer program stored on the memory, characterized in that the computer program, when being executed by the processor, performs the instructions of the method according to any of claims 1-5.
9. A computer storage medium having stored thereon a computer program, which, when executed by a processor of a computer device, performs the instructions of the method according to any of claims 1-5.
10. A computer program product, characterized in that the computer program product comprises a computer program which, when being executed by a processor, executes instructions of the method according to any of claims 1-5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311385860.7A CN117319474A (en) | 2023-10-24 | 2023-10-24 | Multi-line program columnar data transmission processing method and device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311385860.7A CN117319474A (en) | 2023-10-24 | 2023-10-24 | Multi-line program columnar data transmission processing method and device |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117319474A true CN117319474A (en) | 2023-12-29 |
Family
ID=89288311
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311385860.7A Pending CN117319474A (en) | 2023-10-24 | 2023-10-24 | Multi-line program columnar data transmission processing method and device |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117319474A (en) |
-
2023
- 2023-10-24 CN CN202311385860.7A patent/CN117319474A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11086825B2 (en) | Telemetry system for a cloud synchronization system | |
US20120011101A1 (en) | Integrating client and server deduplication systems | |
WO2019144761A1 (en) | Data synchronization method, distributed system and device | |
US20240004838A1 (en) | Quality score compression for improving downstream genotyping accuracy | |
KR20210005043A (en) | Input and output schema mapping | |
WO2018184535A1 (en) | Insurance service processing method and device, server, and storage medium | |
CN111221793A (en) | Data mining method, platform, computer equipment and storage medium | |
CN113806301B (en) | Data synchronization method, device, server and storage medium | |
US20140201316A1 (en) | Streaming zip | |
US11910486B2 (en) | Commit sets in a Kubernetes environment | |
CN109788251B (en) | Video processing method, device and storage medium | |
CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
US9571353B1 (en) | Compact service status processing | |
CN113934510A (en) | Mirror image processing method and device, electronic equipment and computer readable storage medium | |
CN117319474A (en) | Multi-line program columnar data transmission processing method and device | |
CN110941658A (en) | Data export method, device, server and storage medium | |
CN115757642A (en) | Data synchronization method and device based on filing log file | |
CN112291350A (en) | File transmission method, system, device and medium | |
CN112153148A (en) | Message accumulation processing method and device based on message receiving end | |
CN117435367B (en) | User behavior processing method, device, equipment, storage medium and program product | |
CN109240849A (en) | Data back up method, device and the multipoint control unit for video conferencing system | |
WO2014176954A1 (en) | Processing method, device and system for data of distributed storage system | |
CN114238479A (en) | Data import method, device, equipment and medium based on yellowbreak database | |
CN115730016B (en) | Data synchronization method, system, device, computer equipment and storage medium | |
CN113076151B (en) | Application program interaction method, device, computer equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |