CN102710768A - Service-oriented architecture-based bulk data transmission system and method - Google Patents
Service-oriented architecture-based bulk data transmission system and method Download PDFInfo
- Publication number
- CN102710768A CN102710768A CN2012101767238A CN201210176723A CN102710768A CN 102710768 A CN102710768 A CN 102710768A CN 2012101767238 A CN2012101767238 A CN 2012101767238A CN 201210176723 A CN201210176723 A CN 201210176723A CN 102710768 A CN102710768 A CN 102710768A
- Authority
- CN
- China
- Prior art keywords
- web service
- service
- data
- flow
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a service-oriented architecture (SOA)-based bulk data transmission system, which comprises a Web Service client and a Web Service server, wherein the Web Service client reads bulk data from a database, performs serialization and compression processing, and transmits the data to the Web Service server; and the Web Service server receives the bulk data from the Web Service client for decompression and serialization processing, and writes the data into a database. The invention also provides an SOA-based bulk data transmission method. By using the scheme, the Web Service has bulk data transmission capability, and the bulk data of more than 100,000 rows can be transmitted, so that the data transmission capability of an SOA application system is improved, the consumption of memory resources is reduced, and requirements for the integration of the bulk data in an enterprise are met.
Description
Technical field
The present invention relates to a kind of computer realm; Specifically; (Service-Oriented Architecture, SOA) the Web Service technological means in the field realizes the transmission system and the method for the in enormous quantities data of enterprise application in integrated to the present invention relates to a kind of utilization Service-Oriented Architecture Based.
Background technology
Service-Oriented Architecture Based (Service-Oriented Architecture; SOA) be a kind of method of constructing distributed system; It is used traditional one chip and breaks, and is decomposed into discrete, autonomous business service, utilizes their interoperability of standard upgrading; Thereby can share better, reuse and assemble, thereby make up the variation that business demand is satisfied in compound application fast.At SOA framework lower interface is to adopt the neutral mode of programming language to define, and it should be independent of hardware platform, operating system and the programming language of the service of realization.This makes the service be structured in various such systems to carry out alternately with a kind of unification and universal mode.Current application SOA framework is integrated the information system of each isomery of enterprise becomes a kind of trend, is just promoting the development in market based on the integrated approach of SOA, and mainstream vendors such as IBM, Oracle, Microsoft, SAP have turned to SOA.
Web Service is a kind of implementation of serving in the SOA framework, and in the SOA framework, service can have various ways, like corba interface, and JMS message queue etc.Web Service be use the most extensively at present, the most general a kind of service implementing mode.Web Service employing XML (extend markup language), XSD (XMLSchema), SOAP (Simple Object Access Protocol, Simple Object Access Protocol), WSDL (WSDL) etc. are independent of platform, are independent of the standard of software vendor.The service that Web Service realizes has best interaction and maximum reusability.
J2EE (Java 2 Platform Enterprise Edition; Java 2 platform, enterprise edition) being a kind of bottom technological frame of realizing Web Service, is to utilize Java 2 platforms to simplify the architecture of the relevant challenge of exploitation, deployment and the management of enterprise solution.The basis of J2EE technology is exactly J2SE (a Java standard edition); J2EE has not only consolidated the many advantages in the standard edition; For example the characteristic of " write once, operation " everywhere, the JDBC API that makes things convenient for access database, CORBA technology and can be in Internet uses the safe mode or the like of protected data, the comprehensive support to EJB (Enterprise JavaBeans), Java Servlets API, JSP (Java Server Pages) and XML technology also is provided simultaneously.Its final purpose is exactly to become an architecture that can make the developer of enterprise significantly shorten the time of putting on market.
Because Web Service itself also is not suitable for the process mass data transmission; The CPU and the memory source that analysis, processing, the conversion of XML data are needed labor; Standard Web Service is ubiquity performance bottleneck when the above data record of transmission ten thousand row (for example about 10M size XML data), can produce internal memory overflow error and possibly cause the middleware collapse.And, be ubiquitous in the transmission of the above data of enterprises ten thousand row, above technology can not satisfy the demand of enterprises transfer of data in enormous quantities.
So, be necessary to propose a kind of new solution, to solve the data transmission problems in enormous quantities of the Web Service of enterprise application in integrated.
Summary of the invention
The objective of the invention is to a kind of data transmission system in enormous quantities and method based on Service-Oriented Architecture Based; Can make Web Service possess data transmission capabilities in enormous quantities; Can realize the transmission of the above data in enormous quantities of 100,000 row; Thereby improve the data transmission capabilities of SOA framework applications system, reduce the consumption of data, satisfy the integration demand of enterprises data in enormous quantities memory source.
In order to solve above technical problem, the present invention provides a kind of data transmission system in enormous quantities based on Service-Oriented Architecture Based, comprising: Web Service client and Web Service service end,
Said Web Service client reads data in enormous quantities from the database of service consumption method, system, carry out serializing and processed compressed after, send data to Web Service service end;
Said Web Service service end decompresses after the reception data in enormous quantities from Web Service client and serializing is handled, and data are write in the database.
Further, said Web Service client is carried out the serializing processing, is that data encapsulation in enormous quantities is become the Java object array, again Java object array sequence is changed into flow object.
Further, said Web Service client is carried out the preface processed compressed, is that said Web Service client is created a compressive flow, and flow object is write compressive flow.
Further; Said Web Service client sends data to Web Service service end; Specifically comprise: Web Service client converts compressive flow to file, and file is joined in the SOAP request with attachment version, sends to Web Service service end.
Further, said Web Service service end is carried out decompression, is that the parsing of Web Service service end gets access to annex, obtains flow object, and flow object is decompressed.
Further, said Web Service service end is carried out the serializing processing, is that Web Service service end reads the Java object array through serializing mechanism from the flow data that decompression is come out, and the Java object array is write in the database.
In order to solve above technical problem, the present invention also provides a kind of data transmission method in enormous quantities based on Service-Oriented Architecture Based, comprising:
Step 1, Web Service client are obtained data in enormous quantities from database;
The data in enormous quantities that step 2, Web Service client will get access to are carried out serializing and processed compressed;
Data after step 3, Web Service client will be handled send to Web Service service end;
Step 4, Web Service service end are obtained annex through the SOAP request;
Step 5, Web Service service end decompress to the annex that obtains and serializing is handled;
Step 6, Web Service service end send to Web Service client with execution result.
Further, carrying out serializing in the said step 2 and handle, is that data encapsulation in enormous quantities is become the Java object array, again Java object array sequence is changed into flow object.
Further, carrying out the preface processed compressed in the said step 2, is that said Web Service client is created a compressive flow, and Java object array flow object is write compressive flow.
Further, said step 3 specifically comprises: Web Service client converts compressive flow to file, and file is joined in the SOAP request with attachment version, sends to Web Service service end.
Further, carrying out decompression in the said step 5, is that the parsing of Web Service service end gets access to annex, obtains flow object, and flow object is decompressed.
Further, carrying out serializing in the said step 5 and handle, is that Web Service service end reads the Java object array through serializing mechanism from the flow data that decompression is come out.
Further, said step 5 further comprises: Web Service service end writes the Java object array in the database.
Further, said step 5 further comprises: the data that Web Service service end is come out according to decompress(ion) are carried out related service and are handled operation.
Compared with prior art; The present invention provides a kind of Web Service data transmission technology in enormous quantities based on the SOA framework; The mechanism of special encapsulation and processing is carried out in employing to data in enormous quantities; After adopting Java serializing technology with data sequenceization in enormous quantities, the data after adopting the Java compress technique with serializing are again compressed, and leave in the soap message with the annex mode at last.Like this; Can make Web Service possess data transmission capabilities in enormous quantities, can realize the transmission of the above data in enormous quantities of 100,000 row, thereby improve the data transmission capabilities of SOA framework applications system; Reduce the consumption of data, satisfy the integration demand of enterprises data in enormous quantities memory source.
Description of drawings
Accompanying drawing described herein is used to provide further understanding of the present invention, constitutes a part of the present invention, and illustrative examples of the present invention and explanation thereof are used to explain the present invention, does not constitute improper qualification of the present invention.In the accompanying drawings:
Fig. 1 is the structural representation that the present invention provides a kind of data transmission system in enormous quantities based on SOA.
Fig. 2 is the flow chart that the present invention provides a kind of data transmission method in enormous quantities based on SOA.
Embodiment
In order to make technical problem to be solved by this invention, technical scheme and beneficial effect clearer, clear,, the present invention is further elaborated below in conjunction with accompanying drawing and embodiment.Should be appreciated that specific embodiment described herein only in order to explanation the present invention, and be not used in qualification the present invention.
As shown in Figure 1, the present invention provides a kind of data transmission system in enormous quantities based on Service-Oriented Architecture Based SOA, comprising: Web Service client 10 and Web Service service end 20.Wherein,
Web Service client is carried out the serializing processing, is that Web Service client becomes the Java object array with data encapsulation in enormous quantities, again Java object array sequence is changed into flow object.
Web Service client is carried out the preface processed compressed, is that said Web Service client is created a compressive flow, and flow object is write compressive flow.
Web Service client sends data to Web Service service end, specifically comprises: Web Service client converts compressive flow to file, and file is joined in the SOAP request with attachment version, sends to Web Service service end.
Web Service service end 20; Be deployed in the Web Service provider system (SOA platform method, system), this Web Service service end 20 externally provides Web Service interface, from Web Service client 10, decompresses after the reception data in enormous quantities and the serializing processing; Transfer data to object; Write in the database 22, and carry out related service according to the data of decompress(ion) and handle operation, execution result is returned to Web Service client 10.Wherein,
Web Service service end is carried out decompression, is that the parsing of Web Service service end gets access to annex, obtains flow object, and flow object is decompressed.
Web Service service end is carried out the serializing processing, is that Web Service service end reads the Java object array through serializing mechanism from the flow data that decompression is come out, and the Java object array is write in the database.
As shown in Figure 2, the present invention provides a kind of data transmission method in enormous quantities based on Service-Oriented Architecture Based SOA, may further comprise the steps:
Step 1, Web Service client are obtained data in enormous quantities from database.
Step 1.1, Web Service client are obtained data in enormous quantities through JDBC mode accessing database from database.
The data in enormous quantities that step 2, Web Service client will get access to are carried out serializing and processed compressed, specifically comprise:
Step 2.1, Web Service client become the Java object array with data encapsulation in enormous quantities, the corresponding data-base recording of each element in the array.
Step 2.2, Web Service client change into flow object with Java object array sequence.
Step 2.3, Web Service client are created a compressive flow, and Java object array flow object is write compressive flow.
Step 3, Web Service client convert compressive flow to annex and join in the SOAP request, send to Web Service service end; Specifically comprise:
Step 3.1, Web Service client convert compressive flow to file.
Step 3.2, Web Service client join file in the SOAP request with attachment version, send to Web Service service end.
Step 4, Web Service service end are obtained annex through the SOAP request.
Step 5, Web Service service end decompress to the annex that obtains and serializing is handled.
Step 5.1, Web Service service end are resolved and are got access to annex, obtain flow object.
Step 5.2, Web Service service end decompress to flow object.
Step 5.3, Web Service service end read the Java object array through serializing mechanism from the flow data that decompression is come out.
Step 5.4, Web Service service end are carried out related service according to the data of decompress(ion) and are handled operation.
Step 5.5, Web Service service end write the Java object array in the database.
Step 6, Web Service service end send to Web Service client with service executing result.
The present invention provides a kind of Web Service data transmission technology in enormous quantities based on the SOA framework; After adopting Java serializing technology with data sequenceization in enormous quantities; Data after adopting the Java compress technique with serializing are again compressed, and leave in the soap message with the annex mode at last.Like this; Can make Web Service possess data transmission capabilities in enormous quantities, can realize the transmission of the above data in enormous quantities of 100,000 row, thereby improve the data transmission capabilities of SOA framework applications system; Reduce the consumption of data, satisfy the integration demand of enterprises data in enormous quantities memory source.
Above-mentioned explanation illustrates and has described a preferred embodiment of the present invention; But as previously mentioned; Be to be understood that the present invention is not limited to the form that this paper discloses, should do not regard eliminating as, and can be used for various other combinations, modification and environment other embodiment; And can in invention contemplated scope described herein, change through the technology or the knowledge of above-mentioned instruction or association area.And change that those skilled in the art carried out and variation do not break away from the spirit and scope of the present invention, then all should be in the protection range of accompanying claims of the present invention.
Claims (14)
1. the data transmission system in enormous quantities based on Service-Oriented Architecture Based is characterized in that, comprising: Web Service client and Web Service service end,
Said Web Service client reads data in enormous quantities from the database of service consumption method, system, carry out serializing and processed compressed after, send data to Web Service service end;
Said Web Service service end decompresses after the reception data in enormous quantities from Web Service client and serializing is handled, and data are write in the database.
2. the system of claim 1 is characterized in that, said Web Service client is carried out the serializing processing, is that data encapsulation in enormous quantities is become the Java object array, again Java object array sequence is changed into flow object.
3. system as claimed in claim 2 is characterized in that, said Web Service client is carried out the preface processed compressed, is that said Web Service client is created a compressive flow, and flow object is write compressive flow.
4. system as claimed in claim 3; It is characterized in that; Said Web Service client sends data to Web Service service end; Specifically comprise: Web Service client converts compressive flow to file, and file is joined in the SOAP request with attachment version, sends to Web Service service end.
5. system as claimed in claim 4 is characterized in that, said Web Service service end is carried out decompression, is that the parsing of Web Service service end gets access to annex, obtains flow object, and flow object is decompressed.
6. system as claimed in claim 5; It is characterized in that; Said Web Service service end is carried out the serializing processing, is that Web Service service end reads the Java object array through serializing mechanism from the flow data that decompression is come out, and the Java object array is write in the database.
7. the data transmission method in enormous quantities based on Service-Oriented Architecture Based is characterized in that, comprising:
Step 1, Web Service client are obtained data in enormous quantities from database;
The data in enormous quantities that step 2, Web Service client will get access to are carried out serializing and processed compressed;
Data after step 3, Web Service client will be handled send to Web Service service end;
Step 4, Web Service service end are obtained annex through the SOAP request;
Step 5, Web Service service end decompress to the annex that obtains and serializing is handled;
Step 6, Web Service service end send to Web Service client with execution result.
8. method as claimed in claim 7 is characterized in that, carries out serializing in the said step 2 and handles, and be that data encapsulation in enormous quantities is become the Java object array, again Java object array sequence is changed into flow object.
9. method as claimed in claim 8 is characterized in that, carries out the preface processed compressed in the said step 2, is that said Web Service client is created a compressive flow, and Java object array flow object is write compressive flow.
10. method as claimed in claim 9 is characterized in that, said step 3 specifically comprises: Web Service client converts compressive flow to file, and file is joined in the SOAP request with attachment version, sends to Web Service service end.
11. method as claimed in claim 7 is characterized in that, carries out decompression in the said step 5, is that the parsing of Web Service service end gets access to annex, obtains flow object, and flow object is decompressed.
12. method as claimed in claim 11 is characterized in that, carries out serializing in the said step 5 and handles, and is that Web Service service end reads the Java object array through serializing mechanism from the flow data that decompression is come out.
13. method as claimed in claim 12 is characterized in that, said step 5 further comprises: Web Service service end writes the Java object array in the database.
14. method as claimed in claim 12 is characterized in that, said step 5 further comprises: the data that Web Service service end is come out according to decompress(ion) are carried out related service and are handled operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101767238A CN102710768A (en) | 2012-05-31 | 2012-05-31 | Service-oriented architecture-based bulk data transmission system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101767238A CN102710768A (en) | 2012-05-31 | 2012-05-31 | Service-oriented architecture-based bulk data transmission system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102710768A true CN102710768A (en) | 2012-10-03 |
Family
ID=46903314
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012101767238A Pending CN102710768A (en) | 2012-05-31 | 2012-05-31 | Service-oriented architecture-based bulk data transmission system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102710768A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820668A (en) * | 2014-01-31 | 2015-08-05 | 雅虎公司 | Compression of serialized data for communication from a client-side application |
| CN106713463A (en) * | 2016-12-27 | 2017-05-24 | 浪潮软件集团有限公司 | High-performance cross-network data transmission and import method |
| CN107229670A (en) * | 2017-04-10 | 2017-10-03 | 中国科学院信息工程研究所 | Conventional data serializing and unserializing method based on Avro |
| CN107948334A (en) * | 2018-01-09 | 2018-04-20 | 无锡华云数据技术服务有限公司 | Data processing method based on distributed memory system |
| CN108512865A (en) * | 2017-02-23 | 2018-09-07 | 北京宝兰德软件股份有限公司 | A kind of data transmission method and device of the Enterprise Java Bean container system of Java application servers |
| CN109862095A (en) * | 2019-01-30 | 2019-06-07 | 新华三大数据技术有限公司 | Data processing method and device based on Web service |
| CN112883367A (en) * | 2021-01-26 | 2021-06-01 | 北京高因科技有限公司 | Trigger data secure transmission method and device |
| CN114201114A (en) * | 2021-12-14 | 2022-03-18 | 上海朝阳永续信息技术股份有限公司 | Cross-internet data transmission system based on files |
| CN114201114B (en) * | 2021-12-14 | 2024-05-10 | 上海朝阳永续信息技术股份有限公司 | Cross-Internet data transmission system based on files |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070136492A1 (en) * | 2005-12-08 | 2007-06-14 | Good Technology, Inc. | Method and system for compressing/decompressing data for communication with wireless devices |
| CN102111437A (en) * | 2010-12-30 | 2011-06-29 | 浪潮集团山东通用软件有限公司 | Simple object access protocol (SOAP) extension-based WEB service transmission compression method |
-
2012
- 2012-05-31 CN CN2012101767238A patent/CN102710768A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070136492A1 (en) * | 2005-12-08 | 2007-06-14 | Good Technology, Inc. | Method and system for compressing/decompressing data for communication with wireless devices |
| CN102111437A (en) * | 2010-12-30 | 2011-06-29 | 浪潮集团山东通用软件有限公司 | Simple object access protocol (SOAP) extension-based WEB service transmission compression method |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820668A (en) * | 2014-01-31 | 2015-08-05 | 雅虎公司 | Compression of serialized data for communication from a client-side application |
| CN104820668B (en) * | 2014-01-31 | 2019-06-07 | 奥誓公司 | Compressed sequence is carried out to transmit from client application to data |
| CN106713463A (en) * | 2016-12-27 | 2017-05-24 | 浪潮软件集团有限公司 | High-performance cross-network data transmission and import method |
| CN108512865A (en) * | 2017-02-23 | 2018-09-07 | 北京宝兰德软件股份有限公司 | A kind of data transmission method and device of the Enterprise Java Bean container system of Java application servers |
| CN108512865B (en) * | 2017-02-23 | 2021-04-16 | 北京宝兰德软件股份有限公司 | Data transmission method and device of EJB container system of Java application server |
| CN107229670A (en) * | 2017-04-10 | 2017-10-03 | 中国科学院信息工程研究所 | Conventional data serializing and unserializing method based on Avro |
| CN107948334A (en) * | 2018-01-09 | 2018-04-20 | 无锡华云数据技术服务有限公司 | Data processing method based on distributed memory system |
| CN109862095A (en) * | 2019-01-30 | 2019-06-07 | 新华三大数据技术有限公司 | Data processing method and device based on Web service |
| CN112883367A (en) * | 2021-01-26 | 2021-06-01 | 北京高因科技有限公司 | Trigger data secure transmission method and device |
| CN114201114A (en) * | 2021-12-14 | 2022-03-18 | 上海朝阳永续信息技术股份有限公司 | Cross-internet data transmission system based on files |
| CN114201114B (en) * | 2021-12-14 | 2024-05-10 | 上海朝阳永续信息技术股份有限公司 | Cross-Internet data transmission system based on files |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8010695B2 (en) | Web services archive | |
| US7814060B2 (en) | Apparatus and method for web service client deployment | |
| US8146096B2 (en) | Method and system for implementing built-in web services endpoints | |
| US8024425B2 (en) | Web services deployment | |
| CN102710768A (en) | Service-oriented architecture-based bulk data transmission system and method | |
| US7788338B2 (en) | Web services message processing runtime framework | |
| US9852116B2 (en) | System and method for processing messages using native data serialization/deserialization in a service-oriented pipeline architecture | |
| US7716360B2 (en) | Transport binding for a web services message processing runtime framework | |
| US8745252B2 (en) | Headers protocol for use within a web services message processing runtime framework | |
| Indrasiri et al. | gRPC: up and running: building cloud native applications with Go and Java for Docker and Kubernetes | |
| US7716279B2 (en) | WS addressing protocol for web services message processing runtime framework | |
| US20070156872A1 (en) | Method and system for Web services deployment | |
| US20100241729A1 (en) | Web Services Message Processing Runtime Framework | |
| US8806506B2 (en) | System and method for processing messages using a common interface platform supporting multiple pluggable data formats in a service-oriented pipeline architecture | |
| US8549474B2 (en) | Method and system for implementing WS-policy | |
| US7721293B2 (en) | Web services hibernation | |
| US8495594B2 (en) | Method and system for providing a componentized resource adapter architecture | |
| US20070067475A1 (en) | Runtime execution of a reliable messaging protocol | |
| US20020019884A1 (en) | Accessing legacy applications from the internet | |
| US20100082737A1 (en) | Dynamic service routing | |
| US20060271911A1 (en) | Generating executable objects implementing methods for an information model | |
| CN102710769A (en) | Web Service client and bulk data transmission method thereof | |
| CN102694865A (en) | Web Service server and mass data transmission method thereof | |
| US20090106280A1 (en) | Semantic-Based Lossy Compression | |
| US20110321022A1 (en) | Code generation through metadata programming for mobile devices and web platforms to ease access to web services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB02 | Change of applicant information |
Address after: 518057 Guangdong city of Shenzhen province Nanshan District South Road seven No. 002 Shenzhen Digital Technology Park B1 building 6 floor A District No. 1 Applicant after: Shenzhen Vispractice Technology Corporation Address before: 518057, Guangdong Province, Nanshan District hi tech Zone, North Road, Lang Lang, No. 13 Thunis building, C,, C302 Applicant before: Shenzhen Vispractice Technology Corporation |
|
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20121003 |