CN105867928B - A kind of method and apparatus accessing specified computation model in specified distributed system - Google Patents

A kind of method and apparatus accessing specified computation model in specified distributed system Download PDF

Info

Publication number
CN105867928B
CN105867928B CN201610192917.5A CN201610192917A CN105867928B CN 105867928 B CN105867928 B CN 105867928B CN 201610192917 A CN201610192917 A CN 201610192917A CN 105867928 B CN105867928 B CN 105867928B
Authority
CN
China
Prior art keywords
interface
distributed system
class
call request
computation model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610192917.5A
Other languages
Chinese (zh)
Other versions
CN105867928A (en
Inventor
覃侃
王磊
钟靓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Qihoo Technology Co Ltd
Original Assignee
Beijing Qihoo Technology Co Ltd
Qizhi Software Beijing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Qihoo Technology Co Ltd, Qizhi Software Beijing Co Ltd filed Critical Beijing Qihoo Technology Co Ltd
Priority to CN201610192917.5A priority Critical patent/CN105867928B/en
Publication of CN105867928A publication Critical patent/CN105867928A/en
Application granted granted Critical
Publication of CN105867928B publication Critical patent/CN105867928B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/30—Creation or generation of source code
    • G06F8/31—Programming languages or programming paradigms
    • G06F8/315—Object-oriented languages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
    • G06F9/4482—Procedural

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Multi Processors (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of method and apparatus that specified computation model is accessed in specified distributed system, and specifying distributed system is first kind distributed system, and specified computation model supports the second class distributed system but do not support first kind distributed system；This method comprises: obtaining the interface list of first kind distributed system and the second class distributed system, the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system are established；Intercept the call request for the interface about the second class distributed system that specified computation model is sent；Call request about the interface of the second class distributed system is converted into the call request of the corresponding interface about first kind distributed system, is sent to first kind distributed system；Reception, which returns the result and is sent to specified computation model, carries out calculation processing.The program, which does not change former distributed structure/architecture, ensure that the stability and safety of data, and the specified computation model accessed in former distributed structure/architecture meets corresponding process demand.

Description

A kind of method and apparatus accessing specified computation model in specified distributed system
Technical field
The present invention relates to data mining processing technology fields, and in particular to a kind of access in specified distributed system is specified The method and apparatus of computation model.
Background technique
With the arriving of big data era, the data class of burgeoning data volume, many and diverse multiplicity changes very fast number According to form, more rapidly and effectively mass data processing mode is needed, distributed system is come into being, in order to be adapted to constantly send out The demand of exhibition, during various distributed systems are constantly in update differentiation.In many cases, different editions, classification point Cloth system is not compatible, by taking first kind distributed system and the second class distributed system as an example, in face of supporting the second class point Cloth system but when not supporting the specified computation model of first kind distributed system, if first kind distributed system is wanted to use First kind distributed system, must be just changed to the second class distributed system, be directed to first by specified computation model The change of the framework of class distributed system will be brought once leading to loss of data due to fault during change to user Immeasurable loss.
Summary of the invention
In view of the above problems, it proposes on the present invention overcomes the above problem or at least be partially solved in order to provide one kind State the method and apparatus that specified computation model is accessed in specified distributed system of problem.
According to one aspect of the present invention, a kind of side that specified computation model is accessed in specified distributed system is provided Method, wherein the specified distributed system is first kind distributed system, and the specified computation model is to support the distribution of the second class Formula system but the computation model for not supporting first kind distributed system；
This method comprises:
The interface list of first kind distributed system is obtained, the interface list of the second class distributed system is obtained, establishes the Mapping relations between the interface of a kind of distributed system and the interface of the second class distributed system；
Intercept the call request for the interface about the second class distributed system that the specified computation model is sent；
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, by institute It states and is converted into the corresponding interface about first kind distributed system about the call request of the interface of the second class distributed system Call request, and it is sent to first kind distributed system；
Receive first kind distributed system returns the result and is sent to the specified computation model, by the specified calculating Model returns the result carry out calculation processing according to.
It optionally, include: interface name, interface ginseng in the call request of the interface about the second class distributed system Number；
The mapping relations according between the interface of first kind distributed system and the interface of the second class distributed system, The call request of the interface about the second class distributed system is converted into accordingly connecing about first kind distributed system Mouthful call request include:
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, obtain The interface name of the corresponding first kind distributed system of the interface name；
According to the interface name of acquired first kind distributed system, the format of the parameter information is converted into being adapted to In the format of the interface of corresponding first kind distributed system；
The call request of the corresponding interface about first kind distributed system is obtained, includes: acquired in the call request First kind distributed system interface name, format conversion after parameter information.
Optionally, when the interface name of the corresponding first kind distributed system of the interface name has not been obtained, the party Method further comprises:
According to the interface name and parameter information in the call request of the interface about the second class distributed system, wound Build a dynamic proxy object；
According to the interface name, looked for from the interface list of first kind distributed system using the dynamic proxy object To the interface of the first kind distributed system with respective calls effect；
Using about the call request of the interface of the first kind distributed system as described about first kind distributed system The corresponding interface call request.
Optionally, the interface list for obtaining first kind distributed system, obtains the interface of the second class distributed system List, the mapping relations established between the interface of first kind distributed system and the interface of the second class distributed system include:
By the information for calling the interface list of the first Context class storage first kind distributed system；
By the information for calling the interface list of the 2nd Context class the second class distributed system of storage；
The reflection from the 2nd Context class to the first Context class is realized using reflection mechanism, establishes first kind distribution Mapping relations between the interface of system and the interface of the second class distributed system.
Optionally, the interface list for obtaining first kind distributed system, obtains the interface of the second class distributed system List includes:
The corresponding first Jar packet of information of the interface list of first kind distributed system is stored, the second class of storage is distributed The corresponding 2nd Jar packet of the information of the interface list of system；Wherein, the first Jar packet and the 2nd Jar packet pass through respectively Corresponding Classloader Classloader is loaded.
Optionally, the calling for intercepting the interface about the second class distributed system that the specified computation model is sent Request includes: to intercept the connecing about the second class distributed system that the specified computation model is sent according to the 2nd Jar packet The call request of mouth；
The call request by the interface about the second class distributed system is converted into about first kind distribution The call request of the corresponding interface of system includes: to load the first Jar packet by corresponding Classloader Classloader, Obtain the call request of the corresponding interface about first kind distributed system.
Optionally, the call request of the interface about the second class distributed system includes: about the second class distribution The call request of the interface of the reading data of system, and/or, the calling of the interface for writing data about the second class distributed system is asked It asks；
It include: the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data of interface；
Writing for the second class distributed system includes: that the second class distributed system writes data in the interface of data The corresponding method for writing data of interface.
Optionally, the call request of described the corresponding interface about first kind distributed system includes: about the first kind point The call request of the interface of the reading data of cloth system, and/or, the tune of the interface for writing data about first kind distributed system With request；
It include: the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data of interface；
Writing for the first kind distributed system includes: that the first kind distributed system writes data in the interface of data The corresponding method for writing data of interface.
Optionally, the first kind distributed system is the Hadoop system of first version, the second class distribution system System is the Hadoop system of the second edition, and the specified computation model includes following one or more: Yarn computation model, Spark computation model, Sqoop computation model.
According to another aspect of the invention, it provides a kind of access in specified distributed system and specifies computation model Device, wherein the specified distributed system is first kind distributed system, and the specified computation model is to support the second class point Cloth system but the computation model for not supporting first kind distributed system；
The device includes:
It is distributed to obtain the second class suitable for obtaining the interface list of first kind distributed system for interface message preset unit The interface list of system, the mapping established between the interface of first kind distributed system and the interface of the second class distributed system are closed System；
Call request converting unit, suitable for intercepting that the specified computation model sends about the second class distributed system The call request of interface；It is closed according to the mapping between the interface of first kind distributed system and the interface of the second class distributed system System, the call request of the interface about the second class distributed system is converted into about the corresponding of first kind distributed system The call request of interface；
Processing unit is called, the call request suitable for the corresponding interface about first kind distributed system after converting is sent out It send to first kind distributed system；Receive first kind distributed system returns the result and is sent to the specified computation model, Carry out calculation processing is returned the result according to as the specified computation model.
It optionally, include: interface name, interface ginseng in the call request of the interface about the second class distributed system Number；
The call request converting unit, suitable for the interface and the second class distributed system according to first kind distributed system Interface between mapping relations, obtain the interface name of the corresponding first kind distributed system of the interface name；According to institute The format of the parameter information is converted into adapting to the corresponding first kind by the interface name of the first kind distributed system of acquisition The format of the interface of distributed system；The call request of the corresponding interface about first kind distributed system is obtained, which asks It include: the interface name of acquired first kind distributed system in asking, the parameter information after format conversion.
Optionally, the call request converting unit is further adapted for that the interface name corresponding is having not been obtained When the interface name of a kind of distributed system, according to connecing in the call request of the interface about the second class distributed system Mouth title and parameter information, create a dynamic proxy object；According to the interface name, using the dynamic proxy object from The interface of the first kind distributed system with respective calls effect is found in the interface list of first kind distributed system；It will close In the first kind distributed system interface call request as described the corresponding interface about first kind distributed system Call request.
Optionally, the interface message preset unit is suitable for distributed by calling the first Context class to store the first kind The information of the interface list of system；By the letter for calling the interface list of the 2nd Context class the second class distributed system of storage Breath；The reflection from the 2nd Context class to the first Context class is realized using reflection mechanism, establishes first kind distributed system Interface and the second class distributed system interface between mapping relations.
Optionally, the interface message preset unit, the information of the interface list suitable for storing first kind distributed system Corresponding first Jar packet stores the corresponding 2nd Jar packet of information of the interface list of the second class distributed system；Wherein, described First Jar packet and the 2nd Jar packet are loaded by corresponding Classloader Classloader respectively.
Optionally, the call request converting unit is suitable for intercepting the specified computation model according to the 2nd Jar packet The call request of the interface about the second class distributed system sent；Suitable for passing through corresponding Classloader Classloader The first Jar packet is loaded, the call request of the corresponding interface about first kind distributed system is obtained.
Optionally, the call request of the interface about the second class distributed system includes: about the second class distribution The call request of the interface of the reading data of system, and/or, the calling of the interface for writing data about the second class distributed system is asked It asks；
It include: the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data of interface；
Writing for the second class distributed system includes: that the second class distributed system writes data in the interface of data The corresponding method for writing data of interface.
Optionally, the call request of described the corresponding interface about first kind distributed system includes: about the first kind point The call request of the interface of the reading data of cloth system, and/or, the tune of the interface for writing data about first kind distributed system With request；
It include: the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data of interface；
Writing for the first kind distributed system includes: that the first kind distributed system writes data in the interface of data The corresponding method for writing data of interface.
Optionally, the first kind distributed system is the Hadoop system of first version, the second class distribution system System is the Hadoop system of the second edition, and the specified computation model includes following one or more: Yarn computation model, Spark computation model, Sqoop computation model.
According to the technique and scheme of the present invention, in face of supporting the second class distributed system but not supporting first kind distribution system System specified computation model when, can by by specified computation model call the second class distributed system interface call request The call request for calling the equivalent interface of first kind distributed system is converted to, realizes that specified computation model is distributed the first kind The indirect calling of the method for formula system, so that first kind distributed system and related data can be based on fingers without making any change Determine computation model and carries out relevant calculation processing.The program is suitable for those using first kind distributed system as data processing architecture Again at the same time it is wished that the user calculated using specified computation model, not changing former data processing architecture ensure that the stabilization of data Property and safety, and access specified computation model meet corresponding process demand.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention, And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings:
Fig. 1 shows a kind of access in specified distributed system according to an embodiment of the invention and specifies computation model Method flow chart；
Fig. 2 shows Yarn computation models according to an embodiment of the invention, the Hadoop system of first version and The interaction block diagram of the Hadoop system of two versions；
Fig. 3 shows a kind of access in specified distributed system according to an embodiment of the invention and specifies computation model Device schematic diagram.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure It is fully disclosed to those skilled in the art.
Fig. 1 shows a kind of access in specified distributed system according to an embodiment of the invention and specifies computation model Method flow chart, wherein the specified distributed system be first kind distributed system, the specified computation model be branch It holds the second class distributed system but does not support the computation model of first kind distributed system；As shown in Figure 1, this method comprises:
Step S110 obtains the interface list of first kind distributed system, obtains the interfaces column of the second class distributed system Table establishes the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system.
Step S120, the calling for intercepting the interface about the second class distributed system that the specified computation model is sent are asked It asks.
Step S130, according to the mapping between the interface of first kind distributed system and the interface of the second class distributed system The call request of the interface about the second class distributed system is converted into the phase about first kind distributed system by relationship The call request of interface is answered, and is sent to first kind distributed system.
Step S140 receives returning the result and being sent to the specified computation model for first kind distributed system, by institute It states specified computation model and returns the result carry out calculation processing according to.
As it can be seen that method shown in FIG. 1 is in face of supporting the second class distributed system but not supporting first kind distributed system Specified computation model, by calling specified computation model the call request of interface of the second class distributed system to be converted to calling The call request of the equivalent interface of first kind distributed system realizes specified computation model to first kind distributed system The indirect calling of method, so that first kind distributed system and related data can calculate mould based on specified without making any change Type carries out relevant calculation processing.The program is suitable for those and wishes simultaneously again using first kind distributed system as data processing architecture Hope the user calculated using specified computation model, not changing former data processing architecture ensure that the stability and safety of data Property, and the specified computation model accessed meets corresponding process demand.
In one embodiment of the invention, the connecing about the second class distributed system that above-mentioned specified computation model is sent It include: interface name, interface parameters in the call request of mouth；Interface name is the interface of the interface in the second class distributed system Title, interface parameters are interface parameters compatible with the interface；Specified computation model send about the second class distribution system The call request of the interface of system refers to the call request for determining computation model to some method in the second class distributed system, wherein Method is corresponding with interface.
The interface for the first kind distributed system established in above-mentioned steps S110 connects with the second class distributed system In mapping relations between mouthful between an interface of first kind distributed system and an interface of the second class distributed system With mapping relations, illustrate that method corresponding to the two interfaces is equivalent.For some interfaces, by interface name and connect Mouth parameter can directly determine mapping relations, for other interfaces, can not determine mapping by interface name and interface parameters Relationship needs to safeguard its mapping relations by adapter class, is dynamically, to need to analyze its corresponding method there are also some interfaces Implementation principle determine its mapping relations.
Then step S130 is according to reflecting between the interface of first kind distributed system and the interface of the second class distributed system Relationship is penetrated, the call request of the interface about the second class distributed system is converted into about first kind distributed system The call request of the corresponding interface includes:
Step S131, according to the mapping between the interface of first kind distributed system and the interface of the second class distributed system Relationship obtains the interface name in the call request of the interface about the second class distributed system intercepted corresponding the The interface name of a kind of distributed system obtains and wants connecing in the second class distributed system called with specified computation model The interface name of interface in the equivalent first kind distributed system of mouth.
Step S132 will be intercepted about the second class according to the interface name of acquired first kind distributed system The format of the parameter information in the call request of the interface of distributed system is converted into adapting to corresponding first kind distribution The format of the interface of formula system.
For example, the parameter about user information in the interface a of the second class distributed system is user.name, it is right therewith The parameter about user information in the interface a ' for the first kind distributed system answered is job.ugi, then will be about interface a's During call request is converted to the call request of calling interface a ', the parameter information about user information is from user.name Be converted to job.ugi.
Step S133 obtains the call request of the corresponding interface about first kind distributed system, wraps in the call request It includes: the interface name of acquired first kind distributed system, the parameter information after format conversion.
As it can be seen that the call request in the interface about the second class distributed system for sending specified computation model is converted to During call request about the equivalent interface of first kind distributed system, the second class for needing to find and be called is divided The interface of the equivalent first kind distributed system of the interface of cloth system, it is also necessary to the format of interface parameters be converted to and adapted to The format of the interface of the first kind distributed system found, so that the call request after conversion fundamentally adapts to the first kind The interface of distributed system, call request after enabling first kind distributed system to identify the conversion and providing are returned accordingly Return result.
Further, in one embodiment of the invention, when the method etc. for being related to dynamic socket abstract class does not have One by one when the interface of mapping relations, it is corresponding that the interface name can not be got during being called the conversion of request The interface name of first kind distributed system is not done the conversion of call request further according to mapping relations at this time then, but is used The mode of dynamic proxy is called.
Specifically, be called by the way of dynamic proxy includes: according to described about the second class distributed system Interface call request in interface name and parameter information, create a dynamic proxy object；According to the interface name, First with respective calls effect is found from the interface list of first kind distributed system using the dynamic proxy object The interface of class distributed system；Using about the call request of the interface of the first kind distributed system as described about the first kind The call request of the corresponding interface of distributed system.
Hereafter illustrate the specific implementation process of this programme by one embodiment: in the present embodiment, first kind distribution Formula system is the Hadoop system of first version, and the second class distributed system is the Hadoop system of the second edition, specifies and calculates Model is Yarn computation model, and Yarn computation model supports the Hadoop system of the second edition but do not support first version Hadoop system, the Hadoop system of first version and the Hadoop system of the second edition are not compatible.All data of company A It is all runed in the Hadoop system of first version, since data volume is huge, loss of data risk may occur in escalation process, The Hadoop system of first version cannot be upgraded to the Hadoop system of the second edition by company A, and due to Yarn computation model It can be realized unified resource management and realize that High Availabitity, company A wish the Hadoop system in presently used first version Yarn computation model is accessed in system.
For this purpose, creating a Spinner component, the Spinner according to the Hadoop system that this programme is first version The function of component includes: the interface that those are corresponded to fixing means, by calling the first Context class to store first version Hadoop system interface list information, by calling the 2nd Context class to store the Hadoop system of the second edition The information of interface list is realized the reflection from the 2nd Context class to the first Context class using Java reflection mechanism, is established Mapping relations between the interface of the Hadoop system of the interface and second edition of the Hadoop system of first version.
In the process of implementation, the Hadoop system about the second edition that Spinner interception component Yarn computation model is sent The call request of the interface of system, according to the interface of the interface of the Hadoop system of first version and the Hadoop system of the second edition Between mapping relations, the call request of the interface of the Hadoop system about the second edition is converted into about the first edition The call request of the corresponding interface of this Hadoop system, is directed to the conversion of interface name and the conversion of interface parameters, And it is sent to the Hadoop system of first version.The Hadoop system of first version receive it is described about first version The corresponding method of the corresponding interface and parameter can be returned to Spinner group after the call request of the corresponding interface of Hadoop system Part, Spinner component, which is returned again to, carries out relevant treatment to Yarn computation model.
For example, the HDFS for the Hadoop system about the second edition that Spinner interception component Yarn computation model is sent The call request of the interface of the reading data of (Hadoop Distribute File System, distributed file system), and/or, The call request of the interface for writing data of the HDFS of Hadoop system about the second edition；Wherein, the Hadoop of the second edition It include: the method that data are read from the HDFS of the Hadoop system of the second edition in the interface of the reading data of the HDFS of system；The Writing for the HDFS of the Hadoop system of two versions includes: to write in the interface of data in the HDFS of the Hadoop system of the second edition The method of data.
Spinner component turns the call request about the interface of the reading data of the HDFS of the Hadoop system of the second edition It is changed to the call request of the interface of the reading data of the HDFS of the Hadoop system about first version, and is sent to first version Returning the result for the Hadoop system of first version is transmitted to Yarn computation model by Hadoop system, it is seen then that Yarn calculates mould The method that the script of type to be called is: the method for data is read from the HDFS of the Hadoop system of the second edition；And pass through The intermediate effect of Spinner, so that the method that Yarn computation model finally calls is: from the Hadoop system of first version The method of data is read in HDFS；And then after calling successfully, Yarn computation model can be from the Hadoop system of first version Specified data are read in HDFS to be calculated.Calling writes the process of the method for data similarly, and details are not described herein.
As it can be seen that Spinner component plays the role of translation, conversion in centre, to realize the Hadoop system of first version With the interaction of Yarn computation model, Fig. 2 shows Yarn computation models according to an embodiment of the invention, first version The interaction block diagram of the Hadoop system of Hadoop system and the second edition, bottom are the realizations of HDFS, are called accumulation layer, it is intermediate this One layer includes Spinner component, the Hadoop system of first version and the Hadoop system of the second edition, is called data interaction Layer, top layer is the realization of Yarn computation model, is called computation layer, and script computation layer can by the Hadoop system of the second edition Directly to access HDFS, but in the present embodiment, computation layer be sent to the call request of the Hadoop system of the second edition all by Spinner component converts the Hadoop system for being sent to first version, accesses HDFS by the Hadoop system of first version.
In one embodiment of the invention, turn by the call request about the interface of the Hadoop system of the second edition During the call request for changing the corresponding interface of the Hadoop system about first version into, have a problem that: if the There is the identical interface of title identical parameters in the Hadoop system of two versions and the Hadoop system of first version, turn calling It may be clashed during changing, the Classloader Classloader of Java offer is utilized in this this programme, Which Jar packet Classloader can will currently load with customized, and specific method includes:
The corresponding first Jar packet of information of the interface list of the Hadoop system of first version is stored, the second edition is stored Hadoop system interface list the corresponding 2nd Jar packet of information；Wherein, the first Jar packet and the 2nd Jar packet It is loaded respectively by corresponding Classloader Classloader；Then Spinner component is intercepted according to the 2nd Jar packet The call request of the interface for the Hadoop system about the second edition that Yarn computation model is sent；It will be about the second edition The call request of the interface of Hadoop system is converted into the call request of the corresponding interface of the Hadoop system about first version, The first Jar packet is loaded by corresponding Classloader Classloader, obtains the Hadoop system about first version The call request of the corresponding interface.This sample plan is just two same names and identical parameters in Classloader level Interface is kept apart, and calling conflict will not be generated.
In other embodiments of the invention, specified computation model can also be that Yarn computation model, Spark calculate mould Other computation models such as type, Sqoop computation model, here with no restrictions.
Fig. 3 shows a kind of access in specified distributed system according to an embodiment of the invention and specifies computation model Device schematic diagram, wherein the specified distributed system be first kind distributed system, the specified computation model be branch It holds the second class distributed system but does not support the computation model of first kind distributed system；As shown in figure 3, should be in specified distribution The device 200 of the specified computation model of access includes: in system
Interface message preset unit 310 obtains the distribution of the second class suitable for obtaining the interface list of first kind distributed system The interface list of formula system establishes the mapping between the interface of first kind distributed system and the interface of the second class distributed system Relationship.
Call request converting unit 320, suitable for the interception specified computation model transmission about the second class distribution system The call request of the interface of system；According to reflecting between the interface of first kind distributed system and the interface of the second class distributed system Relationship is penetrated, the call request of the interface about the second class distributed system is converted into about first kind distributed system The call request of the corresponding interface.
Processing unit 330 is called, suitable for asking the calling of the corresponding interface about first kind distributed system after conversion It asks and is sent to first kind distributed system；Receive returning the result and being sent to the specified calculating mould for first kind distributed system Type returns the result carry out calculation processing as the specified computation model according to.
As it can be seen that device surface shown in Fig. 3 is to the second class distributed system of support but does not support first kind distributed system Specified computation model, by calling specified computation model the call request of interface of the second class distributed system to be converted to calling The call request of the equivalent interface of first kind distributed system realizes specified computation model to first kind distributed system The indirect calling of method, so that first kind distributed system and related data can calculate mould based on specified without making any change Type carries out relevant calculation processing.The program is suitable for those and wishes simultaneously again using first kind distributed system as data processing architecture Hope the user calculated using specified computation model, not changing former data processing architecture ensure that the stability and safety of data Property, and the specified computation model accessed meets corresponding process demand.
In one embodiment of the invention, it is wrapped in the call request of the above-mentioned interface about the second class distributed system It includes: interface name, interface parameters；Then call request converting unit 320, suitable for according to the interface of first kind distributed system and Mapping relations between the interface of two class distributed systems obtain in the call request about the interface of the second class distributed system The corresponding first kind distributed system of interface name interface name；According to the interface of acquired first kind distributed system The format of the parameter information is converted into adapting to the format of the interface of corresponding first kind distributed system by title；It obtains The call request of the corresponding interface about first kind distributed system includes: acquired first kind distribution in the call request The interface name of formula system, the parameter information after format conversion.
Wherein, call request converting unit 320 is further adapted for that the corresponding first kind of the interface name is having not been obtained When the interface name of distributed system, according to the interface name in the call request of the interface about the second class distributed system Title and parameter information, create a dynamic proxy object；According in the call request of the interface about the second class distributed system Interface name, found from the interface list of first kind distributed system with respective calls using the dynamic proxy object The interface of the first kind distributed system of effect；Using about the call request of the interface of the first kind distributed system as described in The call request of the corresponding interface about first kind distributed system.
In one embodiment of the invention, interface message preset unit 310 is suitable for by calling the first Context class Store the information of the interface list of first kind distributed system；By calling the 2nd Context class to store the second class distribution system The information of the interface list of system；The reflection from the 2nd Context class to the first Context class is realized using reflection mechanism, is established Mapping relations between the interface of first kind distributed system and the interface of the second class distributed system.
In one embodiment of the invention, interface message preset unit 310, suitable for storing first kind distributed system The corresponding first Jar packet of the information of interface list stores the information corresponding second of the interface list of the second class distributed system Jar packet；Wherein, the first Jar packet and the 2nd Jar packet are carried out by corresponding Classloader Classloader respectively Load.
Then call request converting unit 320, suitable for intercepting what the specified computation model was sent according to the 2nd Jar packet The call request of interface about the second class distributed system；Suitable for loading institute by corresponding Classloader Classloader The first Jar packet is stated, the call request of the corresponding interface about first kind distributed system is obtained.
In one embodiment of the invention, the call request of the interface about the second class distributed system includes: The call request of the interface of reading data about the second class distributed system, and/or, number is write about the second class distributed system According to interface call request；It wherein, include: second class distribution in the interface of the reading data of the second class distributed system The corresponding method for reading data of the interface of the reading data of formula system；The writing in the interface of data for second class distributed system is wrapped It includes: the corresponding method for writing data of the interface for writing data of the second class distributed system.
Correspondingly, the call request of described the corresponding interface about first kind distributed system includes: about the first kind The call request of the interface of the reading data of distributed system, and/or, the interface for writing data about first kind distributed system Call request；It wherein, include: the reading of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data of the interface of data；The first kind distributed system write in the interface of data include: this first The corresponding method for writing data of the interface for writing data of class distributed system.
In a specific embodiment, the first kind distributed system is the Hadoop system of first version, described Second class distributed system is the Hadoop system of the second edition, and the specified computation model includes following one or more: Yarn computation model, Spark computation model, Sqoop computation model.
It should be understood that each embodiment of Fig. 3 shown device and Fig. 1-each embodiment shown in Fig. 2 are corresponding identical, on Wen Zhongyi is described in detail, and details are not described herein.
In conclusion according to the technique and scheme of the present invention, in face of supporting the second class distributed system but not supporting first When the specified computation model of class distributed system, can by by specified computation model call the second class distributed system interface Call request be converted to the call request for calling the equivalent interface of first kind distributed system, realize specified computation model pair The indirect calling of the method for first kind distributed system, so that first kind distributed system and related data are without making any change Relevant calculation processing can be carried out based on specified computation model.The program is suitable for those using first kind distributed system as number According to processing framework again at the same time it is wished that the user calculated using specified computation model, not changing former data processing architecture ensure that The stability and safety of data, and the specified computation model accessed meets corresponding process demand.
It should be understood that
Algorithm and display be not inherently related to any certain computer, virtual bench or other equipment provided herein. Various fexible units can also be used together with teachings based herein.As described above, it constructs required by this kind of device Structure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use various Programming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hair Bright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the invention Example can be practiced without these specific details.In some instances, well known method, structure is not been shown in detail And technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects, Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimes In example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protect Shield the present invention claims features more more than feature expressly recited in each claim.More precisely, as following Claims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore, Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itself All as a separate embodiment of the present invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodiment Change and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodiment Member or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement or Sub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use any Combination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosed All process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint power Benefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purpose It replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodiments In included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the invention Within the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointed Meaning one of can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processors Software module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practice Microprocessor or digital signal processor (DSP) access to realize according to an embodiment of the present invention in specified distributed system The some or all functions of some or all components in the device of specified computation model.The present invention is also implemented as using In executing some or all device or device programs of method as described herein (for example, computer program and meter Calculation machine program product).It is such to realize that program of the invention can store on a computer-readable medium, or can have one The form of a or multiple signals.Such signal can be downloaded from an internet website to obtain, or mention on the carrier signal For, or be provided in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and ability Field technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims, Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of not Element or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple such Element.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer real It is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branch To embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fame Claim.
The invention discloses A1, a kind of method that specified computation model is accessed in specified distributed system, wherein described Specified distributed system is first kind distributed system, and the specified computation model is to support the second class distributed system but do not prop up Hold the computation model of first kind distributed system；
This method comprises:
The interface list of first kind distributed system is obtained, the interface list of the second class distributed system is obtained, establishes the Mapping relations between the interface of a kind of distributed system and the interface of the second class distributed system；
Intercept the call request for the interface about the second class distributed system that the specified computation model is sent；
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, by institute It states and is converted into the corresponding interface about first kind distributed system about the call request of the interface of the second class distributed system Call request, and it is sent to first kind distributed system；
Receive first kind distributed system returns the result and is sent to the specified computation model, by the specified calculating Model returns the result carry out calculation processing according to.
A2, method as described in a1, wherein wrapped in the call request of the interface about the second class distributed system It includes: interface name, interface parameters；
The mapping relations according between the interface of first kind distributed system and the interface of the second class distributed system, The call request of the interface about the second class distributed system is converted into accordingly connecing about first kind distributed system Mouthful call request include:
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, obtain The interface name of the corresponding first kind distributed system of the interface name；
According to the interface name of acquired first kind distributed system, the format of the parameter information is converted into being adapted to In the format of the interface of corresponding first kind distributed system；
The call request of the corresponding interface about first kind distributed system is obtained, includes: acquired in the call request First kind distributed system interface name, format conversion after parameter information.
A3, as described in A2 method, wherein the corresponding first kind distributed system of the interface name is having not been obtained When interface name, this method further comprises:
According to the interface name and parameter information in the call request of the interface about the second class distributed system, wound Build a dynamic proxy object；
According to the interface name, looked for from the interface list of first kind distributed system using the dynamic proxy object To the interface of the first kind distributed system with respective calls effect；
Using about the call request of the interface of the first kind distributed system as described about first kind distributed system The corresponding interface call request.
A4, method as described in a1, wherein the interface list for obtaining first kind distributed system obtains the second class The interface list of distributed system is established between the interface of first kind distributed system and the interface of the second class distributed system Mapping relations include:
By the information for calling the interface list of the first Context class storage first kind distributed system；
By the information for calling the interface list of the 2nd Context class the second class distributed system of storage；
The reflection from the 2nd Context class to the first Context class is realized using reflection mechanism, establishes first kind distribution Mapping relations between the interface of system and the interface of the second class distributed system.
A5, method as described in a1, wherein the interface list for obtaining first kind distributed system obtains the second class The interface list of distributed system includes:
The corresponding first Jar packet of information of the interface list of first kind distributed system is stored, the second class of storage is distributed The corresponding 2nd Jar packet of the information of the interface list of system；Wherein, the first Jar packet and the 2nd Jar packet pass through respectively Corresponding Classloader Classloader is loaded.
A6, method as described in a5, wherein
The call request packet for intercepting the interface about the second class distributed system that the specified computation model is sent It includes: intercepting the tune for the interface about the second class distributed system that the specified computation model is sent according to the 2nd Jar packet With request；
The call request by the interface about the second class distributed system is converted into about first kind distribution The call request of the corresponding interface of system includes: to load the first Jar packet by corresponding Classloader Classloader, Obtain the call request of the corresponding interface about first kind distributed system.
A7, method as described in a1, wherein
The call request of the interface about the second class distributed system includes: the reading about the second class distributed system The call request of the interface of data, and/or, the call request of the interface for writing data about the second class distributed system；
It include: the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data of interface；
Writing for the second class distributed system includes: that the second class distributed system writes data in the interface of data The corresponding method for writing data of interface.
A8, the method as described in A7, wherein
The call request of described the corresponding interface about first kind distributed system includes: about first kind distributed system Reading data interface call request, and/or, the call request of the interface for writing data about first kind distributed system；
It include: the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data of interface；
Writing for the first kind distributed system includes: that the first kind distributed system writes data in the interface of data The corresponding method for writing data of interface.
A9, method as described in a1, wherein
The first kind distributed system is the Hadoop system of first version, and the second class distributed system is second The Hadoop system of version, the specified computation model include following one or more: Yarn computation model, Spark calculate mould Type, Sqoop computation model.
The invention also discloses B10, a kind of device that specified computation model is accessed in specified distributed system, wherein The specified distributed system be first kind distributed system, the specified computation model be support the second class distributed system but The computation model of first kind distributed system is not supported；
The device includes:
It is distributed to obtain the second class suitable for obtaining the interface list of first kind distributed system for interface message preset unit The interface list of system, the mapping established between the interface of first kind distributed system and the interface of the second class distributed system are closed System；
Call request converting unit, suitable for intercepting that the specified computation model sends about the second class distributed system The call request of interface；It is closed according to the mapping between the interface of first kind distributed system and the interface of the second class distributed system System, the call request of the interface about the second class distributed system is converted into about the corresponding of first kind distributed system The call request of interface；
Processing unit is called, the call request suitable for the corresponding interface about first kind distributed system after converting is sent out It send to first kind distributed system；Receive first kind distributed system returns the result and is sent to the specified computation model, Carry out calculation processing is returned the result according to as the specified computation model.
B11, the device as described in B10, wherein wrapped in the call request of the interface about the second class distributed system It includes: interface name, interface parameters；
The call request converting unit, suitable for the interface and the second class distributed system according to first kind distributed system Interface between mapping relations, obtain the interface name of the corresponding first kind distributed system of the interface name；According to institute The format of the parameter information is converted into adapting to the corresponding first kind by the interface name of the first kind distributed system of acquisition The format of the interface of distributed system；The call request of the corresponding interface about first kind distributed system is obtained, which asks It include: the interface name of acquired first kind distributed system in asking, the parameter information after format conversion.
B12, device as described in b11, wherein
The call request converting unit is further adapted for that the corresponding first kind distribution of the interface name is having not been obtained When the interface name of formula system, according in the call request of the interface about the second class distributed system interface name and Parameter information creates a dynamic proxy object；According to the interface name, divided using the dynamic proxy object from the first kind The interface of the first kind distributed system with respective calls effect is found in the interface list of cloth system；Will about this first Call request of the call request of the interface of class distributed system as described the corresponding interface about first kind distributed system.
B13, the device as described in B10, wherein
The interface message preset unit, suitable for by calling the first Context class to store first kind distributed system The information of interface list；By the information for calling the interface list of the 2nd Context class the second class distributed system of storage；It utilizes Reflection mechanism realize established from the reflection of the 2nd Context class to the first Context class the interface of first kind distributed system with Mapping relations between the interface of second class distributed system.
B14, the device as described in B10, wherein
The interface message preset unit, suitable for storing the information of interface list of first kind distributed system corresponding the One Jar packet stores the corresponding 2nd Jar packet of information of the interface list of the second class distributed system；Wherein, the first Jar Packet and the 2nd Jar packet are loaded by corresponding Classloader Classloader respectively.
B15, the device as described in B14, wherein
The call request converting unit, suitable for intercepting what the specified computation model was sent according to the 2nd Jar packet The call request of interface about the second class distributed system；Suitable for loading institute by corresponding Classloader Classloader The first Jar packet is stated, the call request of the corresponding interface about first kind distributed system is obtained.
B16, the device as described in B10, wherein
The call request of the interface about the second class distributed system includes: the reading about the second class distributed system The call request of the interface of data, and/or, the call request of the interface for writing data about the second class distributed system；
It include: the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data of interface；
Writing for the second class distributed system includes: that the second class distributed system writes data in the interface of data The corresponding method for writing data of interface.
B17, the device as described in B16, wherein
The call request of described the corresponding interface about first kind distributed system includes: about first kind distributed system Reading data interface call request, and/or, the call request of the interface for writing data about first kind distributed system；
It include: the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data of interface；
Writing for the first kind distributed system includes: that the first kind distributed system writes data in the interface of data The corresponding method for writing data of interface.
B18, the device as described in B10, wherein
The first kind distributed system is the Hadoop system of first version, and the second class distributed system is second The Hadoop system of version, the specified computation model include following one or more: Yarn computation model, Spark calculate mould Type, Sqoop computation model.

Claims (18)

1. a kind of method for accessing specified computation model in specified distributed system, wherein the specified distributed system is First kind distributed system, the specified computation model are to support the second class distributed system but do not support first kind distribution system The computation model of system；
This method comprises:
The interface list of first kind distributed system is obtained, the interface list of the second class distributed system is obtained, establishes the first kind Mapping relations between the interface of distributed system and the interface of the second class distributed system；
Intercept the call request for the interface about the second class distributed system that the specified computation model is sent；
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, by the pass The calling of the corresponding interface about first kind distributed system is converted into the call request of the interface of the second class distributed system Request, and it is sent to first kind distributed system；
Receive first kind distributed system returns the result and is sent to the specified computation model, by the specified computation model Carry out calculation processing is returned the result according to described；Realize the specified computation model between the method for first kind distributed system Connect calling.
2. the method for claim 1, wherein being wrapped in the call request about the interface of the second class distributed system It includes: interface name, interface parameters；
The mapping relations according between the interface of first kind distributed system and the interface of the second class distributed system, by institute It states and is converted into the corresponding interface about first kind distributed system about the call request of the interface of the second class distributed system Call request includes:
According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, described in acquisition The interface name of the corresponding first kind distributed system of interface name；
According to the interface name of acquired first kind distributed system, it is converted into the format of the parameter information to adapt to phase The format of the interface for the first kind distributed system answered；
The call request for obtaining the corresponding interface about first kind distributed system includes: acquired the in the call request The interface name of a kind of distributed system, the parameter information after format conversion.
3. method according to claim 2, wherein the corresponding first kind distributed system of the interface name is having not been obtained Interface name when, this method further comprises:
According to the interface name and parameter information in the call request of the interface about the second class distributed system, creation one A dynamic proxy object；
According to the interface name, tool is found from the interface list of first kind distributed system using the dynamic proxy object There is the interface of the first kind distributed system of respective calls effect；
Using the phase about the call request of the interface of the first kind distributed system as described in about first kind distributed system Answer the call request of interface.
4. the method for claim 1, wherein interface list for obtaining first kind distributed system, obtains second The interface list of class distributed system is established between the interface of first kind distributed system and the interface of the second class distributed system Mapping relations include:
By the information for calling the interface list of the first Context class storage first kind distributed system；
By the information for calling the interface list of the 2nd Context class the second class distributed system of storage；
The reflection from the 2nd Context class to the first Context class is realized using reflection mechanism, establishes first kind distributed system Interface and the second class distributed system interface between mapping relations.
5. the method for claim 1, wherein interface list for obtaining first kind distributed system, obtains second The interface list of class distributed system includes:
The corresponding first Jar packet of information of the interface list of first kind distributed system is stored, the second class distributed system is stored Interface list the corresponding 2nd Jar packet of information；Wherein, the first Jar packet and the 2nd Jar packet are respectively by corresponding Classloader Classloader loaded.
6. method as claimed in claim 5, wherein
The call request for intercepting the interface about the second class distributed system that the specified computation model is sent includes: root It is asked according to the calling that the 2nd Jar packet intercepts the interface about the second class distributed system that the specified computation model is sent It asks；
The call request by the interface about the second class distributed system is converted into about first kind distributed system The call request of the corresponding interface include: that the first Jar packet, acquisition are loaded by corresponding Classloader Classloader The call request of the corresponding interface about first kind distributed system.
7. the method for claim 1, wherein
The call request of the interface about the second class distributed system includes: the reading data about the second class distributed system Interface call request, and/or, the call request of the interface for writing data about the second class distributed system；
It include: the interface of the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data；
The second class distributed system write include: in the interface of data the second class distributed system the interface for writing data The corresponding method for writing data.
8. the method for claim 7, wherein
The call request of described the corresponding interface about first kind distributed system includes: the reading about first kind distributed system The call request of the interface of data, and/or, the call request of the interface for writing data about first kind distributed system；
It include: the interface of the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data；
The first kind distributed system write include: in the interface of data the first kind distributed system the interface for writing data The corresponding method for writing data.
9. the method for claim 1, wherein
The first kind distributed system is the Hadoop system of first version, and the second class distributed system is the second edition Hadoop system, the specified computation model includes following one or more: Yarn computation model, Spark computation model, Sqoop computation model.
10. a kind of device for accessing specified computation model in specified distributed system, wherein the specified distributed system is First kind distributed system, the specified computation model are to support the second class distributed system but do not support first kind distribution system The computation model of system；
The device includes:
Interface message preset unit obtains the second class distributed system suitable for obtaining the interface list of first kind distributed system Interface list, establish the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system；
Call request converting unit, the interface about the second class distributed system sent suitable for intercepting the specified computation model Call request；According to the mapping relations between the interface of first kind distributed system and the interface of the second class distributed system, The call request of the interface about the second class distributed system is converted into accordingly connecing about first kind distributed system The call request of mouth；
Processing unit is called, suitable for the call request of the corresponding interface about first kind distributed system after conversion to be sent to First kind distributed system；Returning the result and being sent to the specified computation model for first kind distributed system is received, by institute It states specified computation model and returns the result carry out calculation processing according to；Realize the specified computation model to first kind distribution The indirect calling of the method for system.
11. device as claimed in claim 10, wherein in the call request of the interface about the second class distributed system It include: interface name, interface parameters；
The call request converting unit, suitable for according to the interface of first kind distributed system and connecing for the second class distributed system Mapping relations between mouthful, obtain the interface name of the corresponding first kind distributed system of the interface name；According to acquired First kind distributed system interface name, the format of the parameter information is converted into adapting to the corresponding first kind and is distributed The format of the interface of formula system；The call request for obtaining the corresponding interface about first kind distributed system, in the call request It include: the interface name of acquired first kind distributed system, the parameter information after format conversion.
12. device as claimed in claim 11, wherein
The call request converting unit is further adapted for that the interface name corresponding first kind distribution system is having not been obtained When the interface name of system, according to the interface name and parameter in the call request of the interface about the second class distributed system Information creates a dynamic proxy object；It is distributed from the first kind using the dynamic proxy object according to the interface name The interface of the first kind distributed system with respective calls effect is found in the interface list of system；It will be about the first kind point Call request of the call request of the interface of cloth system as described the corresponding interface about first kind distributed system.
13. device as claimed in claim 10, wherein
The interface message preset unit, suitable for by calling the first Context class to store the interface of first kind distributed system The information of list；By the information for calling the interface list of the 2nd Context class the second class distributed system of storage；Utilize reflection Mechanism realizes the reflection from the 2nd Context class to the first Context class, establishes the interface and second of first kind distributed system Mapping relations between the interface of class distributed system.
14. device as claimed in claim 10, wherein
The interface message preset unit, the information corresponding first of the interface list suitable for storing first kind distributed system Jar packet stores the corresponding 2nd Jar packet of information of the interface list of the second class distributed system；Wherein, the first Jar packet It is loaded respectively by corresponding Classloader Classloader with the 2nd Jar packet.
15. device as claimed in claim 14, wherein
The call request converting unit, suitable for according to the 2nd Jar packet intercept that the specified computation model sends about The call request of the interface of second class distributed system；Suitable for loading described the by corresponding Classloader Classloader One Jar packet obtains the call request of the corresponding interface about first kind distributed system.
16. device as claimed in claim 10, wherein
The call request of the interface about the second class distributed system includes: the reading data about the second class distributed system Interface call request, and/or, the call request of the interface for writing data about the second class distributed system；
It include: the interface of the reading data of the second class distributed system in the interface of the reading data of the second class distributed system The corresponding method for reading data；
The second class distributed system write include: in the interface of data the second class distributed system the interface for writing data The corresponding method for writing data.
17. device as claimed in claim 16, wherein
The call request of described the corresponding interface about first kind distributed system includes: the reading about first kind distributed system The call request of the interface of data, and/or, the call request of the interface for writing data about first kind distributed system；
It include: the interface of the reading data of the first kind distributed system in the interface of the reading data of the first kind distributed system The corresponding method for reading data；
The first kind distributed system write include: in the interface of data the first kind distributed system the interface for writing data The corresponding method for writing data.
18. device as claimed in claim 10, wherein
The first kind distributed system is the Hadoop system of first version, and the second class distributed system is the second edition Hadoop system, the specified computation model includes following one or more: Yarn computation model, Spark computation model, Sqoop computation model.
CN201610192917.5A 2016-03-30 2016-03-30 A kind of method and apparatus accessing specified computation model in specified distributed system Active CN105867928B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610192917.5A CN105867928B (en) 2016-03-30 2016-03-30 A kind of method and apparatus accessing specified computation model in specified distributed system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610192917.5A CN105867928B (en) 2016-03-30 2016-03-30 A kind of method and apparatus accessing specified computation model in specified distributed system

Publications (2)

Publication Number Publication Date
CN105867928A CN105867928A (en) 2016-08-17
CN105867928B true CN105867928B (en) 2019-06-04

Family

ID=56626818

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610192917.5A Active CN105867928B (en) 2016-03-30 2016-03-30 A kind of method and apparatus accessing specified computation model in specified distributed system

Country Status (1)

Country Link
CN (1) CN105867928B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111522787B (en) * 2019-02-01 2023-04-07 阿里巴巴集团控股有限公司 Data processing method and device of distributed system and storage medium
CN111767092B (en) * 2020-06-30 2023-05-12 深圳前海微众银行股份有限公司 Job execution method, apparatus, system and computer readable storage medium
CN112104706B (en) * 2020-08-24 2022-12-20 中国银联股份有限公司 Method, device, equipment and storage medium for releasing model in distributed system
CN114116153A (en) * 2020-08-31 2022-03-01 华为技术有限公司 Proxy service-based device calling method, electronic device and readable storage medium
CN112445543B (en) * 2020-11-26 2023-03-10 杭州趣链科技有限公司 Class calling method and device of intelligent contract and electronic equipment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101216760A (en) * 2007-12-28 2008-07-09 北京方正国际软件系统有限公司 Dynamic mapping interface calling system and method
CN101277301A (en) * 2008-04-24 2008-10-01 华为技术有限公司 Method and device for invocation of distributed system interface
CN101339559A (en) * 2008-07-18 2009-01-07 北京航空航天大学 Long lasting implementing method for data
CN100578458C (en) * 2007-03-22 2010-01-06 华为技术有限公司 Call request processing method in distributed system, distributed system and server
US8495664B2 (en) * 2005-07-06 2013-07-23 International Business Machines Corporation System, method and program product for invoking a remote method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8495664B2 (en) * 2005-07-06 2013-07-23 International Business Machines Corporation System, method and program product for invoking a remote method
CN100578458C (en) * 2007-03-22 2010-01-06 华为技术有限公司 Call request processing method in distributed system, distributed system and server
CN101216760A (en) * 2007-12-28 2008-07-09 北京方正国际软件系统有限公司 Dynamic mapping interface calling system and method
CN101277301A (en) * 2008-04-24 2008-10-01 华为技术有限公司 Method and device for invocation of distributed system interface
CN101339559A (en) * 2008-07-18 2009-01-07 北京航空航天大学 Long lasting implementing method for data

Also Published As

Publication number Publication date
CN105867928A (en) 2016-08-17

Similar Documents

Publication Publication Date Title
CN105867928B (en) A kind of method and apparatus accessing specified computation model in specified distributed system
WO2021051914A1 (en) Gpu resource-based data processing method and system, and electronic device
US20140215446A1 (en) Automated Porting of Application to Mobile Infrastructures
US9525592B2 (en) Client/server network environment setup method and system
CN103942225B (en) A kind of resource transfer method, client and the system of mixed type applications client
US20220188170A1 (en) Multi-Tenant Control Plane Management on Computing Platform
US9311161B2 (en) Automatically configured management service payloads for cloud IT services delivery
US20150006608A1 (en) Networked solutions integration using a cloud business object broker
CN107395762A (en) A kind of application service based on Docker containers accesses system and method
CN104461567B (en) The dissemination method and system of software package
CN107911470A (en) Distributed dynamic load-balancing method, computing device and computer-readable storage medium
US20130346542A1 (en) Common web accessible data store for client side page processing
EP2791822A1 (en) Lightweight framework for web applications
US20160359954A1 (en) Web services documentation
CN110427225A (en) A kind of method and apparatus starting small routine
CN108881111A (en) A kind of method and device for realizing multi-tenant system
WO2013163237A1 (en) Providing client and service compatibility through cloud-hosted adapters
CN111818194A (en) Domain name based access system and method
US10552591B2 (en) Resource optimization using data isolation to provide sand box capability
US20190114081A1 (en) Scale-out container volume service for multiple frameworks
CN104881294B (en) Realize the system and method for automatic packaging
US9893936B2 (en) Dynamic management of restful endpoints
CN108023905A (en) Internet of things application system and method
CN106156308B (en) Local updating method and system for webpage
US8214389B2 (en) Common architecture for administration of client side property settings in a distributed and heterogeneous environment

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20220801

Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015

Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd.

Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park)

Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd.

Patentee before: Qizhi software (Beijing) Co.,Ltd.

TR01 Transfer of patent right