CN110197064B

CN110197064B - Process processing method and device, storage medium and electronic device

Info

Publication number: CN110197064B
Application number: CN201910118910.2A
Authority: CN
Inventors: 罗伟; 龚高晟; 邹建平
Original assignee: Tencent Technology Shenzhen Co Ltd; Tencent Cloud Computing Beijing Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd; Tencent Cloud Computing Beijing Co Ltd
Priority date: 2019-02-18
Filing date: 2019-02-18
Publication date: 2023-08-25
Anticipated expiration: 2039-02-18
Also published as: CN110197064A

Abstract

The invention discloses a process processing method and device, a storage medium and an electronic device. Wherein the method comprises the following steps: acquiring a process processing request, wherein the process processing request carries a target process task to be processed; responding to the process processing request, and loading a file data packet matched with the target process task; under the condition that the file data packet comprises a target program file data packet corresponding to a user-defined function, adding a first target indication parameter in a process starting instruction for starting a target process, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, and the sandbox is configured with target process processing permission of the target process; and starting a target process provided with the sandboxes according to the process starting instruction, and processing target process tasks through the target process. The method and the device solve the technical problem of low process processing efficiency in the public cloud service platform caused by the fact that manual auditing occupies a large amount of processing time in the prior art.

Description

Process processing method and device, storage medium and electronic device

Technical Field

The present invention relates to the field of computers, and in particular, to a process processing method and apparatus, a storage medium, and an electronic apparatus.

Background

When a user performs big data analysis based on public cloud service, a user-defined function (User Defined Function, abbreviated as UDF) is often required to be called to complete a complex analysis function. The UDF function needs to be packaged to generate a jar application program file package and then uploaded to the public cloud service platform so that a resource manager in the public cloud service platform can be used in the running process.

At present, in order to avoid threat to security and stability of public cloud service platforms after a jar application program file package of a UDF function is uploaded, security audit is usually required to be carried out on user codes in the jar application program file package by professional staff. However, as the number of users of the public cloud service platform increases, the manual auditing method occupies a large amount of processing time, so that the processing efficiency of the process in the public cloud service platform is low.

In view of the above problems, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the invention provides a process processing method and device, a storage medium and an electronic device, which at least solve the technical problem of low process processing efficiency in a public cloud service platform caused by the fact that manual auditing occupies a large amount of processing time in the prior art.

According to an aspect of the embodiment of the present invention, there is provided a process processing method, including: acquiring a process processing request, wherein the process processing request carries a target process task to be processed; responding to the process processing request, and loading a file data packet matched with the target process task; under the condition that the file data packet comprises a target program file data packet corresponding to a user-defined function, adding a first target indication parameter into a process starting instruction for starting a target process, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, the sandbox is configured with target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions; and starting the target process provided with the sandbox according to the process starting instruction, and processing the target process task through the target process.

According to another aspect of the embodiment of the present invention, there is also provided a process processing apparatus, including: the first acquisition unit is used for acquiring a process processing request, wherein the process processing request carries a target process task to be processed; the loading unit is used for responding to the process processing request and loading a file data packet matched with the target process task; the first adding unit is configured to add a first target indication parameter to a process starting instruction for starting a target process if the file data packet includes a target program file data packet corresponding to a user-defined function, where the first target indication parameter is used to indicate to start a sandbox set in the target process, and the sandbox is configured with target process processing permissions of the target process, and the target process processing permissions are smaller than all process processing permissions; and the processing unit is used for starting the target process provided with the sandbox according to the process starting instruction and processing the target process task through the target process.

According to still another aspect of the embodiments of the present invention, there is also provided a storage medium having stored therein a computer program, wherein the computer program is configured to execute the above-described process processing method when run.

According to still another aspect of the embodiments of the present invention, there is further provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above-mentioned process processing method through the computer program.

In the embodiment of the invention, after a process processing request is acquired, a file data packet matched with a target process task required to be processed is loaded in response to the process processing request, and if the file data packet comprises a target program file data packet corresponding to a user-defined function, a first target indication parameter for starting a sandbox arranged in the target process is added in a process starting instruction for starting the target process, so that the sandbox with the target process processing permission arranged in the target process is automatically started while the target process is started. The target process can safely run within the target process processing authority range smaller than all process processing authorities without the need of manually checking the security of codes, and the effects of saving process processing time and improving process processing efficiency are achieved. And further, the technical problem of low process processing efficiency in the public cloud service platform caused by the fact that a large amount of processing time is occupied by manual auditing in the prior art is solved. Furthermore, the sandboxes are uniformly arranged in the process, so that the problem of security holes caused by different manual auditing standards is avoided, and the safety and stability of process treatment are improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

FIG. 1 is a schematic diagram of a hardware environment of an alternative process handling method according to an embodiment of the application;

FIG. 2 is a flow diagram of an alternative process method according to an embodiment of the application;

FIG. 3 is a schematic diagram of a network architecture of an alternative process handling method according to an embodiment of the application;

FIG. 4 is a schematic diagram of an alternative process treatment method according to an embodiment of the application;

FIG. 5 is a flow diagram of another alternative process method according to an embodiment of the application;

FIG. 6 is a schematic diagram of another alternative process treatment method according to an embodiment of the application;

FIG. 7 is a flow diagram of yet another alternative process method according to an embodiment of the application;

FIG. 8 is an interface diagram of an alternative process method according to an embodiment of the application;

FIG. 9 is an interface diagram of another alternative process treatment method according to an embodiment of the application;

FIG. 10 is a schematic diagram of an alternative process processing arrangement according to an embodiment of the invention;

fig. 11 is a schematic structural view of an alternative electronic device according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of the embodiment of the present invention, a process processing method is provided, and optionally, as an alternative implementation manner, the process processing method may be applied, but not limited to, in a hardware environment as shown in fig. 1. The user 102 uses the user device 104 to upload the file data packet matched with the target process task to the cloud service platform 112 through the network 110, as in step S102, for use by the resource coordination cluster 116 in the cloud service platform 112 during running. Wherein the user device 102 controls the uploading of the file data packets stored in the memory 106 via the processor 108. After the process processing request is acquired, the resource coordination cluster 116 in the cloud service platform 112 executes steps S104-S108, according to the acquired process processing request, a file data packet matched with a target process task of the requested process is loaded from the database 114, and if it is determined in the loading process that the file data packet includes a target program file data packet corresponding to a user-defined function (User Defined Function, abbreviated as UDF), a first target indication parameter is added in a start instruction for starting a target process, where the first target indication parameter is used for indicating a sandbox set in starting the target process, and the sandbox is configured with a target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions. Finally, step S110 is executed, where the target process with the sandbox is started according to the process start instruction, and the target process task is processed by the target process.

It should be noted that, in the process processing method provided in this embodiment, after a process processing request is obtained, a file data packet matched with a target process task of a requested process is loaded in response to the process processing request, and if the file data packet includes a target program file data packet corresponding to a user-defined function, a first target instruction parameter for starting a sandbox set in the target process is added in a process starting instruction for starting the target process, so that the sandbox with the target process processing permission set in the target process is automatically started while the target process is started. The target process can safely run within the target process processing authority range smaller than all process processing authorities without the need of manually checking the security of codes, and the effects of saving process processing time and improving process processing efficiency are achieved. Furthermore, the sandboxes are uniformly arranged in the process, so that the problem of security holes caused by different manual auditing standards is avoided, and the safety and stability of process treatment are improved.

Optionally, the above process processing method may be applied to, but not limited to, a user equipment, and the user equipment 104 may include, but not limited to, a terminal device such as a mobile phone, a tablet computer, a notebook computer, a PC, etc. for uploading a file data packet of a user-defined function; the network 110 may include, but is not limited to, a wireless network or a wired network. Wherein the wireless network comprises: bluetooth, WIFI, and other networks that enable wireless communications. The wired network may include, but is not limited to: wide area network, metropolitan area network, local area network. The cloud service platform 112 may be, but is not limited to, a public cloud service platform, and a lightweight big data analysis, processing and display platform with a visual development interface and task scheduling as cores. The public cloud service platform can be a platform based on a Hadoop open source software framework, can include, but is not limited to, a data warehouse tool Hive, is used for mapping a structured data file into a database table, provides a simple structured language query (Structure Query Language, SQL for short) function, and can convert SQL sentences into MapReduce task operation; further, it may include, but is not limited to: a resource coordination cluster (Yet Another Resource Negotiator, YARN for short) is a universal resource management system for providing uniform resource management and scheduling for upper layer applications.

Optionally, as an optional embodiment, as shown in fig. 2, the above process processing method includes:

s202, acquiring a process processing request, wherein the process processing request carries a target process task to be processed;

s204, responding to the process processing request, and loading a file data packet matched with the target process task;

s206, under the condition that a file data packet comprises a target program file data packet corresponding to a user-defined function, adding a first target indication parameter into a process starting instruction for starting a target process, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, the sandbox is configured with target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions;

s208, starting a target process provided with a sandbox according to the process starting instruction, and processing a target process task through the target process.

Optionally, in this embodiment, the above-mentioned process processing method may be, but not limited to, applied to a scenario in which a user-defined function UDF is invoked to perform process processing when a public cloud big data service platform (hereinafter may be simply referred to as a cloud service platform) based on multiple users performs data analysis. That is, a sandbox environment is set for the process in the cloud service platform, so that the process processing process is automatically isolated and limited by the process processing authority corresponding to the sandbox, thereby ensuring that the target process to be processed can automatically run within the safety authority range, saving the event of safety audit and achieving the purpose of improving the process processing efficiency.

It should be noted that, the scheme provided in this embodiment may support that a sandbox is embedded in a process to operate, so that after an acquired process processing request is determined, a first target indication parameter for starting a sandbox set in a target process is automatically added under the condition that a target program file data packet corresponding to a UDF function is included in a file data packet loaded, and the sandbox with target process processing permission set in the target process is automatically started while the target process is started. The target process can safely run within the target process processing authority range smaller than all process processing authorities without the need of manually checking the security of codes, and the effects of saving process processing time and improving process processing efficiency are achieved. Furthermore, the sandboxes are uniformly arranged in the process, so that the problem of security holes caused by different manual auditing standards is avoided, and the safety and stability of process treatment are improved.

Alternatively, in the present embodiment, the above-described process processing method may be applied, but is not limited to, in the cloud service platform 302 shown in fig. 3. The cloud service platform 302 may include, but is not limited to, the following structures: hive, hadoop, YARN clusters. The Hadoop is an open source software framework supporting data intensive distributed application programs and issued by Apache2.0 license protocol, and supports application programs running on large clusters constructed by commodity hardware. The Hive is a data warehouse tool based on a Hadoop open source software framework, and is used for mapping a structured data file into a database table and providing a simple SQL query function. The YARN cluster is a universal resource management system for providing uniform resource management and scheduling for upper layer applications. The above process processing method may be, but not limited to, applied to a yan cluster, for example, after Hive parses SQL query statements to obtain some columns of MapReduce process tasks (MR tasks for short), and then the MR tasks are submitted to the yan cluster to run, so that the yan cluster loads file data packets required by the yan tasks and processes the MR tasks safely and efficiently in a process with sandboxes.

Optionally, in this embodiment, before the process processing request is acquired, parameters are configured for the structured query language in the cloud service platform, so that the structured query language is parsed into the process tasks for processing in the resource manager cluster.

It should be noted that, for some simple query process tasks, the cloud service platform may directly read the stored data (such as the data in Hadoop) through the cloud server (such as HiveServer 2), so as to improve the query efficiency, without running on the yan cluster of the cloud service platform each time. However, the solution provided in this embodiment is to make the UDF function run in a sandbox set by the yann cluster. Thus, in this embodiment, the cloud service platform may optimize the SQL procedure tasks, such as performing parameter configuration for SQL, to ensure that all UDF functions in the cloud service platform can run in the sandboxes set by the yann cluster. Alternatively, in this embodiment, the YARN cluster may, but is not limited to, include a node manager (NodeManager) for caching all file data packets required for processing the process task, and in this embodiment, the sandbox may, but is not limited to, be provided in the node manager to enable efficient combination with the YARN cluster. The above is merely an example, and is not limited in any way in the present embodiment.

Optionally, in this embodiment, in the process of loading the file data packet matched with the target process task, matching may include, but is not limited to, matching the packet identifier of the file data packet. Wherein, the liquid crystal display device comprises a liquid crystal display device,

1) And under the condition that the current file data packet is determined to be the target program file data packet corresponding to the user-defined function according to the matching result of the data packet identification, storing the target program file data packet under the first target path.

It should be noted that the authority of the sandbox is determined according to the content configured in the sandbox policy file. The sandbox strategy file is used for configuring a target program file data packet corresponding to the user-defined function and corresponding operation process processing permission thereof. Further, in the sandboxed policy file, it is necessary to minimize the path of the target program file packet, so as to limit the authority of the target program file packet of the UDF function, and reduce the influence on other system components. The object program file package may be, but not limited to, a Java archive file (referred to as jar package) of a UDF function, which is not only used for compression and release, but also used for deploying and encapsulating libraries, components and plug-ins, and may be directly used by tools such as an editor and a Java virtual machine (referred to as JVM).

In this embodiment, the first target path may include, but is not limited to, a storage path of the target program file data packet in the local device, where the first target path may include, but is not limited to: root path, user field, sub path. The user field in the first target path for indicating the uploading target program file data packet is a variable so as to present the user information of different uploading file data packets.

Optionally, in this embodiment, a target monitoring variable for monitoring whether the target program file packet is included in the file packet is provided. Under the condition that the target monitoring variable is adjusted to the target value, determining that a target program file data packet appears in the loaded file data packet, and adding a first target indication parameter into a process starting instruction for starting a target process.

It should be noted that, in this embodiment, the first target indication parameter may include, but is not limited to, a Java security management parameter, such as-djva. That is, after the above parameters are added to the process start instruction, the system will determine to automatically start the sandbox (e.g. Java sandbox) set in the process according to the identified-djva. The above is merely an example, and is not limited in any way in the present embodiment.

2) And under the condition that the current file data packet is not the target program file data packet corresponding to the user-defined function according to the matching result of the data packet identification, storing the current file data packet under a second target path.

It should be noted that the second target path may include, but is not limited to, a storage path that is local to a file packet other than the target program file packet (e.g., other jar packets that are not UDF functions). Wherein the second target path may include, but is not limited to, a different storage path than the first target path. For example, both may have different sub-paths. For another example, both may be stored at different path levels, e.g., the second target path may be a default sub-path.

Optionally, in this embodiment, a target process provided with a sandbox is started according to a process start instruction, and a target process task is processed by the target process. That is, in this embodiment, after optimizing the logic for starting the process, it may be determined whether to start the sandbox set in the process when starting the process, so as to reduce the security audit time and improve the process processing efficiency. Under the condition that the Java security management parameters are added in the process starting instruction, when a target process is started, automatically starting a sandbox arranged in the process according to the identified Java security management parameters, otherwise, not starting the sandbox, further processing the task of the target process within the authority range limited by the sandbox so as to ensure the security and stability of the task processing process,

Optionally, in this embodiment, before the process processing request is acquired, the cloud service platform acquires a target program file data packet corresponding to the user-defined function and uploaded by the target user account; and generating an access permission record for the target program file data packet, wherein the access permission record is used for indicating a user account belonging to the same account set as the target user account and has permission to access the target program file data packet. In order to receive the authority verification request sent by the resource manager cluster in the process of loading the file data packet matched with the target process task, and carry out authority verification by utilizing the access authority record. Therefore, when the permission verification request indicates that verification is passed, the file data packet matched with the target process task is sent to the resource manager cluster.

It should be noted that the above access right record may be, but not limited to, a record generated by an open source security management framework Ranger. That is, after the target user account uploads the jar packet, the cloud service platform may generate an access permission record (also referred to as a range record) for the target user account to record the read-write permission of the jar packet, for example, the target user account is configured with the read-write permission, and other user accounts belonging to the same engineering with the target user account are configured with the read-write permission and/or the writable permission. Therefore, before loading the jar packet, an authority verification request needs to be initiated to an open source security management framework Ranger, and under the condition that verification passes and access is confirmed to be allowed, a resource coordinator (such as a YARN cluster) is authorized to load the jar packet.

Specifically, the description will be given with reference to fig. 4. The user uploads the file data package to a cloud service platform (indicated by Hadoop418 in fig. 4) constructed based on Hadoop through a client logged in by using a target user account, wherein the file data package includes a target program file data package (indicated by UDF jar package 416 in fig. 4) corresponding to a UDF function. In addition, a node manager is included in the YARN cluster in the cloud service platform, such as node manager 402 shown in FIG. 4. Assume that a Java virtual machine (hereinafter referred to as JVM) 404 is running in the node manager 402, and is configured to control a process Container 408 provided with a Java sandbox 406 according to a configuration in a sandbox policy file 410, and perform a process on an operating system resource 412 to complete a process task.

Specifically, after the process processing request is acquired, the node manager 402 traverses the file data packet loaded from the Hadoop418 and matched with the target process task to be processed, and sequentially performs identification comparison on the data packet identification of the file data packet. In the case that the packet identifier of the current file packet matches the key identifier for indicating the UDF jar packet, the current file packet (e.g., UDF jar packet 416) is stored in the first target path in the cache 414, and the target monitoring variable is adjusted to the target value, otherwise the current file packet is stored in the second target path in the cache 414.

Further, after the node manager 402 completes the traversal, if the target monitored variable is the target value, it determines that the UDF jar packet 416 is included in the loaded file packet. In this case, a first target indication parameter (e.g. -dqava security. Manager) for opening a sandbox set in the target process is added to the process start instruction. Further, the JVM 404 will open the Java sandbox 406 according to the-Djava security. During operation, java sandbox 406 will invoke corresponding UDF jar package 416 according to the configuration in sandbox policy file 410 to perform the corresponding process on operating system resource 412 to complete the process task of the requested process.

According to the embodiment of the application, after the process processing request is acquired, the file data packet matched with the target process task required to be processed is loaded in response to the process processing request, and when the file data packet comprises the target program file data packet corresponding to the user-defined function, a first target indication parameter for starting the sandbox arranged in the target process is added in a process starting instruction for starting the target process, so that the sandbox with the target process processing permission arranged in the target process is automatically started while the target process is started. The target process can safely run within the target process processing authority range smaller than all process processing authorities without the need of manually checking the security of codes, and the effects of saving process processing time and improving process processing efficiency are achieved. Furthermore, the sandboxes are uniformly arranged in the process, so that the problem of security holes caused by different manual auditing standards is avoided, and the safety and stability of process treatment are improved.

As an alternative, loading the file data packet matched with the target process task includes:

s1, repeatedly executing the following steps until all file data packets matched with the target process task are traversed:

s12, acquiring a data packet identifier of a current file data packet loaded currently;

s14, comparing the data packet identifier with a key identifier, wherein the key identifier is used for identifying the target program file data packet;

s16, under the condition that the data packet identifier is matched with the key identifier, determining that the current file data packet is a target program file data packet, and storing the target program file data packet under a first target path;

s18, under the condition that the data packet identifier is not matched with the key identifier, determining that the current file data packet is not the target program file data packet, and storing the current file data packet under a second target path;

s20, loading the next file data packet as the current file data packet.

Optionally, in this embodiment, the packet identifier may include, but is not limited to: the file name of the file data packet, the ID of the file data packet, etc. are used to uniquely distinguish the identification of the file data packet. The above is merely an example, and is not limited in any way in the present embodiment.

In order to minimize the limitation of the path of the object file packet, the authority of the object file packet corresponding to the UDF function is limited, and the influence on other system components is reduced. In this embodiment, the first target path may be configured locally for a target program file data packet (e.g., a jar packet of a UDF function), and the second target path may be configured locally for other file data packets (e.g., jar packets of a non-UDF function) other than the target program file data packet. Wherein, the first target path may include, but is not limited to: root path, user field, sub path. For example, the first target path may be:

${yarn.nodemanager.local-dirs}/usercache/${user}/filecache/

where $ { yarn.notify-dirs } is the root path configured in the cache and $ { user } is the user field used to indicate the user information to upload the file packet. The $ { user } is a variable to cover all target program file packages uploaded by different users. Fileche/is a sub-path configured in the cache.

For example, the second target path may be:

${yarn.nodemanager.local-dirs}/usercache/${user}/

that is, the second target path may include, but is not limited to, a different storage path than the first target path. For example, both may have different sub-paths. For another example, both may be stored at different path levels, e.g., the second target path may be a default sub-path.

Optionally, in this embodiment, after determining that the current file data packet is the target program file data packet, the method further includes: and adjusting a target monitoring variable for monitoring a target program file data packet included in the file data packet to a target value, wherein the target value is used for indicating that the target program file data packet appears in the loaded file data packet.

In the process of processing the target process task in the process, all file data packets required by the target process task are traversed, and when at least one target program file data packet appears, the target monitoring variable can be adjusted to the target value. Wherein the values of the target monitoring variables can be identified by, but not limited to, using contracted characters. The contracted characters may be, but are not limited to, any one of letters, numbers, symbols, etc., or a combination thereof. For example, the variable "has UDF" is used to identify the target monitored variable, and Boolean characters are used to identify the value of the target monitored variable. It may be assumed that the target program file data packet has not been loaded yet, denoted by "false", and that the target program file data packet has been loaded, i.e. as target value, is identified by "true". The above is merely an example, and is not limited in any way in the present embodiment.

Specifically, the description will be given with reference to fig. 5. The above hypothetical scenario is still described as an example:

after acquiring the process processing request, the node manager 402 in the yan cluster will execute step S500, traverse all the file data packets recorded that match the target process task to be processed, set the value of the initialization target monitoring variable has UDF to "false", and then as shown in step S502: judging whether all file data packets are processed or not; under the condition that all file data packets are not processed, executing steps S504-S506, loading new file data packets from Hadoop, and judging whether the new file data packets are UDF jar packets or not; if it is determined that the file packet is not a UDF jar packet, storing the file packet in a second target path, and returning to step S502; if it is determined that the file packet is a UDF jar packet, step S508 is executed to store the file packet in the first target path, adjust the value of the target monitoring variable has UDF to the target value "true", and return to step S502. If it is determined that all the file packets have been processed, step S510 is executed to call the stored UDF jar packet, and the target process task is processed in the target process provided with the sandbox.

According to the embodiment of the application, the object program file data packets corresponding to the user-defined function are accurately distinguished and cached by traversing all the file data packets required by processing the object process task. The method and the system can not only realize the minimum limitation of the path of the target program file data packet uploaded by the user and reduce the influence on other components of the system, but also be beneficial to accurately finding the storage positions of the target program file data packets of different users by the sandbox, and inquiring the corresponding operation process processing permission in the sandbox strategy file, thereby achieving the safe processing of the target process task and avoiding the threat to the safety and stability of the platform.

As an alternative, when adding the first target indication parameter to the process starting instruction for starting the target process, the method further includes:

s1, adding a second target indication parameter into a process starting instruction, wherein the second target indication parameter is used for indicating a file storage path where a sandbox strategy file matched with a sandbox is located, the sandbox strategy file is used for storing a corresponding relation between a target program file data packet and an operation process processing permission under the first target path, and the target process processing permission comprises the operation process processing permission.

Optionally, in this embodiment, the second target indication parameter may include, but is not limited to, a Java security policy parameter, for example, -djva. Security. Policy, for indicating a file storage path where a sandbox policy file in which a target process processing permission corresponding to a sandbox is recorded is located. That is, after the second target indication parameter is added to the process start instruction, the system determines a file storage path of the sandbox policy file according to the identified-djva. In addition, the sandboxed policy file may further include, but is not limited to: and recording the corresponding relation between other file data packets under the second target path and the corresponding operation process processing permission. For example, to ensure the security and stability of the cloud service platform, all process processing permissions may be configured for the file data packet of the second target path, and the target process processing permissions configured for the target program file data packet of the first target path may be one of the following: partial process processing permission and no process processing permission.

It should be noted that, the sandbox policy file matched with the sandbox corresponds to a process authority white list, which can be used for limiting the authority of codes under each path in the process after the sandbox is opened, and the codes under each path can be operated and run under the explicitly allowed operation process processing authority, otherwise, the codes under each path can be reported to be wrong.

Specifically, in conjunction with the description shown in fig. 6, in the sandbox policy file, an operation process processing authority a may be configured, but not limited to, for a UDF jar packet, and an operation process processing authority B may be configured for other jar packets, where a < B. That is, when the process Container is started, if the process starting instruction carries a first target indication parameter-djuva. Security. Manager and a second target indication parameter-djuva. Security. Policy, starting a Java sandbox in the process Container and finding a corresponding sandbox policy file when the process Container is started, and respectively calling a corresponding file data packet to process the operating system resource according to an operation process processing permission A configured for a target program file data packet (such as a UDF jar packet) and an operation process processing permission B configured for other file data packets (such as a non-UDF jar packet) to complete a target process task. Thereby realizing the processing of the target process task within the authorized security authority range.

According to the embodiment of the application, the second target indication parameter for indicating the file storage path where the sandbox strategy file matched with the sandbox is located is added in the process starting instruction to acquire the corresponding relation between the target program file data packet under the first target path and the operation process processing permission, so that the process task is automatically processed within the limited security permission range corresponding to the sandbox according to the configuration, the process security is ensured, a large amount of time occupied by manual auditing is avoided, and the process processing efficiency is improved. In addition, the unified sandbox configuration can avoid security holes caused by different manual auditing standards, and further ensures the safety and stability of platform operation.

As an alternative solution, starting a target process provided with a sandbox according to a process starting instruction, and processing a target process task through the target process includes:

s1, analyzing a first target indication parameter and a second target indication parameter carried in a process starting instruction;

s2, when the target process is started, starting a sandbox set in the target process according to the first target indication parameter, and acquiring a sandbox strategy file according to the second target indication parameter;

S3, calling a target program file data packet under the first target path according to the sandbox strategy file;

s4, processing the target process task by utilizing the target program file data packet within the authorized range of the operation process processing authority corresponding to the target program file data packet.

Optionally, in this embodiment, in the process of processing a target process task in a process, traversing all file data packets required by the target process task, and when at least one target program file data packet (such as a UDF jar packet) appears, a condition of adding a first target indication parameter and a second target indication parameter in a process start instruction is satisfied; and when the target program file data packet (such as a UDF jar packet) does not appear in all the file data packets, the condition of adding the first target indication parameter and the second target indication parameter in the process starting instruction is not satisfied.

That is, when the first target instruction parameter and the second target instruction parameter are parsed in the process start instruction, it indicates that at least one target program file packet (for example, UDF jar packet) appears in all the file packets described. After the sandbox is opened, the target process may call the corresponding file data packet to process the operating system resource according to the configuration of the sandbox policy file, and according to the operation process processing authority a configured for the target program file data packet (e.g., UDF jar packet) and the operation process processing authority B configured for other file data packets (e.g., non-UDF jar packet), respectively, so as to complete the target process task.

Specifically, the step S510 in the dashed box shown in fig. 7 is described. The above hypothetical scenario is still described as an example:

after the buffer setting for all the file packets is completed as in steps S500-S508, at the time of starting the process Container, as in step S702, it is determined whether the value of the target monitoring variable has UDF is the target value "true"? If it is determined that the value of the target monitoring variable has UDF is the target value "true", step S704 is executed, where the first target indication parameter-djva security. And then executing steps S706-S708, when the process Container is started, analyzing a first target indication parameter-Djava. Security. Manager and a second target indication parameter-Djava. Security. Policy from the process starting instruction, starting a sandbox arranged in the process Container according to the indication of the first target indication parameter-Djava. Security. Manager, searching a sandbox policy file in a corresponding file storage path according to the indication of the second target indication parameter-Djava. Security. Policy, and processing a target process task according to the operation process processing permission configured therein. In step S702, when it is determined that the value of the target monitoring variable has UDF is not the target value "true", but is "false", the target process is directly started, but the sandbox is not opened in the started target process, and the target process task is processed within the range of the original configuration authority in the target process.

It should be noted that the order of steps shown in the above figures is an example, and in other embodiments, the order of execution of some steps may be adjusted. The step labels of step S704 and step S706 are sequence labels under one execution logic branch (for example, determining that the value of the target monitoring variable has UDF is the target value "true"). In another execution logic branch (if it is determined that the value of the target monitoring variable has UDF is not the target value "true"), steps S706-S708 will be directly executed by default step S704. This is not limited in this embodiment.

According to the embodiment of the application, after the first target indication parameter and the second target indication parameter carried in the process starting instruction are analyzed, the sandbox arranged in the target process is started, and the sandbox strategy file recorded with the target process processing permission of the sandbox is obtained, so that the target program file data packet under the first target path is called according to the permission corresponding relation recorded by the sandbox strategy file, and the processing of the target process task is completed within the authorized range of the operation process processing permission.

As an alternative, before the process processing request is acquired, the method further includes: s1, a cloud service platform acquires a file data packet uploaded by a target user account; s2, the cloud service platform generates an access right record for the file data packet, wherein the access right record is used for indicating a user account belonging to the same account set as the target user account and has the right of accessing the file data packet;

Before loading the file data packet matched with the target process task, the method further comprises the following steps: s1, a cloud service platform receives a permission verification request; s2, the cloud service platform performs authority verification by using the access authority record; and S3, under the condition that the permission verification request indicates that verification is passed, the cloud service platform sends a file data packet matched with the target process task to the resource manager cluster.

It should be noted that the above access right record may be, but not limited to, a record generated by an open source security management framework Ranger. That is, after the target user account uploads the jar packet, the cloud service platform may generate an access permission record (also referred to as a range record) for the target user account to record the read-write permission of the jar packet, for example, the target user account is configured with the read-write permission, and other user accounts belonging to the same engineering with the target user account are configured with the read-write permission and/or the writable permission.

Further, before loading the jar packet, a permission verification request needs to be initiated to an open source security management frame Ranger, and after querying the access permission record, the open source security management frame Ranger determines whether permission verification is passed. In the event that verification passes, access is determined to be allowed, then the resource coordinator (e.g., YARN cluster) is authorized to load the jar packet, e.g., notify the node manager that the jar packet is allowed to be loaded.

For example, as shown in fig. 8-9, when a user uploads a file data packet through a target user account, the user may, but is not limited to, implement uploading configuration of the file data packet through the uploading interface shown in fig. 8, and implement the target program file data packet of the new user-defined function through the new interface shown in fig. 9. The above is merely an example, and this is not limited in any way in the present embodiment.

According to the embodiment provided by the application, the user resources are effectively isolated by adopting the anger authority management UDF function and the corresponding jar resources, so that the safety and stability of the platform operation are ensured.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

According to another aspect of the embodiment of the present application, there is also provided a process processing apparatus for implementing the above-mentioned process processing method. As shown in fig. 10, the apparatus includes:

1) A first obtaining unit 1002, configured to obtain a process processing request, where the process processing request carries a target process task to be processed;

2) A loading unit 1004, configured to load a file data packet matched with a target process task in response to a process processing request;

3) A first adding unit 1006, configured to add, in a case that a file data packet includes a target program file data packet corresponding to a user-defined function, a first target indication parameter in a process start instruction for starting a target process, where the first target indication parameter is used to indicate a sandbox set in the starting target process, the sandbox is configured with a target process processing permission of the target process, and the target process processing permission is less than all process processing permissions;

4) And the processing unit 1008 is used for starting the target process provided with the sandbox according to the process starting instruction and processing the target process task through the target process.

Optionally, in this embodiment, the above-mentioned process processing device may be, but not limited to, applied to a scenario in which a user-defined function UDF is invoked to perform process processing when a public cloud big data service platform (hereinafter may be simply referred to as a cloud service platform) based on multiple users performs data analysis. That is, a sandbox environment is set for the process in the cloud service platform, so that the process processing process is automatically isolated and limited by the process processing authority corresponding to the sandbox, thereby ensuring that the target process to be processed can automatically run within the safety authority range, saving the event of safety audit and achieving the purpose of improving the process processing efficiency.

Alternatively, in the present embodiment, the above-described process processing apparatus may be applied, but not limited to, in the cloud service platform 302 shown in fig. 3. The cloud service platform 302 may include, but is not limited to, the following structures: hive, hadoop, YARN clusters. The Hadoop is an open source software framework supporting data intensive distributed application programs and issued by Apache2.0 license protocol, and supports application programs running on large clusters constructed by commodity hardware. The Hive is a data warehouse tool based on a Hadoop open source software framework, and is used for mapping a structured data file into a database table and providing a simple SQL query function. The YARN cluster is a universal resource management system for providing uniform resource management and scheduling for upper layer applications. The above-mentioned process processing device may be, but not limited to, applied to a yan cluster, for example, after Hive parses SQL query statement to obtain some columns of MapReduce process tasks (MR tasks for short), and then the MR tasks are submitted to the yan cluster to run, so that the yan cluster loads file data packets required by the yan task and the MR tasks, and processes the MR tasks safely and efficiently in a process with a sandbox.

As an alternative, the loading unit 1004 includes:

1) The first processing module is used for repeatedly executing the following steps until all file data packets matched with the target process task are traversed:

s1, acquiring a data packet identifier of a current file data packet loaded currently;

s2, comparing the data packet identifier with a key identifier, wherein the key identifier is used for identifying the target program file data packet;

s3, under the condition that the data packet identifier is matched with the key identifier, determining that the current file data packet is a target program file data packet, and storing the target program file data packet under a first target path;

s4, under the condition that the data packet identifier is not matched with the key identifier, determining that the current file data packet is not the target program file data packet, and storing the current file data packet under a second target path;

s5, loading the next file data packet as the current file data packet.

${yarn.nodemanager.local-dirs}/usercache/${user}/filecache/

For example, the second target path may be:

${yarn.nodemanager.local-dirs}/usercache/${user}/

Optionally, in this embodiment, the first processing module is further configured to: after determining that the current file data packet is the target program file data packet, adjusting a target monitoring variable for monitoring the target program file data packet included in the file data packet to a target value, wherein the target value is used for indicating that the target program file data packet appears in the loaded file data packet.

As an alternative, the method further comprises:

1) The second adding unit is used for adding a second target indication parameter into the process starting instruction when the first target indication parameter is added into the process starting instruction for starting the target process, wherein the second target indication parameter is used for indicating a file storage path where a sandbox strategy file matched with a sandbox is located, the sandbox strategy file is used for storing a corresponding relation between a target program file data packet under the first target path and operation process processing permission, and the target process processing permission comprises the operation process processing permission.

As an alternative, the processing unit 1008 includes:

1) The analysis module is used for analyzing the first target indication parameter and the second target indication parameter carried in the process starting instruction;

2) The starting module is used for starting a sandbox arranged in the target process according to the first target indicating parameter and acquiring a sandbox strategy file according to the second target indicating parameter when the target process is started;

3) The calling module is used for calling the target program file data packet under the first target path according to the sandbox strategy file;

4) And the second processing module is used for processing the target process task in the authorized range of the operation process processing authority corresponding to the target program file data packet.

According to a further aspect of the embodiments of the present application there is also provided an electronic device for implementing the above-described process-handling method, as shown in fig. 11, the electronic device comprising a memory 1102 and a processor 1104, the memory 1102 having stored therein a computer program, the processor 1104 being arranged to perform the steps of any of the method embodiments described above by means of the computer program.

Alternatively, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of the computer network.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

s1, acquiring a process processing request, wherein the process processing request carries a target process task to be processed;

s2, responding to the process processing request, and loading a file data packet matched with the target process task;

s3, under the condition that a file data packet comprises a target program file data packet corresponding to a user-defined function, adding a first target indication parameter into a process starting instruction for starting a target process, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, the sandbox is configured with target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions;

s4, starting a target process provided with a sandbox according to the process starting instruction, and processing a target process task through the target process.

Alternatively, it will be understood by those skilled in the art that the structure shown in fig. 11 is merely illustrative, and the electronic device may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, and a mobile internet device (Mobile Internet Devices, MID), a PAD, or other network processing hardware devices for implementing the cloud service platform processing function. Fig. 11 is not limited to the structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 11, or have a different configuration than shown in FIG. 11.

The memory 1102 may be used to store software programs and modules, such as program instructions/modules corresponding to the process processing methods and apparatuses in the embodiments of the present invention, and the processor 1104 executes the software programs and modules stored in the memory 1102 to perform various functional applications and data processing, i.e., implement the above-mentioned process processing methods. Memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 1102 may further include memory located remotely from processor 1104, which may be connected to the terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 1102 may be, but is not limited to, a file packet, including information such as a target program file packet corresponding to a user-defined function. As an example, as shown in fig. 11, the memory 1102 may include, but is not limited to, a first obtaining unit 1002, a loading unit 1004, a first adding unit 1006, and a processing unit 1008 in the process processing device. In addition, other module units in the above process processing apparatus may be included, but are not limited to, and are not described in detail in this example.

Optionally, the transmission device 1106 is used to receive or transmit data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 1106 includes a network adapter (Network Interface Controller, NIC) that may be connected to other network devices and routers via a network cable to communicate with the internet or a local area network. In one example, the transmission device 1106 is a Radio Frequency (RF) module for communicating wirelessly with the internet.

In addition, the electronic device further includes: a display 1108 for displaying configuration information, loading progress information, process processing information, or the like; and a connection bus 1110 for connecting the respective module parts in the above-described electronic apparatus.

According to a further aspect of embodiments of the present invention there is also provided a storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

Alternatively, in the present embodiment, the above-described storage medium may be configured to store a computer program for performing the steps of:

Alternatively, in this embodiment, it will be understood by those skilled in the art that all or part of the steps in the methods of the above embodiments may be performed by a program for instructing a terminal device to execute the steps, where the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

The integrated units in the above embodiments may be stored in the above-described computer-readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing one or more computer devices (which may be personal computers, servers or network devices, etc.) to perform all or part of the steps of the method described in the embodiments of the present application.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In several embodiments provided by the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims

1. A process processing method, comprising:

acquiring a process processing request, wherein the process processing request carries a target process task to be processed;

And responding to the process processing request, loading a file data packet matched with the target process task, wherein the loading of the file data packet matched with the target process task comprises the following steps: repeating the following steps until all file data packets matched with the target process task are traversed: acquiring a data packet identifier of a current file data packet loaded currently; comparing the data packet identifier with a key identifier, determining that the current file data packet is a target program file data packet corresponding to a user-defined function under the condition that the data packet identifier is matched with the key identifier, adjusting a target monitoring variable for monitoring the target program file data packet included in the file data packet matched with the target process task to be a target value, and storing the target program file data packet under a first target path; under the condition that the data packet identifier is not matched with the key identifier, determining that the current file data packet is not the target program file data packet, and storing the current file data packet under a second target path; loading a next file data packet as the current file data packet, wherein the key identifier is used for identifying the target program file data packet, and the target value is used for indicating that the target program file data packet appears in the loaded file data packet;

Determining that a file data packet matched with the target process task comprises the target program file data packet under the condition that the target monitoring variable is the target value, and adding a first target indication parameter into a process starting instruction for starting a target process, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, the sandbox is configured with target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions;

and starting the target process provided with the sandbox according to the process starting instruction, and processing the target process task through the target process.

2. The process processing method according to claim 1, wherein when the first target instruction parameter is added in a process start instruction for starting a target process, further comprising:

adding a second target indication parameter into the process starting instruction, wherein the second target indication parameter is used for indicating a file storage path where a sandbox strategy file matched with the sandbox is located, the sandbox strategy file is used for storing a corresponding relation between the target program file data packet and operation process processing permission under the first target path, and the target process processing permission comprises the operation process processing permission.

3. The process processing method according to claim 2, wherein the starting the target process provided with the sandbox according to the process start instruction and processing the target process task by the target process comprises:

analyzing the first target indication parameter and the second target indication parameter carried in the process starting instruction;

when the target process is started, starting the sandboxes arranged in the target process according to the first target indication parameters, and acquiring the sandbox strategy files according to the second target indication parameters;

invoking the target program file data packet under the first target path according to the sandbox strategy file;

and processing the target process task by utilizing the target program file data packet within the authorized range of the operation process processing authority corresponding to the target program file data packet.

4. The process processing method according to claim 1, wherein,

before the acquiring process processes the request, the method further comprises: the cloud service platform acquires the file data packet uploaded by the target user account; the cloud service platform generates an access right record for the file data packet, wherein the access right record is used for indicating a user account belonging to the same account set as the target user account and has the right of accessing the file data packet;

Before loading the file data packet matched with the target process task, the method further comprises the following steps: the cloud service platform receives a permission verification request; the cloud service platform performs authority verification by using the access authority record; and under the condition that the permission verification request indicates that verification is passed, the cloud service platform sends the file data packet matched with the target process task to a resource manager cluster.

5. The process processing method according to claim 1, further comprising, before the acquiring the process processing request:

parameter configuration is performed for the structured query language in the cloud service platform, so that the structured query language is parsed into process tasks for processing in the resource manager cluster.

6. A process processing apparatus, comprising:

the first acquisition unit is used for acquiring a process processing request, wherein the process processing request carries a target process task to be processed;

the loading unit is used for responding to the process processing request and loading a file data packet matched with the target process task, wherein the loading of the file data packet matched with the target process task comprises the following steps: repeating the following steps until all file data packets matched with the target process task are traversed: acquiring a data packet identifier of a current file data packet loaded currently; comparing the data packet identifier with a key identifier, determining that the current file data packet is a target program file data packet corresponding to a user-defined function under the condition that the data packet identifier is matched with the key identifier, adjusting a target monitoring variable for monitoring the target program file data packet included in the file data packet matched with the target process task to be a target value, and storing the target program file data packet under a first target path; under the condition that the data packet identifier is not matched with the key identifier, determining that the current file data packet is not the target program file data packet, and storing the current file data packet under a second target path; loading a next file data packet as the current file data packet, wherein the key identifier is used for identifying the target program file data packet, and the target value is used for indicating that the target program file data packet appears in the loaded file data packet;

The first adding unit is used for determining that a file data packet matched with the target process task comprises the target program file data packet and adding a first target indication parameter in a process starting instruction for starting a target process under the condition that the target monitoring variable is the target value, wherein the first target indication parameter is used for indicating to start a sandbox arranged in the target process, the sandbox is configured with target process processing permission of the target process, and the target process processing permission is smaller than all process processing permissions;

and the processing unit is used for starting the target process provided with the sandbox according to the process starting instruction and processing the target process task through the target process.

7. The process processing apparatus according to claim 6, further comprising:

the second adding unit is configured to add a second target indication parameter to the process start instruction when the first target indication parameter is added to the process start instruction for starting the target process, where the second target indication parameter is used to indicate a file storage path where a sandbox policy file matched with the sandbox is located, and the sandbox policy file is used to store a correspondence between the target program file data packet and an operation process processing permission under the first target path, where the target process processing permission includes the operation process processing permission.

8. The process processing apparatus according to claim 7, wherein the processing unit includes:

the analysis module is used for analyzing the first target indication parameter and the second target indication parameter carried in the process starting instruction;

the starting module is used for starting the sandbox set in the target process according to the first target indicating parameter and acquiring the sandbox strategy file according to the second target indicating parameter when the target process is started;

the calling module is used for calling the target program file data packet under the first target path according to the sandbox strategy file;

and the second processing module is used for processing the target process task in the authorized range of the operation process processing authority corresponding to the target program file data packet.

9. A computer readable storage medium comprising a stored program, wherein the program when run performs the method of any one of the preceding claims 1 to 5.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 5 by means of the computer program.