CN117251301A - Data calling method, component, system, computer equipment and storage medium - Google Patents
Data calling method, component, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN117251301A CN117251301A CN202310757157.8A CN202310757157A CN117251301A CN 117251301 A CN117251301 A CN 117251301A CN 202310757157 A CN202310757157 A CN 202310757157A CN 117251301 A CN117251301 A CN 117251301A
- Authority
- CN
- China
- Prior art keywords
- data
- calling
- processed
- service device
- instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000006870 function Effects 0.000 claims description 35
- 230000005856 abnormality Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 11
- 238000005520 cutting process Methods 0.000 claims description 8
- 230000004044 response Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 10
- 238000012545 processing Methods 0.000 abstract description 9
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 8
- 230000002349 favourable effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000013515 script Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data calling method, a data calling component, a data calling system, computer equipment and a storage medium, and relates to the technical field of data processing. The data calling method comprises the following steps: acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction; connecting a target data source to acquire data to be processed; and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service. The method can modularize the data calling function, so that the service device passively receives the data to be processed, the burden of writing data calling realization codes to each service device is reduced, and the method is beneficial to simplifying the updating and maintaining processes of the data calling function.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data calling method, a data calling component, a data calling system, a computer device, and a computer readable storage medium.
Background
With the rapid development of internet technology, technologies such as data batch calling and data batch processing have been developed. In general, service devices such as application programs and web pages are used as active invokers respectively to actively invoke data from at least one data source to execute services. Wherein the at least one data source comprises a database, a file, an external system, etc.
However, when there are multiple service devices, the data sources matched by the service devices may be different, and the data calling function of each service device needs to be designed, which has the problems of repeated development of the same function and high updating and maintenance costs.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data calling method, a data calling component, a data calling system, a computer device, and a computer-readable storage medium that are capable of modularizing a data calling function, enabling a service device to passively receive data to be processed, reducing the burden of writing data calling implementation codes to each service device, and facilitating the process of updating and maintaining the data calling function.
In one aspect, a data calling method is provided, the data calling method includes: acquiring a calling instruction and identifying a target data source indicated in the calling instruction; connecting a target data source to acquire data to be processed; and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
In an embodiment of the present application, forwarding data to be processed to an external service device includes: cutting the data to be processed to form a plurality of data units; and forwarding the plurality of data units to an external service device in turn.
In an embodiment of the present application, performing a cutting process on data to be processed to form a plurality of data units includes: acquiring the file size of data to be processed and the expected file size of a data unit; defining the reading starting position of the data to be processed as zero, reading the data to be processed and accumulating the file size until the accumulated file size is matched with the expected file size, loading the read data to be processed into a data unit, continuing to read the data to be processed until a first separator appears, and loading the data to be processed before the first separator which is continuously read into a data unit; to form a plurality of data units.
In an embodiment of the present application, forwarding the plurality of data units to the external service device sequentially includes: dividing a plurality of data units to form a preset number of data groups; and transmitting the preset number of data groups by using the preset number of threads, and sequentially transmitting each data unit in the data groups matched with the threads by each thread.
In an embodiment of the present application, a data calling method includes: acquiring a data source configuration instruction, establishing connection with a required data source based on the data source configuration instruction, and establishing a mapping relation between a data original field of the data source and a data expected field of an input external service device; acquiring a service configuration instruction, wherein the service configuration instruction is used for identifying the position of a service device so as to forward data to be processed to the service device matched with the service device through field mapping based on the service configuration instruction; and acquiring an execution configuration instruction, identifying a data calling time indicated in the execution configuration instruction, acquiring data to be processed from a data source according to the data calling time indicated by the execution configuration instruction, and forwarding the data to the service device.
In an embodiment of the present application, the data calling method further includes: counting the acquired original data to be processed, state change information of the data calling full period and calling task completion information; and triggering an abnormality prompt in response to the number of task execution abnormalities in the task completion information reaching an abnormality threshold.
In another aspect, a data calling component is provided, the data calling component comprising: the device comprises a control module, a calling module and a sending module; the control module is used for acquiring or generating a calling instruction, and the calling instruction is used for indicating to forward the data of the target data source to the target service device; the calling module is used for calling the data of the target data source in the calling instruction; the sending module is connected with the calling module and used for forwarding the appointed data to the matched external service device.
In another aspect, a data calling system is provided, the data calling system including: the data calling component of the embodiment is connected with the data source and is used for calling and forwarding the data in the data source; and the at least one business device is connected with the data calling component and is used for receiving the data forwarded by the data calling component and executing business functions on the data.
In yet another aspect, a computer device is provided comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction; connecting a target data source to acquire data to be processed; and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
In yet another aspect, a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of: acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction; connecting a target data source to acquire data to be processed; and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
The data calling method, the data calling component, the data calling system, the computer equipment and the computer readable storage medium are characterized in that the data calling component obtains a calling instruction, and the data to be processed is obtained by connecting a target data source based on the calling instruction. The data calling component forwards the data to be processed to the service device. The service device passively receives the data to be processed and executes the service on the received code to be processed. That is, an independent data calling component is designed to realize the data calling function of the service device, the data calling component is independent of other function codes of the service device, the modularization of relevant codes of the data calling function is realized, and the data calling component can be respectively embedded into a plurality of service devices to realize the data calling function of the plurality of service devices, so that the burden of respectively writing the data calling realization codes for each service device is reduced. When the data calling function is updated and maintained, one data calling component can be updated and maintained, and redundant operations for updating and maintaining the data calling function codes of a plurality of service devices respectively are reduced.
Drawings
FIG. 1 is a schematic diagram illustrating the structure of one embodiment of a data calling component of the present application;
FIG. 2 is a flow diagram of one embodiment of a data recall method of the present application;
FIG. 3 is a schematic diagram illustrating an embodiment of a data call system of the present application;
FIG. 4 is a flow chart of another embodiment of a data recall method of the present application;
FIG. 5 is a schematic diagram of an embodiment of a computer device of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
In order to solve the technical problem that the updating and maintenance of the data calling function are difficult in the prior art, the application provides a data calling method. The data calling method comprises the following steps. The following illustrates the data calling method of the present application.
In one embodiment, as shown in fig. 1, a data calling component is provided, and fig. 1 is a schematic structural diagram of an embodiment of the data calling component of the present application.
The data calling component comprises a control module 11, a calling module 12 and a sending module 13.
The control module 11 is used for acquiring or generating a call instruction. The calling instruction is used for indicating to forward the data of the target data source to the target service device. The call instruction may include information of the target data source, or may include both the target data source and a range of the specified call data, which is not limited herein.
The control module 11 may generate the call instruction based on a preset; the calling instruction input by the user/service device can also be received; the method may also be to receive an execution configuration instruction of the user/service device, identify a data call timing indicated in the execution configuration instruction, and generate a call instruction at the data call timing, which is not limited herein.
Alternatively, the data call occasion may be a data call period. For example, the data call timing may be to periodically perform data call at intervals of a predetermined duration, or may refer to monitoring a data source, and performing data call when data in the data source reaches a predetermined file size, which is not limited herein.
The calling module 12 may be connected to the control module 11, or the calling module 12 may be integral with the control module 11. The calling module 12 may also be connected to at least one external data source.
Alternatively, at the beginning of the data calling component, the connection method with at least one data source and the data calling method can be only included, and after receiving the data source configuration instruction of the user, the connection with the currently required data source is established. Or, when the device is connected with the service device, the device can acquire the information of the data source required by the service device by means of active identification, receiving the instruction of the service device and the like, and establish connection with the data source required by the service device. Or, the method for connecting the data source and the data calling method can be configured by the user through the interface capable of interacting with the user without the method for connecting the data source and the data calling method, so that the functionality and the reusability of the data calling component of the embodiment can be improved. Of course, the data calling component can select one of the three configuration modes, or can select at least two configuration modes in combination, so that the functionality of the data calling component is further improved.
The call module 12 is capable of making calls to pending data within a target data source indicated in the call instruction. All data of the target data source may be called as data to be processed, or data in a specified range in a call instruction may be called as data to be processed, which is not limited herein.
The sending module 13 is connected with the calling module 12, and the sending module 13 is also connected with an external service device. The sending module 13 can receive the data to be processed acquired by the calling module 12 and forward the specified data to the matched external service device.
It can be seen that, in this embodiment, a data calling component independent of other functions of a service device is provided, where the data calling component can be connected to a data source to obtain data to be processed of the data source, and can also forward the obtained data to be processed to an external service device. In other words, in this embodiment, the implementation codes of the data calling function are independently written to form an independent data calling component, so that when the service device needs data calling, the data calling component is connected with the service device or the data calling component is embedded into the service device, and the same data calling component can be adapted to multiple service devices, which is favorable for reducing the tedious work of repeatedly developing the logic of the data calling function and improving the reusability of the public function.
In the embodiment, the decoupling of the data calling function and the service function can be realized, the embedding of the code realized by the data calling function is reduced, and the problem of excessively high coupling degree of the data calling function and the service function is solved as far as possible. Even if the data calling component is embedded into the service device, the positions of the realization codes of the data calling function are concentrated, which is favorable for reducing the difficulty of updating and maintaining the data calling function, and further is favorable for reducing the updating and maintaining cost.
In one embodiment, as shown in fig. 2, a data calling method is provided, and fig. 2 is a schematic flow chart of an embodiment of the data calling method of the present application. It should be noted that, in this embodiment, the working process of the data calling component in the foregoing embodiment is described by way of example, including all the details described in the foregoing embodiment, and will not be described in detail in this embodiment.
S201: and acquiring a calling instruction and identifying a target data source indicated in the calling instruction.
In this embodiment, the call instruction is used to instruct to call the pending data from the target data source to send to the service device. The call instruction can identify the service device and the target data source which are currently required to execute the service.
The data calling component can be compatible with at least one service device, and the service device is an application program, a webpage and the like for realizing specific service functions. The number of business devices may be one or more.
The data calling component can be compatible with at least one data source. The number of the data sources can be one or more, and the types of the data sources can comprise databases, document files, cloud resource files and the like, so that the functionality and compatibility of the data calling component are improved, and more different service devices can be adapted.
Optionally, the call instruction may include information of the target data source, and may also include information of the target data source and information of the required call data at the same time, which is not limited herein. The call instruction may instruct to call all data of the target data source as data to be processed, or instruct to call specified data as data to be processed, which is not limited herein.
Wherein the number of target data sources indicated in the call instruction may be one or more. That is, the number of target data sources indicated in the call instruction may be one, two, three, four, ten, and is not limited herein.
S202: and connecting a target data source to acquire data to be processed.
In this embodiment, the data calling component is connected to a target data source indicated in the calling instruction, and obtains data to be processed.
When a plurality of target data sources are provided, the target data sources are respectively connected with the target data sources, and the data to be processed in the target data sources are acquired simultaneously or sequentially.
S203: and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
In this embodiment, the data calling component forwards the data to be processed obtained from the target data source to the external service device. If so, the service device can passively receive the data to be processed from the data calling component, and process the data to be processed to execute the service.
It should be noted that, the term "external" in the external service device refers to a relative data calling component itself, and does not limit a connection relationship between the service device and the data calling component.
In the data calling method, the data calling component acquires the calling instruction, and the data to be processed is acquired by connecting the target data source based on the calling instruction. The data calling component forwards the data to be processed to the service device. The service device passively receives the data to be processed and executes the service on the received code to be processed. That is, an independent data calling component is designed to realize the data calling function of the service device, the data calling component is independent of other function codes of the service device, the modularization of relevant codes of the data calling function is realized, and the data calling component can be respectively embedded into a plurality of service devices to realize the data calling function of the plurality of service devices, so that the burden of respectively writing the data calling realization codes for each service device is reduced. When the data calling function is updated and maintained, one data calling component can be updated and maintained, and redundant operations for updating and maintaining the data calling function codes of a plurality of service devices respectively are reduced.
In one embodiment, as shown in fig. 3, a data calling system is provided, and fig. 3 is a schematic structural diagram of an embodiment of the data calling system of the present application.
The data call system comprises a data call component 31 and a business device 32.
The data calling component 31 is connected with the data source and is used for calling the data in the data source and forwarding. The structure and the working principle of the data calling component 31 are as described in the above embodiments, and will not be described herein.
Alternatively, the data call component 31 may be a JAR file. JAR files are a software package file format that is commonly used to aggregate a large number of Java class files, associated metadata and resource (text, pictures, etc.) files into one file in order to develop Java platform applications or libraries. Of course, in alternative embodiments, the data calling component 31 may also be written in the Phython language, C language, which is not limited herein.
The number of service devices 32 may be at least one, and the service devices 32 are connected to the data calling component 31, and are configured to receive data (i.e. data to be processed) forwarded by the data calling component 31, and process the data to perform service functions.
Optionally, the data calling component 31 may further have an identification function, which can identify each service device 32 connected thereto, so as to ensure that the acquired data to be processed can be sent to the service device 32 matched with the data to be processed.
In one embodiment, as shown in fig. 4, another data calling method is provided, and fig. 4 is a schematic flow chart of another embodiment of the data calling method of the present application.
In this embodiment, at least one of the data source, the service device, and the execution mode may be configured in advance.
For example, the data source, the service device, or the execution mode may be preconfigured, or at least two of the data source, the service device, and the execution mode may be preconfigured, or all three may be preconfigured.
Specifically, configuring the data source may be: and acquiring a data source configuration instruction, establishing connection with a required data source based on the data source configuration instruction, and establishing a mapping relation between a data original field of the data source and a data expected field of an input external service device.
And, the data call component can be configured to establish connections with a plurality of data sources. For example, the user may select a data source desired to be connected on a page of the interactive module, or the data calling component obtains a data source required by a service device connected with the data source, to form a data source configuration instruction.
For example, when the data source comprises a database, the data source configuration instructions may include database instances, library names, table names, and scripts that may also accept queries.
When the data source includes a document file, the data source configuration instruction includes a required document file, and the document file may be an excel table file, a csv (common-separation values) file, or the like.
When the data source comprises a cloud resource file, the data source configuration instruction comprises a bucket and a path corresponding to the cloud resource file.
When the data source comprises calling the downstream system to acquire, the data source configuration instruction comprises a downstream service request address, parameter information and the like.
The configuration of the service device may be: the service configuration instruction is acquired and used for identifying the position of the service device so as to forward the data to be processed to the service device matched with the service device through field mapping based on the service configuration instruction, thereby improving the functionality of the data calling component of the embodiment and being beneficial to enriching the applicable scene of the data calling component.
For example, if the data field in the data source is u-name, and the service device/data calling component calls the uName corresponding to the interface, then a mapping relationship between the two is established.
Meanwhile, downstream call configuration can be performed, and after the read data is mapped through the field, a calling party (namely a service device) needing to be sent is configured. The call configuration instruction may include call information such as a domain name, an interface, and the like, and may also send MQ (message queue) information directly through data source information.
The configuration of the execution mode may be: and acquiring an execution configuration instruction, identifying a data calling time indicated in the execution configuration instruction, acquiring data to be processed from a data source according to the data calling time indicated by the execution configuration instruction, and forwarding the data to the service device.
In particular, task execution may be configured. For example, a trigger mode of the data invocation task may be selected, which may include a timed trigger, an immediate trigger, a trigger based on an accumulated amount of data, etc. When the task execution is configured, one data triggering mode can be selected, and multiple data triggering modes can be selected to be performed simultaneously.
The timed triggering data invoking task can set a preset time length, and periodically trigger the data invoking task with the preset time length as an interval. For example, a cron (project task) expression can be configured, and when the configured data is called, the data calling task is automatically triggered, so that the data calling function is realized. Or, the data call timing may also be specified; for example, a daily of 7: the 00am triggers the data call task, or designates the last month to trigger the data call task, without limitation.
The immediate triggering of the data calling task means that after the configuration is completed, the user manually selects to execute the data calling task to trigger the data calling task, and/or the data calling component can receive the related control instruction of the service device to trigger the data calling task immediately.
Triggering the data call task based on the accumulated data amount means that the newly added data amount and/or the total data amount in the data source can be monitored, and the data call task is triggered when the preset file size is met.
Further, the data calling component may be provided with an interaction module for interacting with a user. For example, the interaction module may be a visual page configuration that may include displays, inputs, etc., such that the data calling component may also support the user in configuring data sources, business devices, execution modes, etc. through the interaction module.
The following illustrates the process of executing a data call method based on a pre-configured data call component:
s401: and acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction.
In this embodiment, the call instruction may include both the target data source and the data information to be called to be processed.
The get call instruction may be a call instruction that is actively obtained by the data call component, or is passively received, or is generated by itself, which is not limited herein.
The data calling component acquires a calling instruction and identifies a target data source and data to be processed indicated by the calling instruction.
Wherein the target data source is a data source from which the data call component needs to call data, and the target data source is at least one of a plurality of data sources connected with the data call component. If the data calling component identifies that the target data source indicated in the calling instruction is not connected with the target data source, the data calling component feeds back prompt information to the user so as to allow the user to perform data source configuration, and the data calling component is connected with a new data source.
The data to be processed is the data in the data source which needs to be called by the data calling component.
S402: and connecting a target data source to acquire data to be processed.
In this embodiment, in response to the data calling component identifying the target data source and the data to be processed indicated in the call instruction, the connection target data source obtains the data to be processed.
S403: cutting the data to be processed to form a plurality of data units.
In this embodiment, the data to be processed may be cut when the data to be processed is acquired from the target data source, and/or the data to be processed may be cut when the data to be processed is forwarded to the service device.
The data to be processed is cut to form a plurality of data units, and each data unit is sequentially transmitted to an external service device, so that the risk of occurrence of abnormality is reduced by reducing single data transmission quantity, the prior transmitted data units and the data units which are not transmitted are affected when abnormality occurs, and the risk of memory overflow is reduced.
Further, the data to be processed may be cut into a relatively uniform plurality of data units.
Specifically, the file size of the data to be processed and the desired file size of the data unit may be obtained, so that the file size of each data to be processed matches the desired file size.
Alternatively, the desired file size may be set directly in advance, or the desired file size may be obtained by combining the file size of the data to be processed and the number of desired data units.
That is, the file sizes of the data units may be equal to the desired file size, or the difference between the file sizes of the data units and the desired file size may be within a predetermined range, or the file sizes of the data units may be constrained in combination with the desired file size.
Still further, the file size of the data unit is constrained to a desired file size.
When forming each data unit, the initial position of the data to be processed can be defined as zero, the data to be processed is read, the file size is accumulated until the accumulated file size is matched with the expected file size, and the read data to be processed is loaded to form a data unit. And the data to be processed is continuously read until the first separator appears, the data to be processed before the first separator which is continuously read is loaded into a data unit, if so, a data unit is formed, and the division of the data belonging to the same main body into two data units can be reduced, so that the integrity of the data of each main body can be guaranteed, if the service device can directly read each data unit, the data units can be spliced according to the splitting mode, and the data of the same main body can be read together.
Wherein, separator can be line feed, comma, period, page break, etc. Since the data of the data unit has been loaded to form the data unit, the starting position of the data to be processed can still be defined as zero, and so on to form a plurality of data units.
S404: and forwarding the plurality of data units to an external service device in turn.
In this embodiment, each data unit may be sequentially transmitted in a single-threaded manner, and each data unit may be sequentially forwarded to an external service device, so as to reduce the risk of memory overflow. Or, each data unit can be sequentially transmitted in a multithreading mode, and the data units are forwarded to an external service device in a multithreading mode, so that the data forwarding/reading efficiency is improved, the file is quickly read, and the risk of memory overflow is reduced.
For example, when data units are passed through the multiline Cheng Zhuaifa, a plurality of data units may be divided to form a predetermined number of data groups. And transmitting the preset number of data groups by using the preset number of threads, and sequentially transmitting each data unit in the data groups matched with each thread by each thread, so that the data transmission quantity of each thread is balanced.
Wherein the number of threads may be matched to the number of data clusters. The number of data groups may be the same as the number of threads, or the number of data groups may be a multiple of the number of threads, which is not limited herein.
If so, the data to be processed is forwarded to an external service device, and the service device receives the data to be processed to execute the service.
In an alternative embodiment, when the data calling component cuts the data to be processed when the data to be processed is obtained from the target data source, each data unit may be read by a single-thread mode or by a multi-thread mode.
In addition, the data calling component in the embodiment can be configured to accommodate multiple return formats of the downstream data processing party. For example, the return format may be json, xml, etc.
Meanwhile, the data calling method can also count the acquired original data to be processed, state change information of the data calling full period and calling task completion information. And triggering an abnormality prompt in response to the number of task execution abnormalities in the task completion information reaching an abnormality threshold. If so, the embodiment can monitor the service condition, missing/complete condition and the like of the data source, which is favorable for users to find the working state of the data calling function in time and judge whether maintenance and update are needed or not in combination with the working state.
In summary, the data calling method of the present application may be at least divided into the following processes: data source configuration, service configuration, downstream call configuration, task execution configuration, data storage table falling and alarm configuration. According to the method and the device, redundant codes can be reduced, through the data calling component for batch file processing, a plurality of business devices of the business device/data calling system can reduce writing of repeated codes, and batch data processing can be performed through visual configuration. And supporting uploading and reading of the oversized file, dividing the oversized file into a plurality of files according to the byte size in a file cutting mode, and simultaneously avoiding splitting the same data into two different files during division. Multiple data sources are supported for selection, multiple data sources can be selected, each data source has own processing logic, the dynamic expansion is realized, and the requirements of different data sources are met. The method supports various data processing logics, can send the acquired data to a downstream application party for processing, and can also directly process, save and send MQ messages on the basis of a data source.
It should be understood that, although the steps in the flowcharts of fig. 2 and 4 are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 2, 4 may comprise a plurality of sub-steps or phases, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or sub-steps of other steps.
For specific limitations of the data calling component and the data calling system, reference may be made to the above limitation of the data calling method, and no further description is given here. The above-described data calling components, and the respective modules in the data calling system may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, as shown in fig. 5, a computer device is provided, and fig. 5 is a schematic structural diagram of an embodiment of the computer device of the present application.
The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data calling method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 5 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided that includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
s201: and acquiring a calling instruction and identifying a target data source indicated in the calling instruction.
S202: and connecting a target data source to acquire data to be processed.
S203: and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
In an embodiment, the processor when executing the computer program further performs the steps of:
at least one of the data source, the service device and the execution mode may be configured in advance.
For example, the data source, the service device, or the execution mode may be configured in advance, or at least two of the data source, the service device, and the execution mode may be configured in advance, or the data source, the service device, and the execution mode may be configured in advance.
Specifically, configuring the data source may be: and acquiring a data source configuration instruction, establishing connection with a required data source based on the data source configuration instruction, and establishing a mapping relation between a data original field of the data source and a data expected field of an input external service device.
The configuration of the service device may be: and acquiring a service configuration instruction, wherein the service configuration instruction is used for identifying the position of the service device so as to forward the data to be processed to the service device matched with the data through field mapping based on the service configuration instruction.
The configuration of the execution mode may be: and acquiring an execution configuration instruction, identifying a data calling time indicated in the execution configuration instruction, acquiring data to be processed from a data source according to the data calling time indicated by the execution configuration instruction, and forwarding the data to the service device.
The following illustrates the process of executing a data call method based on a pre-configured data call component:
s401: and acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction.
S402: and connecting a target data source to acquire data to be processed.
S403: cutting the data to be processed to form a plurality of data units.
In this embodiment, the file size of the data to be processed and the desired file size of the data unit may be acquired.
Defining the reading starting position of the data to be processed as zero, reading the data to be processed and accumulating the file size until the accumulated file size is matched with the expected file size, loading the read data to be processed into a data unit, continuing to read the data to be processed until a first separator appears, and loading the data to be processed before the first separator which is continuously read into a data unit; to form a plurality of data sheets.
S404: and forwarding the plurality of data units to an external service device in turn.
In this embodiment, a plurality of data units may be divided to form a predetermined number of data groups. And transmitting the preset number of data groups by using the preset number of threads, and sequentially transmitting each data unit in the data groups matched with the threads by each thread.
If so, the data to be processed is forwarded to an external service device, and the service device receives the data to be processed to execute the service.
Meanwhile, the data calling method can also count the acquired original data to be processed, state change information of the data calling full period and calling task completion information. And triggering an abnormality prompt in response to the number of task execution abnormalities in the task completion information reaching an abnormality threshold.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:
s201: and acquiring a calling instruction and identifying a target data source indicated in the calling instruction.
S202: and connecting a target data source to acquire data to be processed.
S203: and forwarding the data to be processed to an external service device, and enabling the service device to receive the data to be processed so as to execute the service.
In one embodiment, the computer program when executed by the processor further performs the steps of:
at least one of the data source, the service device and the execution mode may be configured in advance.
For example, the data source, the service device, or the execution mode may be configured in advance, or at least two of the data source, the service device, and the execution mode may be configured in advance, or the data source, the service device, and the execution mode may be configured in advance.
Specifically, configuring the data source may be: and acquiring a data source configuration instruction, establishing connection with a required data source based on the data source configuration instruction, and establishing a mapping relation between a data original field of the data source and a data expected field of an input external service device.
The configuration of the service device may be: and acquiring a service configuration instruction, wherein the service configuration instruction is used for identifying the position of the service device so as to forward the data to be processed to the service device matched with the data through field mapping based on the service configuration instruction.
The configuration of the execution mode may be: and acquiring an execution configuration instruction, identifying a data calling time indicated in the execution configuration instruction, acquiring data to be processed from a data source according to the data calling time indicated by the execution configuration instruction, and forwarding the data to the service device.
The following illustrates the process of executing a data call method based on a pre-configured data call component:
s401: and acquiring a calling instruction, and identifying a target data source and data to be processed indicated in the calling instruction.
S402: and connecting a target data source to acquire data to be processed.
S403: cutting the data to be processed to form a plurality of data units.
In this embodiment, the file size of the data to be processed and the desired file size of the data unit may be acquired.
Defining the reading starting position of the data to be processed as zero, reading the data to be processed and accumulating the file size until the accumulated file size is matched with the expected file size, loading the read data to be processed into a data unit, continuing to read the data to be processed until a first separator appears, and loading the data to be processed before the first separator which is continuously read into a data unit; to form a plurality of data sheets.
S404: and forwarding the plurality of data units to an external service device in turn.
In this embodiment, a plurality of data units may be divided to form a predetermined number of data groups. And transmitting the preset number of data groups by using the preset number of threads, and sequentially transmitting each data unit in the data groups matched with the threads by each thread.
If so, the data to be processed is forwarded to an external service device, and the service device receives the data to be processed to execute the service.
Meanwhile, the data calling method can also count the acquired original data to be processed, state change information of the data calling full period and calling task completion information. And triggering an abnormality prompt in response to the number of task execution abnormalities in the task completion information reaching an abnormality threshold.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.
Claims (10)
1. A data calling method, characterized in that the data calling method comprises:
acquiring a calling instruction and identifying a target data source indicated in the calling instruction;
connecting the target data source to acquire data to be processed;
and forwarding the data to be processed to an external service device, so that the service device receives the data to be processed to execute the service.
2. The data calling method according to claim 1, wherein forwarding the data to be processed to an external service device comprises:
cutting the data to be processed to form a plurality of data units;
and forwarding the plurality of data units to an external service device in turn.
3. The method of claim 2, wherein the performing a cutting process on the data to be processed to form a plurality of data units includes:
acquiring the file size of the data to be processed and the expected file size of a data unit;
defining the reading starting position of the data to be processed as zero, reading the data to be processed and accumulating the file size until the accumulated file size is matched with the expected file size, loading the read data to be processed into a data unit, continuing to read the data to be processed until a first separator appears, and loading the data to be processed before the first separator which is continuously read into the data unit; to form a plurality of said data units.
4. The data call method of claim 2, wherein forwarding the plurality of data units to an external service device in sequence comprises:
dividing the plurality of data units to form a preset number of data groups;
and transmitting the preset number of data groups by using the preset number of threads, wherein each thread sequentially transmits each data unit in the data groups matched with each thread.
5. The data calling method according to claim 1, wherein the data calling method comprises:
acquiring a data source configuration instruction, establishing connection with a required data source based on the data source configuration instruction, and establishing a mapping relation between a data original field of the data source and a data expected field of an input external service device;
acquiring a service configuration instruction, wherein the service configuration instruction is used for identifying the position of a service device so as to forward the data to be processed to the service device matched with the data to be processed through field mapping based on the service configuration instruction;
and acquiring an execution configuration instruction, identifying a data calling time indicated in the execution configuration instruction, acquiring data to be processed from a data source according to the data calling time indicated by the execution configuration instruction, and forwarding the data to a service device.
6. The data calling method of claim 1, wherein the data calling method further comprises:
counting the acquired original data to be processed, state change information of the data calling full period and calling task completion information;
and triggering an abnormality prompt in response to the abnormal times of task execution in the task completion information reaching an abnormality threshold.
7. A data calling component, the data calling component comprising:
the control module is used for acquiring or generating a calling instruction, and the calling instruction is used for indicating to forward the data of the target data source to the target service device;
the calling module is used for calling the data of the target data source in the calling instruction;
and the sending module is connected with the calling module and used for forwarding the appointed data to the matched external service device.
8. A data call system, the data call system comprising:
the data calling component of claim 7, coupled to a data source, for calling and forwarding data within the data source;
and the service device is connected with the data calling component and is used for receiving the data forwarded by the data calling component and executing service functions on the data.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the data recall method of any one of claims 1 to 6 when the computer program is executed by the processor.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the data invoking method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310757157.8A CN117251301A (en) | 2023-06-26 | 2023-06-26 | Data calling method, component, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310757157.8A CN117251301A (en) | 2023-06-26 | 2023-06-26 | Data calling method, component, system, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117251301A true CN117251301A (en) | 2023-12-19 |
Family
ID=89133917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310757157.8A Pending CN117251301A (en) | 2023-06-26 | 2023-06-26 | Data calling method, component, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117251301A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110580244A (en) * | 2019-09-12 | 2019-12-17 | 深圳乐信软件技术有限公司 | file processing method and device, server and storage medium |
| CN110599080A (en) * | 2019-08-05 | 2019-12-20 | 江苏苏宁物流有限公司 | Service data distribution processing method and device, computer equipment and storage medium |
| CN111026797A (en) * | 2019-12-02 | 2020-04-17 | 卫盈联信息技术（深圳）有限公司 | Service data processing method, server and storage medium |
| CN111147548A (en) * | 2019-12-06 | 2020-05-12 | 深圳华锐金融技术股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN114817357A (en) * | 2022-03-09 | 2022-07-29 | 北京中交兴路信息科技有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN115510012A (en) * | 2022-09-13 | 2022-12-23 | 中国建设银行股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN116028246A (en) * | 2023-02-15 | 2023-04-28 | 苏州浪潮智能科技有限公司 | Scheduling method, system, computer equipment and storage medium for data processing |
| CN116204575A (en) * | 2023-03-09 | 2023-06-02 | 中国建设银行股份有限公司 | Method, device, equipment and computer storage medium for importing data into database |
-
2023
- 2023-06-26 CN CN202310757157.8A patent/CN117251301A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110599080A (en) * | 2019-08-05 | 2019-12-20 | 江苏苏宁物流有限公司 | Service data distribution processing method and device, computer equipment and storage medium |
| CN110580244A (en) * | 2019-09-12 | 2019-12-17 | 深圳乐信软件技术有限公司 | file processing method and device, server and storage medium |
| CN111026797A (en) * | 2019-12-02 | 2020-04-17 | 卫盈联信息技术（深圳）有限公司 | Service data processing method, server and storage medium |
| CN111147548A (en) * | 2019-12-06 | 2020-05-12 | 深圳华锐金融技术股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN114817357A (en) * | 2022-03-09 | 2022-07-29 | 北京中交兴路信息科技有限公司 | Data processing method, device, electronic equipment, storage medium and system |
| CN115510012A (en) * | 2022-09-13 | 2022-12-23 | 中国建设银行股份有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN116028246A (en) * | 2023-02-15 | 2023-04-28 | 苏州浪潮智能科技有限公司 | Scheduling method, system, computer equipment and storage medium for data processing |
| CN116204575A (en) * | 2023-03-09 | 2023-06-02 | 中国建设银行股份有限公司 | Method, device, equipment and computer storage medium for importing data into database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2016382741B2 (en) | Interface data displaying method and device | |
| CN108804618B (en) | Database configuration method, device, computer equipment and storage medium | |
| CN107273195B (en) | Big data batch processing method and device and computer system | |
| CN110351342A (en) | Service order processing method, device, computer equipment and storage medium | |
| CN108959385B (en) | Database deployment method, device, computer equipment and storage medium | |
| CN109766253B (en) | Performance data sending method and device, computer equipment and storage medium | |
| CN103942225A (en) | Method and system for invoking resources of Hybrid App client and client | |
| CN110765001B (en) | Modularized automatic test method and device for application system | |
| CN113032125B (en) | Job scheduling method, job scheduling device, computer system and computer readable storage medium | |
| CN108900627B (en) | Network request method, terminal device and storage medium | |
| CN113703954A (en) | Message backup method and device, electronic equipment and computer storage medium | |
| CN113360807A (en) | Page display method and device of mixed-mode mobile application and related equipment | |
| CN111736923A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN112882846A (en) | Data processing method and device of message queue, computer equipment and storage medium | |
| CN113626225B (en) | Program calling method, terminal device and medium | |
| KR101689984B1 (en) | Programmable controller, programmable controller system, and execute error information creation method | |
| CN109101340B (en) | Dynamic distribution method, device, equipment and storage medium of barrage message | |
| CN117251301A (en) | Data calling method, component, system, computer equipment and storage medium | |
| CN112132652A (en) | Order information acquisition method and device, computer equipment and storage medium | |
| CN117194147A (en) | Container cluster monitoring system, method, device and storage medium | |
| EP4414836A1 (en) | Container engine, container engine implementation methods, electronic device and storage medium | |
| CN115952227A (en) | Data acquisition system and method, electronic device and storage medium | |
| CN116226045A (en) | File data aggregation method, file data aggregation device and query system | |
| CN114675940A (en) | Application instance construction method, device and equipment | |
| CN114996038A (en) | Management method, system and storage medium of software entity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |