CN115826934B - Application development system and method - Google Patents
Application development system and method Download PDFInfo
- Publication number
- CN115826934B CN115826934B CN202310111760.9A CN202310111760A CN115826934B CN 115826934 B CN115826934 B CN 115826934B CN 202310111760 A CN202310111760 A CN 202310111760A CN 115826934 B CN115826934 B CN 115826934B
- Authority
- CN
- China
- Prior art keywords
- data
- input
- application
- input data
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000011161 development Methods 0.000 title claims abstract description 109
- 238000000034 method Methods 0.000 title claims abstract description 82
- 238000012545 processing Methods 0.000 claims abstract description 178
- 230000008569 process Effects 0.000 claims abstract description 55
- 238000004590 computer program Methods 0.000 claims description 9
- 230000003139 buffering effect Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 238000000926 separation method Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Stored Programmes (AREA)
Abstract
The invention discloses an application development system and method, and relates to the technical field of big data. One embodiment of the method comprises the following steps: receiving a service application development request; determining an execution starting point, an execution end point, data input, data output and data processing logic corresponding to the service application according to the service application development request; configuring an application programming interface to define the data input, data output, and data processing logic; configuring a service application development component to enable the service development component to acquire input data and process the input data by analyzing the definition of the application programming interface; and configuring a business application generating component to define the execution starting point and the execution end point, and generating a business application by assembling the application programming interface and the business application development component. The implementation mode avoids dependence of service application development on a specific technical platform and products, is flexible, controllable, simple and easy to learn, and has high universality.
Description
Technical Field
The invention relates to the technical field of big data, in particular to an application development system and method.
Background
Along with the explosive growth of data, to improve the processing efficiency of mass data, corresponding service applications are often developed according to service requirements so as to process big data. And if the large data development tool is used, corresponding codes are written in the built large data development environment by utilizing large data technologies such as a distributed computing theory and the like, and corresponding business application development is carried out.
In the process of implementing the present invention, the inventor finds that at least the following problems exist in the prior art: the development of business application often depends on a specific technical platform and a third party product, and autonomous controllability cannot be realized; the development complexity of the service application is high, the technical level requirement on the developer is high, the service application cannot be popularized well, and the development efficiency of the service application is reduced to a certain extent.
Disclosure of Invention
In view of the above, the embodiment of the invention provides a system and a method for developing service application, which can avoid the dependence of service application development on a specific technical platform and products, are flexible, controllable, simple and easy to learn, have high universality and can improve the development efficiency of service application.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided an application development system including: an application programming interface, a business application development component, and a business application generation component; wherein,
The application programming interface is used for defining data input, data output and data processing logic corresponding to the business application;
the business application development component is used for acquiring input data and processing the input data by analyzing the definition of the application programming interface;
the business application generating component is used for defining an execution starting point and an execution end point corresponding to the business application and generating the business application by assembling the application programming interface and the business application development component.
Optionally, the service application development component includes: an external data reader, a processing logic executor; wherein,
the external data reader is used for acquiring input data by analyzing data input definition;
the processing logic executor is used for processing the input data through analyzing the data processing logic definition so as to acquire an output result corresponding to the input data.
Optionally, the method further comprises: an application running container; the application running container is used for running the business application to process the input data.
Optionally, the application running container includes: an application loader, a calculation thread pool and a data cache pool; wherein,
The application loader is used for reading and analyzing the components of the business application;
the computing thread pool is used for supporting the processing logic executor to process the input data;
the data buffer pool is used for buffering the input data and the output result corresponding to the input data.
Optionally, the data buffer pool is configured to buffer the input data and an output result corresponding to the input data by using a data queue.
Optionally, the processing logic executor includes one or more execution units, where the execution units are configured to process the input data.
Optionally, the execution unit performs cooperative processing on the input data by adopting the following three threads: a data input thread, a business processing thread and a result output thread.
Optionally, the data input thread is configured to obtain input data according to a data input definition, and push the input data to a first data queue;
the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue;
And the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
Optionally, the data input thread is further configured to generate an error information record, push the error information record to the first data queue, and end pushing the input data to the first data queue when an error occurs in the input data.
Optionally, the service processing thread is further configured to return an error prompt message to a call site of the processing logic executor when an error occurs in a process of processing the input data in a case that an error information record is acquired from the first data queue.
Optionally, the result output thread is further configured to interrupt the data input thread and the service processing thread when an output result stored in the second data queue is wrong, and return an error prompt message to a call place of the processing logic executor.
To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided a service application development method, including:
Receiving a service application development request;
determining an execution starting point, an execution end point, data input, data output and data processing logic corresponding to the service application according to the service application development request;
configuring an application programming interface to define the data input, data output, and data processing logic;
configuring a service application development component to enable the service development component to acquire input data and process the input data by analyzing the definition of the application programming interface;
and configuring a business application generating component to define the execution starting point and the execution end point, and generating a business application by assembling the application programming interface and the business application development component.
Optionally, the service application development component includes: an external data reader, a processing logic executor; wherein,
the external data reader is used for acquiring input data by analyzing data input definition;
the processing logic executor is used for processing the input data through analyzing the data processing logic definition so as to acquire an output result corresponding to the input data.
Optionally, the business application is run by an application running container to process the input data.
Optionally, the application running container includes: an application loader, a calculation thread pool and a data cache pool; wherein,
the application loader is used for reading and analyzing the components of the business application;
the computing thread pool is used for supporting the processing logic executor to process the input data;
the data buffer pool is used for buffering the input data and the output result corresponding to the input data.
Optionally, in the data buffer pool, a data queue is used to buffer the input data and the output result corresponding to the input data.
Optionally, the processing logic executor includes one or more execution units, where the execution units are configured to process the input data.
Optionally, the execution unit performs cooperative processing on the input data by adopting the following three threads: a data input thread, a business processing thread and a result output thread.
Optionally, the data input thread is configured to obtain input data according to a data input definition, and push the input data to a first data queue;
the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue;
And the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
Optionally, the data input thread is further configured to generate an error information record, push the error information record to the first data queue, and end pushing the input data to the first data queue when an error occurs in the input data.
Optionally, the service processing thread is further configured to return an error prompt message to a call site of the processing logic executor when an error occurs in a process of processing the input data in a case that an error information record is acquired from the first data queue.
Optionally, the result output thread is further configured to interrupt the data input thread and the service processing thread when an output result stored in the second data queue is wrong, and return an error prompt message to a call place of the processing logic executor.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic device for application development, including: one or more processors; and storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the methods of application development described above.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the application development methods described above.
One embodiment of the above invention has the following advantages or benefits: by means of the mode that the application development system comprising the application programming interface, the service application development component and the service application generation component is assembled to generate the service application, a developer can develop the service application by only determining data input, data output and data processing logic, complete decoupling of service application development and software and hardware technical architecture is achieved, dependence of service application development on a specific technical platform and products is avoided, the method is flexible, controllable, simple and easy to learn, technical level requirements on the developer are reduced, universality is high, and development efficiency of the service application is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main structure of an application development system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the primary structure of a business application development component in accordance with an embodiment of the present invention;
FIG. 3 is a schematic diagram of the main structure of an application execution container according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of the main flow of an application development method according to an embodiment of the invention;
FIG. 5 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 7 is a schematic diagram of a computer system suitable for use in implementing an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of the main structure of an application development system according to an embodiment of the present invention, and as shown in fig. 1, the application development system 100 includes: an application programming interface 101, a business application development component 102, and a business application generation component 103; wherein,
the application programming interface 101 is configured to define data input, data output, and data processing logic corresponding to a service application.
The service application development component 102 is configured to obtain input data and process the input data by parsing a definition of the application programming interface.
The service application generating component 103 is configured to define an execution start point and an execution end point corresponding to a service application, and generate the service application by assembling the application programming interface and the service application development component.
Specifically, the application programming interface (Application Program Interface, API) in this embodiment is a declarative application programming interface, that is, in the development process of any business application, only a defined description corresponding to a data input (view), a data output (result) and a data processing logic (batch) needs to be submitted for the business.
More specifically, the application programming interface includes: an input data definition interface, an output data definition interface, and a data processing logic definition interface. The input data definition interface is used for defining a service application data source and an acquisition mode. In addition, the input data definition interface can be used for defining a data splitting rule for splitting massive input data so as to perform distributed processing on a plurality of split input data to improve the processing efficiency of the massive data. The data output data definition interface is used for defining a storage target and a storage mode of the business application for storing the output result. The data processing logic definition interface is used for defining data processing logic of the business application, and the data processing logic indicates business logic operation required to be performed on each piece of data and a processing process for generating a corresponding result. The application programming interface may further include an execution result return value definition interface for defining a business application execution result state. The data processing logic definition interface may further include a service exception definition interface, where the service exception definition interface is configured to feed back service exception conditions found during the data processing logic definition process, and to provide recovery operations when an exception occurs.
The service application generating component refers to a service logic execution entry, and indicates an execution start point and an execution end point of the service application, wherein the execution start point refers to a trigger condition for the service application to start executing the data processing logic, and the execution end point refers to an end condition for the service application to stop executing the data processing logic.
Further, the data input and data output may be adapted to a variety of data storage systems, such as databases, files, message queues, memory data structures, for reading or writing input data and output results. The data processing logic is data-driven business processing logic oriented to transaction by transaction, and is convenient for business application to process the data again according to the definition of data input, data output and data processing logic when the data processing is abnormal, namely, the business recovery or retry function is realized.
On this basis, referring to fig. 2, the embodiment of the present invention further provides a schematic diagram of a main structure of a service application development component, where the service application development component 102 includes: an external data reader 1021, and a processing logic executor 1022; wherein the external data reader 1021 is configured to obtain input data by parsing a data input definition; the processing logic executor 1022 is configured to process the input data by parsing a data processing logic definition, so as to obtain an output result corresponding to the input data.
Specifically, an external data reader (ExtendedData) obtains input data by parsing data inputs defined by the aforementioned declarative application programming interface to read data from a data source. Processing logic executorBatchRunner) The data processing logic may be supported either in parallel or in serial form or in batch or stream form.
That is, the business application developed in the embodiment of the present invention is composed of components such as an application programming interface, an external data reader, a processing logic executor, a generating component, and the like. Based on the method, the lightweight development of the business application is realized, the business application is convenient and quick, and the business application can realize various big data processing modes such as batch processing, stream processing and the like.
Still referring to fig. 1, the application development system 100 in the embodiment of the present invention further includes: an application run container 104; the application running container 104 is configured to run the service application to process the input data.
It can be understood that, in order to simplify the development procedure of the service application and improve the development efficiency of the service application, the service application developed in this embodiment is formed by components such as an application programming interface, an external data reader, a processing logic executor, a generating component, and the like, and does not include various components or resources that need to be relied on in the actual execution process, so that an application running container in the service application development system needs to be relied on to run, that is, the service application of the resource or the functional component included in the application running container can be called to realize efficient processing of massive input data.
Specifically, referring to fig. 3, the application running container 104 includes: an application loader 1041, a compute thread pool 1042, and a data cache pool 1043; the application loader 1041 is configured to read and parse a component of the service application; the computing thread pool 1042 is used for supporting the processing logic executor to process the input data; the data buffer pool 1043 is configured to buffer the input data and an output result corresponding to the input data.
More specifically, the data buffer pool is configured to buffer the input data and an output result corresponding to the input data by using a data queue. The data queue in the embodiment of the invention can be a locked data queue or an unlocked data queue. Therefore, through data caching, separation of data reading, data writing and data processing is realized, and I/O waiting time in the data reading and data processing process is greatly shortened. In order to further save the waiting time of the CPU in the data processing process and improve the data processing efficiency, the priority of adopting the lock-free data queue is considered. Further, the lock-free algorithm based on the CAS principle achieves access security of threaded data. The CAS is an atomic instruction of a CPU, and is used for judging whether the value of a certain position of the memory is an expected value or not, if so, the CAS is updated to be a new value, and the consistency of the data is ensured.
Further, the processing logic executor comprises one or more execution units, and the execution units are used for processing the input data.
In order to improve the processing efficiency of mass data, a distributed operation mode is considered, that is, a plurality of execution units in a logic executor are adopted to process input data at the same time. Specifically, under the condition that massive input data needs to be processed, the massive input data can be split into a plurality of parts, the split plurality of parts of input data are independent, namely the split input data can be regarded as input data corresponding to an execution unit, and further corresponding Batchable examples can be respectively constructed based on the split input data, so that different Batchable examples can be respectively operated by different execution units, and further distributed processing of the massive input data is realized.
More specifically, the execution unit performs cooperative processing on the input data by using the following three threads: a data input thread, a business processing thread and a result output thread.
In one embodiment of the present invention, the data input thread is configured to obtain input data according to a data input definition, and push the input data to a first data queue; the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue; and the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
In one embodiment of the present invention, the data input thread is further configured to generate an error information record, push the error information record to the first data queue, and end pushing the input data to the first data queue in case that an error occurs in the input data.
In one embodiment of the present invention, the service processing thread is further configured to return an error prompt message to a call site of the processing logic executor when an error occurs in a process of processing the input data in a case that an error information record is obtained from the first data queue.
In one embodiment of the present invention, the result output thread is further configured to interrupt the data input thread and the service processing thread and return an error prompt message to a call site of the processing logic executor when an error occurs in the output result stored in the second data queue.
Therefore, in the data processing process, under the condition of abnormality, the monitoring of service personnel on the data processing process is realized by timely feeding back error prompt information to the calling part of the processing logic executor, so that the service personnel can process the abnormality in time.
Based on the above embodiment, by means of the method of generating the service application by assembling the application development system comprising the application programming interface, the service application development component and the service application generation component, a developer can develop the service application by only determining data input, data output and data processing logic, so that complete decoupling of service application development and software and hardware technical architecture is realized, dependence of service application development on a specific technical platform and products is avoided, the method is flexible, controllable, simple and easy to learn, technical level requirements on the developer are reduced, universality is high, and development efficiency of the service application is improved. In addition, the high-efficiency processing of mass data can be realized based on the developed business application.
Referring to fig. 4, on the basis of the foregoing embodiment, an embodiment of the present invention provides an application development method based on the foregoing application development system, where the application development method may specifically include the following steps:
step S401, a service application development request is received.
The service application development request refers to any requirement for developing a service application generated according to an actual service requirement, and the service development request indicates information such as a data source to be processed, processing logic (such as screening, sorting, judging, mathematical operation, etc.) to be performed on the data, a result to be output, and a storage mode of the output result.
Step S402, determining an execution start point, an execution end point, data input, data output and data processing logic corresponding to the service application according to the service application development request.
In step S403, the application programming interface is configured to define the data input, data output and data processing logic.
Step S404, configuring a service application development component to enable the service development component to acquire input data and process the input data by analyzing the definition of the application programming interface.
In one embodiment of the present invention, the business application development component includes: an external data reader, a processing logic executor; the external data reader is used for acquiring input data by analyzing data input definition; the processing logic executor is used for processing the input data through analyzing the data processing logic definition so as to acquire an output result corresponding to the input data.
Step S405, configuring a service application generating component to define the execution start point and the execution end point, and generating a service application by assembling the application programming interface and the service application development component.
In one embodiment of the invention, the business application is run by an application run container to process the input data.
In one embodiment of the present invention, the application execution container includes: an application loader, a calculation thread pool and a data cache pool; the application loader is used for reading and analyzing the components of the business application; the computing thread pool is used for supporting the processing logic executor to process the input data; the data buffer pool is used for buffering the input data and the output result corresponding to the input data.
In one embodiment of the present invention, in the data buffer pool, a data queue is used to buffer the input data and the output result corresponding to the input data.
In one embodiment of the present invention, the processing logic comprises one or more execution units, and the execution units are configured to process the input data.
In one embodiment of the present invention, the execution unit performs cooperative processing on the input data using three threads: a data input thread, a business processing thread and a result output thread.
In one embodiment of the present invention, the data input thread is configured to obtain input data according to a data input definition, and push the input data to a first data queue; the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue; and the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
In one embodiment of the present invention, the data input thread is further configured to generate an error information record, push the error information record to the first data queue, and end pushing the input data to the first data queue when an error occurs in the input data.
In one embodiment of the present invention, the service processing thread is further configured to return an error prompt message to a call site of the processing logic executor when an error occurs in a process of processing the input data or when an error record is obtained from the first data queue.
In one embodiment of the present invention, the result output thread is further configured to interrupt the data input thread and the service processing thread and return an error prompt message to a call site of the processing logic executor when an error occurs in the output result stored in the second data queue.
Based on the embodiment, a developer only needs to define data input and data output, data processing logic, an execution starting point, an execution end point and the like corresponding to the service application, can finish application development based on the application development system, is convenient, quick and independently controllable, reduces the technical requirements on the developer, and has good universality; meanwhile, the developed business application can realize the efficient processing of mass data by running in an application running container.
Referring to fig. 5, on the basis of the foregoing embodiment, the embodiment of the present invention provides a data processing method, which is applied to any execution unit in a processing logic executor of a developed service application, so as to implement efficient processing of massive data by the service application, and specifically includes the following steps:
in step S501, the data input thread acquires a data input definition.
It can be understood that, in the case of needing to process massive data, the business application splits the massive data according to a predefined data splitting rule, so that multiple execution units respectively process a plurality of split input data in a distributed manner, and the execution units process the data by adopting three threads, namely a data input thread, a business processing thread and a result output thread.
Specifically, when the processing logic executor is started, the processing logic executor needs to receive the batch instance as a parameter input, and initialize the execution unit, namely start the thread required by the execution unit, so as to ensure the normal operation of the execution unit. More specifically, to ensure the processing performance of the device, the number of initialized execution units may be determined based on the number of CPU threads of the current device, e.g., the number of initialized execution units is 2 times, 3 times, etc., the number of CPU threads.
In step S502, the data input thread generates input data.
The data input thread may obtain a data input definition from the Batchable instance to obtain input data from a data source based on the data input definition, the input data being comprised of a number of pieces of data.
In step S503, the data input thread pushes the input data to the first data queue.
Specifically, the data input thread circularly extracts one piece of data from the input data, and writes the input data into the first data queue one by one for caching until all the input data are extracted. The first data queue is a first-in first-out queue without locking. In the process, if errors occur due to network faults, insufficient access rights, SQL execution failure, absence of files and the like, an error information record containing error codes, error descriptions and the like is generated, the error information record is pushed to a first data queue, and input data pushing to the first data queue is finished.
In step S504, the service processing thread obtains input data from the first data queue.
Therefore, the business processing thread can start data processing by continuously acquiring data from the first data queue without waiting for the data input thread to read all input data, so that separation of data reading and data processing is realized, and I/O waiting time in the data reading process is reduced. If the error information record is extracted from the first data queue, the error prompt information is returned to a call position of the processing logic executor, so that service personnel can process the abnormality in time according to the error prompt information.
In step S505, the service processing thread processes the input data to generate a corresponding output result.
Specifically, the service processing thread circularly extracts input data from the first data queue, and calls back a corresponding output result calculated by the Batchable instance until the input data in the first data queue is processed. If errors occur in the process, error prompt information is returned to a call position of the processing logic executor, so that service personnel can process the anomalies in time according to the error prompt information. In addition, in order to ensure the reliability of the data processing result, the data input thread and the result output thread can be interrupted, and the output result cached in the second data queue can be discarded, so that the data processing can be carried out again later, namely, the recovery capability and retry capability in the case of abnormal data processing are realized.
In step S506, the service processing thread stores the output result in the second data queue.
Therefore, the separation of data processing and data writing is realized by caching the output result into the second data queue, and the I/O waiting time of U in the data processing process is reduced.
In step S507, when the input data in the first data queue is processed, the result output thread saves the output result as a transaction.
Therefore, the output result corresponding to the input data is used as a mechanism for storing the transaction, so that the integrity and consistency of the output result are ensured, namely, the output result is stored successfully at the same time or is stored failed at the same time, partial storage success or failure is avoided, and the data can be processed again conveniently when an abnormality occurs. In addition, in the data storage process, if errors occur, error prompt information is returned to a call position of the processing logic executor, so that service personnel can process the errors in time according to the error prompt information. Meanwhile, in order to avoid resource waste caused by abnormality of subsequent data processing results, the data input thread and the result output thread can be interrupted, so that the data input thread and the result output thread can be restarted when the abnormality is solved.
Based on the embodiment, the distributed processing of the business application on the mass data is realized through the plurality of execution units in the processing logic executor, and the data processing efficiency is improved. The data reading, data processing and data writing separation are realized by adopting the mode of the multithreading cooperative processing of the data input thread, the business processing thread and the result output thread, and the CPU processing performance is improved. Furthermore, by pushing the input data to the first data queue for caching, the service processing thread can start data processing without waiting for all the input data to be acquired, so that I/O time during a large amount of data reading is saved. The output result is stored in the second data queue for caching, and the result output thread is adopted for asynchronous submission, so that the I/O time during a large amount of data processing is saved. In addition, the output result is cached to the second data queue and is integrated into the same transaction for storage when the input data processing is completed, so that the integrity and consistency of the output result are ensured.
On the basis of the above embodiments, an embodiment of the present invention provides an electronic device for application development, including: one or more processors; and storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the methods of application development described above.
Fig. 6 illustrates an exemplary system architecture 600 to which the application development method or application development system of embodiments of the present invention may be applied.
As shown in fig. 6, the system architecture 600 may include terminal devices 601, 602, 603, a network 604, and a server 605. The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 605 via the network 604 using the terminal devices 601, 602, 603 to receive or send messages, etc. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 601, 602, 603.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.
The server 605 may be a server providing various services, such as a background management server providing support for shopping-type websites browsed by the user using the terminal devices 601, 602, 603. The background management server can analyze and other processing on the received data such as the product information inquiry request and the like, and feed back processing results (such as target push information and product information) to the terminal equipment.
It should be noted that, the application development method provided in the embodiment of the present invention is generally executed by the server 605, and accordingly, the application development system is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to include: receiving a service application development request; determining an execution starting point, an execution end point, data input, data output and data processing logic corresponding to the service application according to the service application development request; configuring an application programming interface to define the data input, data output, and data processing logic; configuring a service application development component to enable the service development component to acquire input data and process the input data by analyzing the definition of the application programming interface; and configuring a business application generating component to define the execution starting point and the execution end point, and generating a business application by assembling the application programming interface and the business application development component.
According to the technical scheme of the embodiment of the invention, by means of the mode of generating the service application by assembling an application development system comprising an application programming interface, a service application development component and a service application generation component, a developer can realize the development of the service application by only determining data input, data output and data processing logic, thereby realizing the complete decoupling of the service application development and a software and hardware technical architecture, avoiding the dependence of the service application development on a specific technical platform and products, being flexible, controllable, simple and easy to learn, reducing the technical level requirements on the developer, having high universality and improving the development efficiency of the service application.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. An application development system, comprising: an application programming interface, a business application development component, a business application generation component, and an application running container; wherein,
The application programming interface is used for defining data input, data output and data processing logic corresponding to the business application; wherein the data processing logic is data-driven business processing logic facing to transaction by transaction;
the business application development component is used for acquiring input data and processing the input data by analyzing the definition of the application programming interface;
the business application generating component is used for defining an execution starting point and an execution end point corresponding to the business application and generating the business application by assembling the application programming interface and the business application development component;
the application running container is used for running the business application to process the input data;
wherein the application execution container comprises: a data cache pool; the data buffer pool is used for buffering the input data and the output result corresponding to the input data by adopting a data queue;
the business application development assembly comprises one or more execution units, wherein the execution units are used for processing the input data; the execution unit adopts the following three threads to cooperatively process the input data: a data input thread, a business processing thread and a result output thread; the data input thread is used for acquiring input data according to data input definition and pushing the input data to the first data queue; the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue; and the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
2. The application development system of claim 1, wherein the business application development component comprises: an external data reader, a processing logic executor; wherein,
the external data reader is used for acquiring input data by analyzing data input definition;
the processing logic executor is used for processing the input data through analyzing the data processing logic definition so as to acquire an output result corresponding to the input data.
3. The application development system of claim 2 wherein the application execution container comprises: an application loader, a calculation thread pool and a data cache pool; wherein,
the application loader is used for reading and analyzing the components of the business application;
the computing thread pool is used for supporting the processing logic executor to process the input data.
4. The application development system of claim 3, wherein,
the processing logic executor comprises one or more execution units, and the execution units are used for processing the input data.
5. The application development system of claim 1, wherein,
The data input thread is further configured to generate an error information record, push the error information record to the first data queue, and end pushing the input data to the first data queue when an error occurs in the input data.
6. The application development system of claim 2, wherein,
the service processing thread is further configured to return an error prompt message to a call site of the processing logic executor when an error occurs in a process of processing input data or when an error information record is acquired from the first data queue.
7. The application development system of claim 2, wherein,
and the result output thread is further used for interrupting the data input thread and the service processing thread and returning error prompt information to the calling position of the processing logic executor under the condition that the output result stored in the second data queue is wrong.
8. A business application development method, comprising:
receiving a service application development request;
determining an execution starting point, an execution end point, data input, data output and data processing logic corresponding to the service application according to the service application development request; wherein the data processing logic is data-driven business processing logic facing to transaction by transaction;
Configuring an application programming interface to define the data input, data output, and data processing logic;
configuring a service application development component to enable the service development component to acquire input data and process the input data by analyzing the definition of the application programming interface;
configuring a business application generating component to define the execution starting point and the execution end point, and generating a business application by assembling the application programming interface and the business application development component;
running the business application by using an application running container to process the input data;
wherein the application execution container comprises: a data cache pool; the data buffer pool is used for buffering the input data and the output result corresponding to the input data by adopting a data queue;
the business application development assembly comprises one or more execution units, wherein the execution units are used for processing the input data; the execution unit adopts the following three threads to cooperatively process the input data: a data input thread, a business processing thread and a result output thread; the data input thread is used for acquiring input data according to data input definition and pushing the input data to the first data queue; the business processing thread is used for acquiring input data from the first data queue, processing the input data to generate a corresponding output result, and storing the output result to a second data queue; and the result output thread is used for storing the output result in the second data queue as a transaction under the condition that the input data in the first data queue are processed.
9. An electronic device for application development, comprising:
one or more processors;
storage means for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of claim 8.
10. A computer readable medium on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310111760.9A CN115826934B (en) | 2023-02-09 | 2023-02-09 | Application development system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310111760.9A CN115826934B (en) | 2023-02-09 | 2023-02-09 | Application development system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115826934A CN115826934A (en) | 2023-03-21 |
| CN115826934B true CN115826934B (en) | 2023-05-26 |
Family
ID=85521313
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310111760.9A Active CN115826934B (en) | 2023-02-09 | 2023-02-09 | Application development system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115826934B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10191765B2 (en) * | 2013-11-22 | 2019-01-29 | Sap Se | Transaction commit operations with thread decoupling and grouping of I/O requests |
| CN104932932B (en) * | 2014-03-20 | 2019-08-23 | 腾讯科技（深圳）有限公司 | A kind of business asynchronous processing method, device and system |
| CN106610837A (en) * | 2016-12-26 | 2017-05-03 | 中国建设银行股份有限公司 | Application development method and development platform |
| US10990459B2 (en) * | 2019-08-30 | 2021-04-27 | Chicago Mercantile Exchange Inc. | Distributed threaded streaming platform reader |
| CN114721629A (en) * | 2022-03-25 | 2022-07-08 | 武汉海昌信息技术有限公司 | Development platform system and method based on micro-service and cloud-native |
-
2023
- 2023-02-09 CN CN202310111760.9A patent/CN115826934B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115826934A (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108920222B (en) | Business processing method and device based on rule engine | |
| CN109145023B (en) | Method and apparatus for processing data | |
| CN110795479A (en) | Method and device for distributed ETL scheduling based on data | |
| CN111427701A (en) | Workflow engine system and business processing method | |
| CN109325002B (en) | Text file processing method, device and system, electronic equipment and storage medium | |
| CN113051055A (en) | Task processing method and device | |
| CN107526838B (en) | Method and device for database cluster capacity expansion | |
| CN112445860B (en) | Method and device for processing distributed transaction | |
| CN115826934B (en) | Application development system and method | |
| CN114697398B (en) | Data processing method, device, electronic equipment, storage medium and product | |
| US20230093004A1 (en) | System and method for asynchronous backend processing of expensive command line interface commands | |
| CN114490050A (en) | Data synchronization method and device | |
| CN113918525A (en) | Data exchange scheduling method, system, electronic device, medium, and program product | |
| CN114968636A (en) | Fault processing method and device | |
| CN114168607A (en) | Global serial number generation method, device, equipment, medium and product | |
| CN112783903B (en) | Method and device for generating update log | |
| CN116503005A (en) | Method, device, system and storage medium for dynamically modifying flow | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN113742355A (en) | Method, device, equipment and computer readable medium for updating inventory | |
| CN112860447A (en) | Interaction method and system between different applications | |
| CN112241332A (en) | Interface compensation method and device | |
| CN111767113A (en) | Method and device for realizing container eviction | |
| CN111008202A (en) | Distributed transaction processing method and framework | |
| CN112749042B (en) | Application running method and device | |
| US11656950B2 (en) | Method, electronic device and computer program product for storage management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |