CN116401117A - Data processing method combining stream computing system and traditional software application system - Google Patents
Data processing method combining stream computing system and traditional software application system Download PDFInfo
- Publication number
- CN116401117A CN116401117A CN202310220742.4A CN202310220742A CN116401117A CN 116401117 A CN116401117 A CN 116401117A CN 202310220742 A CN202310220742 A CN 202310220742A CN 116401117 A CN116401117 A CN 116401117A
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- software application
- computing system
- topic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 23
- 230000001360 synchronised effect Effects 0.000 claims abstract description 16
- 230000008569 process Effects 0.000 claims abstract description 10
- 238000012544 monitoring process Methods 0.000 claims abstract description 6
- 230000000977 initiatory effect Effects 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 10
- 238000013500 data storage Methods 0.000 claims description 7
- 230000002708 enhancing effect Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 9
- 230000003993 interaction Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a data processing method combining a stream computing system and a traditional software application system, which comprises the following steps: determining a message queue configuration; starting an asynchronous thread in the background by other systems, and continuously monitoring a message queue; initiating a request to other systems, acquiring data and writing the data into a message queue; the streaming system monitors the message queue, processes and returns data to the message queue; executing synchronous or asynchronous query logic according to the request type; and returning the result, and entering other subsequent programs. The data processing method, the data processing equipment and the computer readable storage medium which are combined by the stream computing system and the traditional software application system provided by the invention are used for enhancing the joint processing of the complex rule downstream system in the aspects of expansibility, stability, timeliness and the like of real-time computation.
Description
Technical Field
The present invention relates to the field of data processing technology, and in particular, to a data processing method, apparatus and computer readable storage medium for combining a stream computing system with a conventional software application system.
Background
The description of the background art to which the present invention pertains is merely for illustrating and facilitating understanding of the summary of the invention, and should not be construed as an explicit recognition or presumption by the applicant that the applicant regards the prior art as the filing date of the first filed application.
In recent years, the degree of social informatization has been increasing, and the data volume has become huge, and many demands have been derived therefrom. Taking the financial industry as an example, the financial industry data not only has a situation of sharply increasing data volume, but also has diversified requirements for data analysis, and the importance of stream processing technology is also more and more prominent. With the widespread use of streaming technology, the following problems often occur, how are the data processed out of the stream used? How are other systems to acquire this data and use it for subsequent decision making, analysis? This link brings great trouble to development and use personnel, one of which is: how does data of multiple stream processing services do traffic isolation, avoid mixing together? And two,: how can other systems not handle data due to network problems, etc.? And thirdly,: how do other systems do different interactions due to different request patterns? The complex scenes bring great management difficulty to enterprise management staff, and meanwhile, the complex scenes contain extremely valuable customer data, so that extremely important roles are played in aspects of financial enterprises in analyzing customers, avoiding risks, timely finding abnormal financial behaviors and the like, financial customers can be helped to make timely and accurate decision responses to emergency situations, and in recent years, the system and the method for carrying out real-time calculation on financial data provide extremely important roles in improving the fraud early warning capability of the financial enterprises, and further improve the capability of the financial enterprises in fraud early warning.
The concept of "real-time computing" arose from research in computer science into computer hardware and computer software systems that are subject to "real-time constraints," such as the longest time limit between the occurrence of an event to the response of the system. Real-time procedures must ensure that responses are made within a strict time. The real-time computing is also widely applied to scenes such as real-time intelligent recommendation, real-time fraud detection, public opinion analysis, complex event processing, real-time machine learning and the like due to research characteristics.
In order to enable a financial manager to acquire stream data and analyze financial data more conveniently, different interactions are required to be executed on financial data generated in real time according to requirements, a bridge for communicating the real-time data with other systems is searched, and real-time data analysis decision-making capability is provided for financial enterprises.
Based on background configuration and message queue, data interaction between stream computing system and traditional application is realized, the background of stream computing system maintains read-write capability, and data obtained from message queue is written into buffer middleware. Depending on the business capability requirements, the main thread may provide different ways of processing. If the main thread is to synchronously return, the main thread continuously reads the cache data; if the cache token is to be returned asynchronously, the cache token is generated to be returned directly for the next access. A disadvantage of the prior art is that the acquisition of stream processing data fluctuates with the network and the synchronization request may occur when no stream processing data is acquired in a short time.
In order to solve the technical problems, the invention provides a data processing method, equipment and a computer readable storage medium for combining a stream computing system and a traditional software application system, which are used for enhancing the joint processing of a complex rule downstream system in the aspects of expansibility, stability, timeliness and the like of real-time computing.
Disclosure of Invention
The invention provides a data processing method, equipment and a computer readable storage medium for combining a stream computing system and a traditional software application system, which are used for enhancing the joint processing of a complex rule downstream system in the aspects of expansibility, stability, timeliness and the like of real-time computation.
An embodiment of the first aspect of the present invention provides a data processing method for combining a stream computing system with a legacy software application system, including the steps of: determining a message queue configuration; starting an asynchronous thread in the background by other systems, and continuously monitoring a message queue; initiating a request to other systems, acquiring data and writing the data into a message queue; the streaming system monitors the message queue, processes and returns data to the message queue; executing synchronous or asynchronous query logic according to the request type; and returning the result, and entering other subsequent programs.
Preferably, the step of determining the message queue configuration specifically includes the following operations: defining message queue configuration information, wherein the configuration information comprises url, topic of data input and topic of data output; topic, which specifies the message queue configuration, guarantees globally unique under the same url.
Preferably, the other systems start an asynchronous thread in the background, and continuously monitor the message queue, including the following operations: when other systems issue functions, an asynchronous thread is generated in the background, and the output topic of the message queue is continuously monitored.
Preferably, the step of initiating a request to the other system, obtaining data and writing the data to the message queue includes the following operations: synchronous or asynchronous requests are initiated to other systems, and data is written into the message queue input topic.
Preferably, the streaming system listens to the message queue, processes and returns data to the message queue, comprising the following operations: the stream processing system continuously monitors the input topic of the message queue, reads the data, performs stream processing, and writes the message into the output topic of the queue after the processing is completed; and monitoring the output topic of the message queue by the background thread of other systems, reading the data, and writing the data into the data storage middleware according to naming rules.
Preferably, the step of executing synchronous or asynchronous query logic, depending on the request type, comprises the following operations: if the request is an asynchronous request, generating a query token to return according to the message queue configuration and the naming rule so as to be used for subsequent query; if the data is synchronous request, the data source middleware is circularly read according to the naming rule, if the data is not read in the limited time, the exception is returned, and if the data is successfully read, the result is returned.
Preferably, the step of returning the result and entering the subsequent other procedure comprises the following operations: and integrating the returned result, and using the processed data for subsequent other programs.
Preferably, the data obtained from the message queue is written to the caching middleware or database.
Embodiments of the second aspect of the present invention also provide a data processing apparatus in combination with a streaming computing system and a legacy software application system, comprising a memory and a processor; wherein the memory is for storing executable program code; the processor is configured to read the executable program code stored in the memory to perform a data processing method in combination with a streaming computing system and a legacy software application system. .
Embodiments of the third aspect of the present invention also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements a data processing method for a streaming computing system in combination with a legacy software application system.
The data processing method, the data processing equipment and the computer readable storage medium for combining the stream computing system and the traditional software application system are used for enhancing the joint processing of the complex rule downstream system in the aspects of expansibility, stability, timeliness and the like of real-time computation.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the invention will become apparent and may be better understood from the following description of embodiments taken in conjunction with the accompanying drawings in which:
FIG. 1 illustrates a flow chart of a data processing method in combination with a streaming computing system and a legacy software application system in accordance with an embodiment of the present invention;
FIG. 2 is a block diagram of one embodiment of a data processing apparatus in combination with a streaming computing system and a legacy software application system of the present specification;
FIG. 3 is a block diagram of one embodiment of a computer readable storage medium of a data processing method in combination with a streaming computing system and a legacy software application system of the present specification.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
The following discussion provides various embodiments of the invention. While each embodiment represents a single combination of the invention, different embodiments of the invention may be substituted or combined, and the invention is thus to be considered to include all possible combinations of the same and/or different embodiments described. Thus, if one embodiment comprises A, B, C and another embodiment comprises a combination of B and D, then the present invention should also be considered to include embodiments comprising one or more of all other possible combinations comprising A, B, C, D, although such an embodiment may not be explicitly recited in the following.
FIG. 1 illustrates a flow chart of a data processing method in combination with a streaming computing system and a legacy software application system in accordance with an embodiment of the present invention. As shown in fig. 1, a data processing method of a stream computing system and a conventional software application system, includes the following steps: step S01, determining message queue configuration; step S02, other systems start an asynchronous thread in the background and monitor the message queue continuously; step S03, a request is initiated to other systems, and data is acquired and written into a message queue; step S04, the stream system monitors the message queue, processes and returns data to the message queue; step S05, executing synchronous or asynchronous query logic according to the request type; step S06, returning the result and entering the subsequent other programs.
In the data processing method combining the stream computing system and the traditional software application system provided by the embodiment of the invention, step S01, the step of determining the configuration of the message queue, comprises the following operations: step S101, defining message queue configuration information, wherein the configuration information comprises url, topic of data input and topic of data output; step S102, defining topic configured by message queue, guaranteeing global uniqueness under the same url, and avoiding confusion of service data. In step S02, the other systems start an asynchronous thread in the background, and continuously monitor the message queue, which includes the following operations: in step S201, when other systems want to issue functions, an asynchronous thread is generated in the background, and the output topic of the message queue is continuously monitored. Step S03, a request is initiated to other systems, and the step of acquiring data and writing the data into a message queue comprises the following operations: in step S301, a synchronous or asynchronous request is initiated to other systems, and a data write message queue is input to topic. Step S04, the flow system monitors the message queue, processes and returns data to the message queue, and comprises the following operations: step S401, the stream processing system continuously monitors the input topic of the message queue, reads the data, performs stream processing, and writes the message into the output topic of the queue after the processing is completed; and step S402, monitoring the output topic of the message queue by the background thread of other systems, reading the data, and writing the data into the data storage middleware according to naming rules. Step S05, executing synchronous or asynchronous query logic according to the request type, wherein the step comprises the following operations: step S501, if the request is an asynchronous request, generating a query token to return according to the message queue configuration and naming rules for subsequent query; step S502, if the data source middleware circularly reads the data according to the naming rule if the data source middleware is a synchronous request, if the data source middleware does not read the data within the limited time, the data source middleware returns an exception, and if the data source middleware successfully reads the data, the data source middleware returns a result. Step S06, returning a result, and entering the steps of other subsequent programs, wherein the steps comprise the following operations: in step S601, the returned results are integrated, and the processed data is used for subsequent other programs.
In the data processing method combining the stream computing system and the traditional software application system provided by the embodiment of the invention, the data obtained from the message queue is written into the cache middleware or the database.
According to the data processing method combining the stream computing system and the traditional software application system, which is provided by the embodiment of the invention, a background starts a thread, continuously reads and writes business data, performs data interaction with a stream platform through a message queue, and realizes data business isolation based on the message queue; depending on the synchronous/asynchronous request mode, different processing modes are provided by means of the caching middleware.
With specific reference to fig. 1, the flow of the data processing method for combining the stream computing system and the conventional software application system provided by the embodiment of the present invention is as follows:
starting;
the common application system reads the message queue configuration (such as Kafka), wherein the background thread continuously operates the configuration link url, inputs topic and outputs topic;
whether the input and output topic is guaranteed to be globally unique under the same url; if not, the message queue configuration is adjusted, and the common application system returns to read the message queue configuration.
The input and output topic guarantees global uniqueness under the same url, and then a common application system writes data into the input topic of Kafka; in the process, a background thread continuously operates a streaming system to monitor Kafka input topic to obtain data for streaming processing; if the processing is unsuccessful, the processing is not performed, and the common application system throws out abnormality when reading; if the processing is successful, writing the result, inputting the output topic of the Kafak into a common application system, starting a background thread to continuously read the output topic data of the Kafak, and continuously operating a write-in data storage middleware (such as Redis) by the background thread.
If the main thread request is a synchronous request, returning a token for inquiring the data storage middleware for subsequent inquiry and ending.
If the main thread request is a synchronous request, the data storage middleware is read according to the circulation, whether the data is read according to a specific naming rule is judged, if not, whether the data is within a limited time is judged, specifically, if the data is within the limited time, the data storage middleware is returned, and the circulation is started; if not, the exception is returned synchronously, and the process is finished.
And judging that the data is read according to the specific naming rule, synchronously returning the result and ending.
FIG. 2 is a block diagram of one embodiment of a data processing apparatus in combination with a streaming computing system and a legacy software application system of the present specification; with reference now to FIG. 2, a block diagram of a data processing apparatus 300 suitable for use in conjunction with a streaming computing system and a legacy software application system for implementing embodiments of the present disclosure is shown. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 2 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 2, the electronic device 300 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 301 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 302 or a program loaded from a storage means 308 into a Random Access Memory (RAM) 303. In the RAM303, various programs and data required for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
In general, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 308 including, for example, magnetic tape, hard disk, etc.; and communication means 309. The communication means 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. While fig. 2 shows an electronic device 300 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing means 301.
FIG. 3 is a block diagram of one embodiment of a computer readable storage medium of a data processing method in combination with a streaming computing system and a legacy software application system of the present specification. As shown in fig. 3, a computer-readable storage medium 40 according to an embodiment of the present disclosure has stored thereon non-transitory computer-readable instructions 41. When the non-transitory computer readable instructions 41 are executed by a processor, all or part of the steps of the data processing method described above in connection with the streaming computing system and legacy software application of the various embodiments of the present disclosure are performed.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: constructing a basic page, wherein the page code of the basic page is used for constructing an environment required by the operation of the service page and/or realizing the same abstract workflow in the similar service scene; constructing one or more page templates, wherein the page templates are used for providing code templates for realizing service functions in service scenes; based on the corresponding page template, generating a final page code of each page of the service scene through code conversion of a specific function of each page of the service scene; and merging the generated final page code of each page into the page code of the basic page to generate the code of the service page.
Alternatively, the computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: constructing a basic page, wherein the page code of the basic page is used for constructing an environment required by the operation of the service page and/or realizing the same abstract workflow in the similar service scene; constructing one or more page templates, wherein the page templates are used for providing code templates for realizing service functions in service scenes; based on the corresponding page template, generating a final page code of each page of the service scene through code conversion of a specific function of each page of the service scene; and merging the generated final page code of each page into the page code of the basic page to generate the code of the service page.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The name of the unit does not in any way constitute a limitation of the unit itself, for example the first acquisition unit may also be described as "unit acquiring at least two internet protocol addresses".
The data processing method, the data processing equipment and the computer readable storage medium which are combined by the stream computing system and the traditional software application system provided by the invention are used for enhancing the joint processing of the complex rule downstream system in the aspects of expansibility, stability, timeliness and the like of real-time computation.
In the present invention, the terms "first," "second," "third," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance; the term "plurality" means two or more, unless expressly defined otherwise. The terms "mounted," "connected," "secured," and the like are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; "coupled" may be directly coupled or indirectly coupled through intermediaries. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In the description of the present invention, it should be understood that the directions or positional relationships indicated by the terms "upper", "lower", etc. are based on the directions or positional relationships shown in the drawings, are merely for convenience of description and simplification of the description, and do not indicate or imply that the apparatus or unit referred to must have a specific direction, be constructed and operated in a specific direction, and therefore, should not be construed as limiting the present invention.
In the description of the present specification, the terms "one embodiment," "some embodiments," "particular embodiments," and the like, mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing is merely illustrative of certain embodiments of the present invention and is not intended to limit the invention so that various modifications and changes may be made to the invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method for a stream computing system in combination with a legacy software application system, comprising the steps of:
determining a message queue configuration;
starting an asynchronous thread in the background by other systems, and continuously monitoring a message queue;
initiating a request to other systems, acquiring data and writing the data into a message queue;
the streaming system monitors the message queue, processes and returns data to the message queue;
executing synchronous or asynchronous query logic according to the request type;
and returning the result, and entering other subsequent programs.
2. The method for processing data in combination with a legacy software application system according to claim 1, wherein the step of determining the message queue configuration comprises the steps of:
defining message queue configuration information, wherein the configuration information comprises url, topic of data input and topic of data output;
topic, which specifies the message queue configuration, guarantees globally unique under the same url.
3. The method of data processing in combination with a streaming computing system and a legacy software application according to claim 2, wherein the step of the other system starting an asynchronous thread in the background and continually listening for a message queue comprises the operations of:
when other systems issue functions, an asynchronous thread is generated in the background, and the output topic of the message queue is continuously monitored.
4. A method of processing data in combination with a legacy software application system according to claim 3, wherein the step of initiating a request to the other system to obtain data to write to the message queue comprises the operations of:
synchronous or asynchronous requests are initiated to other systems, and data is written into the message queue input topic.
5. The method of claim 4, wherein the step of the streaming system listening to the message queue, processing and returning data to the message queue comprises the steps of:
the stream processing system continuously monitors the input topic of the message queue, reads the data, performs stream processing, and writes the message into the output topic of the queue after the processing is completed;
and monitoring the output topic of the message queue by the background thread of other systems, reading the data, and writing the data into the data storage middleware according to naming rules.
6. The method of claim 5, wherein the step of executing synchronous or asynchronous query logic according to the request type comprises the steps of:
if the request is an asynchronous request, generating a query token to return according to the message queue configuration and the naming rule so as to be used for subsequent query;
if the data is synchronous request, the data source middleware is circularly read according to the naming rule, if the data is not read in the limited time, the exception is returned, and if the data is successfully read, the result is returned.
7. The method of claim 6, wherein the step of returning results to a subsequent further program comprises the steps of:
and integrating the returned result, and using the processed data for subsequent other programs.
8. A method of data processing in combination with a streaming computing system and a legacy software application according to any of claims 1 to 7, wherein the data obtained from the message queue is written to a caching middleware or database.
9. A data processing apparatus for a stream computing system in combination with a legacy software application system, comprising a memory and a processor; wherein the memory is for storing executable program code; the processor is configured to read executable program code stored in the memory to perform a data processing method in combination with a streaming computing system according to any of claims 1-8 and a legacy software application.
10. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements a data processing method in combination with a streaming computing system according to any of claims 1-8 and a legacy software application.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310220742.4A CN116401117B (en) | 2023-03-09 | 2023-03-09 | Data processing method combining stream computing system and traditional software application system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310220742.4A CN116401117B (en) | 2023-03-09 | 2023-03-09 | Data processing method combining stream computing system and traditional software application system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116401117A true CN116401117A (en) | 2023-07-07 |
| CN116401117B CN116401117B (en) | 2024-04-09 |
Family
ID=87013269
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310220742.4A Active CN116401117B (en) | 2023-03-09 | 2023-03-09 | Data processing method combining stream computing system and traditional software application system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116401117B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150040140A1 (en) * | 2013-08-01 | 2015-02-05 | International Business Machines Corporation | Consuming Ordered Streams of Messages in a Message Oriented Middleware |
| US20170272516A1 (en) * | 2016-03-17 | 2017-09-21 | International Business Machines Corporation | Providing queueing in a log streaming messaging system |
| CN109271417A (en) * | 2018-11-23 | 2019-01-25 | 福建榕基软件股份有限公司 | A kind of lightweight messages queue implementing method and storage equipment based on database |
| CN109413043A (en) * | 2018-09-25 | 2019-03-01 | 聚好看科技股份有限公司 | Realize method and device, the electronic equipment, storage medium of Database Dynamic configuration |
| CN115221242A (en) * | 2022-06-30 | 2022-10-21 | 济南浪潮数据技术有限公司 | Data synchronization method and device and computer readable storage medium |
| CN115396494A (en) * | 2022-08-19 | 2022-11-25 | 北京银丰新融科技开发有限公司 | Real-time monitoring method and system based on stream computing |
-
2023
- 2023-03-09 CN CN202310220742.4A patent/CN116401117B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150040140A1 (en) * | 2013-08-01 | 2015-02-05 | International Business Machines Corporation | Consuming Ordered Streams of Messages in a Message Oriented Middleware |
| US20170272516A1 (en) * | 2016-03-17 | 2017-09-21 | International Business Machines Corporation | Providing queueing in a log streaming messaging system |
| CN109413043A (en) * | 2018-09-25 | 2019-03-01 | 聚好看科技股份有限公司 | Realize method and device, the electronic equipment, storage medium of Database Dynamic configuration |
| CN109271417A (en) * | 2018-11-23 | 2019-01-25 | 福建榕基软件股份有限公司 | A kind of lightweight messages queue implementing method and storage equipment based on database |
| CN115221242A (en) * | 2022-06-30 | 2022-10-21 | 济南浪潮数据技术有限公司 | Data synchronization method and device and computer readable storage medium |
| CN115396494A (en) * | 2022-08-19 | 2022-11-25 | 北京银丰新融科技开发有限公司 | Real-time monitoring method and system based on stream computing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116401117B (en) | 2024-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382228B (en) | Method and device for outputting information | |
| WO2022135547A1 (en) | Service operation method and apparatus, electronic device, and computer storage medium | |
| CN111857720B (en) | User interface state information generation method and device, electronic equipment and medium | |
| CN111813465B (en) | Information acquisition method, device, medium and equipment | |
| CN112597047A (en) | Test method, test device, electronic equipment and computer readable medium | |
| CN110928732A (en) | Server cluster performance sampling analysis method and device and electronic equipment | |
| WO2020199659A1 (en) | Method and apparatus for determining push priority information | |
| CN111241043A (en) | Multimedia file sharing method, terminal and storage medium | |
| CN112954056B (en) | Method and device for processing monitoring data, electronic equipment and storage medium | |
| CN112291121B (en) | Data processing method and related equipment | |
| CN111241137B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN110489326B (en) | IDS-based HTTPAPI debugging method device, medium and equipment | |
| CN116401117B (en) | Data processing method combining stream computing system and traditional software application system | |
| CN112379967B (en) | Simulator detection method, device, equipment and medium | |
| CN113807056B (en) | Document name sequence error correction method, device and equipment | |
| CN111414161B (en) | Method, device, medium and electronic equipment for generating IDL file | |
| CN111787043A (en) | Data request method and device | |
| CN111461227B (en) | Sample generation method, device, electronic equipment and computer readable medium | |
| CN115062022B (en) | Aircraft manual splitting method, device, electronic equipment and computer readable medium | |
| CN116820539B (en) | System software operation maintenance system and method based on Internet | |
| CN116796000A (en) | Knowledge graph entity fusion method and device based on specified relation type | |
| CN112346630B (en) | State determination method, device, equipment and computer readable medium | |
| CN114613355B (en) | Video processing method and device, readable medium and electronic equipment | |
| CN111581305B (en) | Feature processing method, device, electronic equipment and medium | |
| CN111258670B (en) | Method and device for managing component data, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |