CN116882931B - Purchase, sale and deposit management system and data processing method thereof - Google Patents
Purchase, sale and deposit management system and data processing method thereof Download PDFInfo
- Publication number
- CN116882931B CN116882931B CN202310886746.6A CN202310886746A CN116882931B CN 116882931 B CN116882931 B CN 116882931B CN 202310886746 A CN202310886746 A CN 202310886746A CN 116882931 B CN116882931 B CN 116882931B
- Authority
- CN
- China
- Prior art keywords
- data
- subsystem
- storage
- center
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000003860 storage Methods 0.000 claims abstract description 192
- 230000005540 biological transmission Effects 0.000 claims description 90
- 230000003993 interaction Effects 0.000 claims description 79
- 238000005516 engineering process Methods 0.000 claims description 67
- 238000000034 method Methods 0.000 claims description 60
- 238000012544 monitoring process Methods 0.000 claims description 48
- 230000007246 mechanism Effects 0.000 claims description 46
- 238000007726 management method Methods 0.000 claims description 42
- 230000006870 function Effects 0.000 claims description 38
- 230000001360 synchronised effect Effects 0.000 claims description 36
- 238000012545 processing Methods 0.000 claims description 33
- 230000008569 process Effects 0.000 claims description 30
- 238000004458 analytical method Methods 0.000 claims description 20
- 230000008878 coupling Effects 0.000 claims description 17
- 238000010168 coupling process Methods 0.000 claims description 17
- 238000005859 coupling reaction Methods 0.000 claims description 17
- 238000012986 modification Methods 0.000 claims description 16
- 230000004048 modification Effects 0.000 claims description 16
- 230000010354 integration Effects 0.000 claims description 14
- 238000013523 data management Methods 0.000 claims description 13
- 238000013507 mapping Methods 0.000 claims description 10
- 230000004044 response Effects 0.000 claims description 10
- 238000004140 cleaning Methods 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 8
- 238000013473 artificial intelligence Methods 0.000 claims description 6
- 238000013506 data mapping Methods 0.000 claims description 6
- 238000007418 data mining Methods 0.000 claims description 6
- 238000013500 data storage Methods 0.000 claims description 6
- 238000010801 machine learning Methods 0.000 claims description 6
- 238000005192 partition Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000012423 maintenance Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 description 8
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000003780 insertion Methods 0.000 description 2
- 230000037431 insertion Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Strategic Management (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- General Engineering & Computer Science (AREA)
- Operations Research (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- Marketing (AREA)
- Economics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a purchase, sale and storage management system and a data processing method thereof.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a purchase, sale and storage management system and a data processing method thereof.
Background
The purchase, sales and inventory management system is an important component of an ERP (Enterprise resource planning) system, and mainly relates to three core processes of purchase, sales and inventory of enterprises and integrates the three core processes. Along with the development of enterprises and the accumulation of business data, the existing purchase, sale and storage management system faces the problem of huge mass data management hysteresis, and the problems of incapacity, low efficiency and the like are all presented in the aspects of data sharing, synchronization, storage and the like.
Disclosure of Invention
Based on the above problems, the invention provides a purchase, sale and storage management system and a data processing method thereof, wherein a unified data service interface is developed through constructing a data center and a data interaction bus to synchronize partial key data databases, and the data interaction specification/protocol and system reconstruction if necessary can realize intelligent and efficient fusion, synchronization and intercommunication of data among the purchase, sale and storage subsystems, optimize the purchase, sale and storage business process for a long time and improve the data operation efficiency and quality of the purchase, sale and storage business process.
In view of this, an aspect of the present invention proposes a purchase-sale-stock management system, including: the system comprises a feeding subsystem, a pin system, a storage subsystem and a first data center;
the first data center is configured to:
collecting, cleaning, storing and managing first data of the purchase, sale and storage system through a data center technology;
establishing a first data interaction bus, and carrying out real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus by adopting a big data flow technology;
establishing a first data service interface, and when the business functions of the other party are required to be called or data are required to be accessed among the entering subsystem, the pin system and the storing subsystem, carrying out business function calling and data acquisition through the first data service interface;
Establishing a local database auxiliary management center, wherein when a first preset data synchronization condition is met, the local database auxiliary management center synchronizes key service data of the access subsystem, the pin system and the storage subsystem in respective local databases by utilizing a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem so as to ensure high-quality and high-efficiency data interaction;
analyzing the first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem.
Optionally, the step of collecting, cleaning, storing and managing the first data of the purchase-sale-storage system by a data center technology, wherein the first data center is configured to:
Acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
establishing a first data standard, a first data dictionary and a first data mapping relation according to the first attribute characteristics of the second data, and carrying out standardized data management and management;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
acquiring first historical operation data for operating/managing the first historical data;
respectively establishing a plurality of first type data open models and a plurality of second type data open models according to the first historical data and the first historical operation data;
Opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
and carrying out operation and maintenance and monitoring on the first data center according to a preset first monitoring model.
Optionally, the step of establishing a first data interaction bus, and performing real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus by adopting a big data flow technology, where the first data center is configured to:
Determining one or more matched first data stream transmission schemes according to first data transmission requirements of the purchase and sale system and first basic characteristics of the first historical data;
a first data stream transmission service is built in the first data center according to the first data stream transmission scheme and is used for receiving data reported by the access subsystem, the pin system and the storage subsystem and sending corresponding data to the access subsystem, the pin system and the storage subsystem in real time;
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, utilizing the partition, batch processing and micro batch functions of the first data stream transmission service to control the data stream flowing process in the purchase-sale-storage system;
and determining fields, updating frequency and transmission contents of data stream interaction through the first data stream transmission service according to the first data interaction specification and protocol.
Optionally, when the service function or the access data of the other party needs to be called mutually among the access subsystem, the pin system and the storage subsystem, the step of calling the service function and acquiring the data is performed through the first data service interface, and the first data center is configured to:
determining a first interface protocol according to the first data interaction specification and protocol;
determining a first interface standard according to subsystem characteristic data of the first interface protocol, the access subsystem, the pin system and the storage subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format;
The access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data;
the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the subsystem internal data service interfaces opened by the access subsystem, the pin system and the storage subsystem, integrates the data and returns the integrated data to a calling party;
the access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or service and perform local service processing;
adding an interface security mechanism for the first data service interface and the subsystem internal data service interface according to the first interface standard;
monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data;
and perfecting and optimizing the first data service interface and the subsystem internal data service interface according to the first interface monitoring data.
Optionally, when the first preset data synchronization condition is met, the local database auxiliary management center uses a database mechanism to synchronize the key service data in the respective local databases of the access subsystem, the pin system and the storage subsystem, respectively, in the step of establishing the local database auxiliary management center:
the local database auxiliary management center is configured to:
identifying the key business data of the entering subsystem, the pin system and the storing subsystem respectively;
respectively analyzing business rules of the access subsystem, the pin system and the storage subsystem, and determining dependency/update relation among the key business data;
determining a plurality of corresponding database synchronization mechanisms according to the subsystem characteristic data and the dependency/update relationship of the entering subsystem, the pin system and the storing subsystem respectively;
obtaining a plurality of synchronous programs according to the database synchronous mechanisms;
deploying and testing the plurality of synchronous programs, checking whether the synchronous result is correct or not, influencing other business processes or not and generating a synchronous checking result or not;
maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
And detecting whether a first preset data synchronization condition is met, and stopping the corresponding database synchronization program in the plurality of synchronization programs to avoid generating repeated operation when the first preset data synchronization condition is not met.
The invention provides a data processing method of a purchase-sale-stock management system, which is applied to the purchase-sale-stock management system, wherein the purchase-sale-stock management system comprises a purchase subsystem, a pin system, a stock subsystem and a first data center, and the data processing method of the purchase-sale-stock management system comprises the following steps:
constructing the first data center, wherein the first data center collects, cleans, stores and manages first data of the purchase, sale and storage system through a data center technology;
establishing a first data interaction bus, wherein the first data center adopts a big data flow technology and performs real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus;
establishing a first data service interface, and when the business functions of the other party are required to be called or data are required to be accessed among the entering subsystem, the pin system and the storing subsystem, carrying out business function calling and data acquisition through the first data service interface;
Establishing a local database auxiliary management center, wherein when a first preset data synchronization condition is met, the local database auxiliary management center synchronizes key service data of the access subsystem, the pin system and the storage subsystem in respective local databases by utilizing a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem so as to ensure high-quality and high-efficiency data interaction;
the first data center analyzes first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem.
Optionally, the step of constructing the first data center, where the first data center collects, cleans, stores and manages the first data of the purchase, sale and storage system through a data center technology includes:
Acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
establishing a first data standard, a first data dictionary and a first data mapping relation according to the first attribute characteristics of the second data, and carrying out standardized data management and management;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
acquiring first historical operation data for operating/managing the first historical data;
respectively establishing a plurality of first type data open models and a plurality of second type data open models according to the first historical data and the first historical operation data;
Opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
and carrying out operation and maintenance and monitoring on the first data center according to a preset first monitoring model.
Optionally, the step of establishing a first data interaction bus, where the first data center uses a big data flow technology and performs real-time data interaction with the access subsystem, the pin system, and the storage subsystem through the first data interaction bus includes:
Determining one or more matched first data stream transmission schemes according to first data transmission requirements of the purchase and sale system and first basic characteristics of the first historical data;
a first data stream transmission service is built in the first data center according to the first data stream transmission scheme and is used for receiving data reported by the access subsystem, the pin system and the storage subsystem and sending corresponding data to the access subsystem, the pin system and the storage subsystem in real time;
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, utilizing the partition, batch processing and micro batch functions of the first data stream transmission service to control the data stream flowing process in the purchase-sale-storage system;
and determining fields, updating frequency and transmission contents of data stream interaction through the first data stream transmission service according to the first data interaction specification and protocol.
Optionally, when the service function or the access data of the other party needs to be called mutually among the access subsystem, the pin system and the storage subsystem, the step of calling the service function and acquiring the data through the first data service interface includes:
determining a first interface protocol according to the first data interaction specification and protocol;
determining a first interface standard according to subsystem characteristic data of the first interface protocol, the access subsystem, the pin system and the storage subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format;
The access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data;
the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the subsystem internal data service interfaces opened by the access subsystem, the pin system and the storage subsystem, integrates the data and returns the integrated data to a calling party;
the access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or service and perform local service processing;
adding an interface security mechanism for the first data service interface and the subsystem internal data service interface according to the first interface standard;
monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data;
and perfecting and optimizing the first data service interface and the subsystem internal data service interface according to the first interface monitoring data.
Optionally, when the first preset data synchronization condition is met, the step of synchronizing the key service data of the access subsystem, the pin system and the storage subsystem in the respective local databases by the local database auxiliary management center using a database mechanism includes:
the local database auxiliary management center respectively identifies the key business data of the access subsystem, the pin system and the storage subsystem;
the local database auxiliary management center respectively analyzes the business rules of the access subsystem, the pin system and the storage subsystem and determines the dependency/update relationship between the key business data;
the local database auxiliary management center determines a plurality of corresponding database synchronization mechanisms according to the subsystem characteristic data of the access subsystem, the pin system and the storage subsystem and the dependence/update relation respectively;
the local database auxiliary management center obtains a plurality of synchronous programs according to the database synchronous mechanisms;
the local database assists the management center to deploy and test the plurality of synchronous programs, check whether the synchronous result is correct or not and influence other business processes or not and generate a synchronous check result or not;
Maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
and detecting whether a first preset data synchronization condition is met, and stopping the corresponding database synchronization program in the plurality of synchronization programs to avoid generating repeated operation when the first preset data synchronization condition is not met.
By adopting the technical proposal of the invention, a unified data service interface is developed by constructing a data center and a data interaction bus, the database synchronization of partial key data is carried out, and the data interaction specification/protocol and the system reconstruction when necessary are added, can realize intelligent and efficient fusion, synchronization and intercommunication of data among the input subsystem, the output subsystem and the storage subsystem, the business process of purchase, sale and storage can be optimized for a long time, and the data operation efficiency and quality of the business process can be improved.
Drawings
FIG. 1 is a schematic block diagram of a stock-in management system provided by one embodiment of the present invention;
FIG. 2 is a flowchart of a data processing method of a purchase-sale-stock management system according to an embodiment of the invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced otherwise than as described herein, and therefore the scope of the present invention is not limited to the specific embodiments disclosed below.
The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
A sales listing management system and a data processing method thereof according to some embodiments of the present invention are described below with reference to fig. 1 to 2.
As shown in fig. 1, one embodiment of the present invention provides a purchase-sale-stock management system, including: the system comprises a feeding subsystem, a pin system, a storage subsystem, a first data center, a security server and a virtual server;
the first data center is configured to:
collecting, cleaning, storing and managing first data of the purchase, sale and storage system through a data center technology; the first data center is used as a central data source, first data of the purchase-sale-storage system firstly arrives at the first data center, and the first data are respectively processed according to respective data processing requirements of each purchase-sale-storage subsystem; establishing and storing a mapping relation for each processed data, and then respectively transmitting the data with the established mapping relation to each subsystem according to different requirements of each subsystem; the three subsystems can also actively acquire required data from the first data center; the scheme can furthest reduce the data dependence and the coupling degree between the systems;
establishing a first data interaction bus, and carrying out real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus by adopting a big data flow technology; the first data center can send the latest data to the three subsystems in real time through the data flow, and the three subsystems can also send the locally updated data to the first data center in real time;
Establishing a first data service interface, and when the business functions of the other party are required to be called or data are required to be accessed among the entering subsystem, the pin system and the storing subsystem, carrying out business function calling and data acquisition through the first data service interface so as to avoid directly relying on the local application system interface of the other party;
when a first preset data synchronization condition is met (for example, when the synchronization mechanism of the first data center and/or the first data interaction bus is abnormal), the local database auxiliary management center synchronizes the key service data of the access subsystem, the pin system and the storage subsystem in the local databases respectively by utilizing a database mechanism; when the first data center or the first data interaction bus does not synchronize data in time, the execution of the business rule can be ensured temporarily through a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem so as to ensure high-quality and high-efficiency data interaction;
Analyzing the first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem; localizing interdependent business logic and data, reducing cross-system calls, which enables each system to run and evolve more independently;
the proposal of the invention develops a unified data service interface by constructing a data center and a data interaction bus, synchronizes partial database of key data, adds data interaction specification/protocol and system reconstruction if necessary, can realize intelligent and efficient fusion, synchronization and intercommunication of data among the input subsystem, the output subsystem and the storage subsystem, the business process of purchase, sale and storage can be optimized for a long time, and the data operation efficiency and quality of the business process can be improved.
It should be noted that the block diagram of the purchase-sale-stock management system shown in fig. 1 is only schematic, and the number of the illustrated modules does not limit the protection scope of the present invention.
In some possible embodiments of the present invention, the step of collecting, cleaning, storing and managing the first data of the pin deposit system by a data center technology, the first data center is configured to:
acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
according to the first attribute characteristics of the second data, a first data standard, a first data dictionary and a first data mapping relation are established, standardized data management and management are carried out, and the method is a basis for effective fusion and utilization of data;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
Acquiring first historical operation data (including but not limited to historical processing data, synchronization data, modification data, historical reporting data of each subsystem, historical subscription data and the like of the first data center on the first historical data) for operating/managing the first historical data;
establishing a plurality of first type data open models (for different internal systems) and a plurality of second type data open models (for different external systems) according to the first historical data and the first historical operation data respectively;
opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
And carrying out operation and monitoring on the first data center according to a preset first monitoring model (so as to ensure high availability and high reliable operation, and quick positioning, solving and repairing can be carried out once faults occur).
By adopting the technical means of the embodiment of the invention, the enterprise data center with the functions of data acquisition, cleaning, storage, analysis, service, system integration and the like can be constructed, which is the basis for realizing the data fusion, optimization and innovation of the purchase-sale-storage system, and can provide powerful support for the data operation and decision of enterprises.
In some possible embodiments of the present invention, the step of establishing a first data interaction bus, and performing real-time data interaction with the access subsystem, the pin system, and the storage subsystem through the first data interaction bus by using a big data flow technology, where the first data center is configured to:
determining one or more matched first data stream transmission schemes (such as distributed stream message queues like Kafka, pulsar, etc. with high throughput and low delay) according to first data transmission requirements (including but not limited to transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc.) of the purchase and sale storage system and first basic characteristics (including but not limited to transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc.) of the first historical data, wherein the first data stream transmission schemes (such as distributed stream message queues like Kafka, pulsar, etc. have the characteristics of high throughput and low delay) are suitable for transmitting a large amount of data between a data center and the system, and can generate the first data stream transmission schemes matched with each according to different requirements of each subsystem because the requirements of transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc. are different in terms of data volume, data response requirements of each subsystem, etc.;
A first data stream transmission service is built in the first data center according to the first data stream transmission scheme, and is used for receiving data reported by the inlet subsystem, the pin system and the storage subsystem, and sending corresponding data to the inlet subsystem, the pin system and the storage subsystem in real time (the first data center can generate the first data stream transmission scheme matched with each subsystem according to different requirements of each subsystem, and the three subsystems are deployed with the corresponding data stream service for reporting and subscribing the data);
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, the data circulation process in the purchase-sale-storage system is controlled by utilizing the partition, batch processing and micro batch functions of the first data flow transmission service so as to improve the transmission efficiency and facilitate the filtering, conversion and arrangement of data;
according to the first data interaction specification and protocol, determining fields, update frequencies and transmission contents for carrying out data stream interaction through the first data stream transmission service so as to ensure consistent understanding of data definition and update mechanism of the two parties;
by the scheme of the embodiment, a data interaction bus based on a big data stream technology can be constructed, real-time and low-delay transmission of mass data between the first data center and the input, output and storage subsystems is realized, the data synchronization efficiency is further improved, and the system coupling degree is reduced.
In some possible embodiments of the present invention, when the service function or access data of the other party needs to be called mutually among the access subsystem, the pin system and the storage subsystem, the step of calling the service function and obtaining the data is performed through the first data service interface, and the first data center is configured to:
Determining a first interface protocol (such as REST API, which is simple and easy to use, supports multiple data formats, and easily realizes cross-language and cross-system call) according to the first data interaction specification and protocol;
determining a first interface standard according to the first interface protocol and subsystem characteristic data of the entering subsystem, the pin system and the storing subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format (the compatibility of the existing interfaces of the three subsystems is required to be met according to the confirmation of the data characteristics, transmission requirements, system architecture characteristics and the like of the entering and storing subsystem);
the access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data;
the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the data service interfaces in the subsystem, which are opened by the access subsystem, the pin system and the storage subsystem, and returns the data to a calling party after integrating the data (the coupling degree among the three systems can be further reduced);
The access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or business service and perform local business processing (interface documents should be checked before calling interfaces to construct correct request parameters);
adding an interface security mechanism (such as request signature, token authentication, etc. to prevent an interface from being called by unauthorized) for the first data service interface and the subsystem internal data service interface according to the first interface standard;
monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data (including indexes such as response time, failure rate, flow limit and the like of an inspection interface, and timely finding out and solving problems);
perfecting and optimizing the first data service interface and the internal data service interface of the subsystem according to the first interface monitoring data so as to ensure that the increasing data intercommunication requirements of three subsystems are met;
by the scheme of the embodiment, a unified data service interface can be constructed, and the intercommunication of service data and service capacity among the purchase, sale and storage three systems is realized, which is a key link for connecting the purchase, sale and storage three subsystems and reducing system coupling.
In some possible embodiments of the present invention, the establishing a local database auxiliary management center, when a first preset data synchronization condition is met, uses a database mechanism to synchronize the key service data in the respective local databases by the access subsystem, the pin system, and the storage subsystem, respectively, in the steps of:
the local database auxiliary management center is configured to:
identifying said critical business data (which directly affect the execution of business rules and data consistency, such as order master, commodity master, customer master, etc.) of said entry subsystem, said pin system, said storage subsystem, respectively;
the business rules of the entering subsystem, the pin system and the storing subsystem are respectively analyzed to determine the dependency/update relation between the key business data (such as the order commodity list is required to be synchronously updated when the order is generated, the customer information of the historical order is required to be synchronously updated when the customer information is changed, and the like);
determining a plurality of corresponding database synchronization mechanisms, such as commonly used triggers, storage processes, custom functions and the like, according to the subsystem characteristic data of the entering subsystem, the pin system, the storage subsystem and the dependence/update relation respectively; a trigger: automatically executing an SQL procedure before or after data insertion, update and deletion; data update synchronization across tables can be achieved; the storage process comprises the following steps: a precompiled SQL statement block can contain logic control flow, and is more suitable for complex synchronous logic realization; custom functions: the method is used for updating other table data in batches, and synchronization is realized through function call during main table updating;
Obtaining a plurality of synchronous programs (synchronous triggers, storage processes or functions can be written according to the dependence/update relation determined by the business rules to realize the synchronous update of data between the main table and the slave table) according to the plurality of database synchronous mechanisms;
deploying and testing the plurality of synchronous programs, checking whether the synchronous result is correct or not, influencing other business processes or not and generating a synchronous checking result or not;
maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
and detecting whether a first preset data synchronization condition is met, and stopping corresponding database synchronization programs in the plurality of synchronization programs to avoid repeated operation when the first preset data synchronization condition is not met (after the synchronization mechanism of the first data center and/or the first data interaction bus is repaired).
By means of the scheme of the embodiment, the synchronization of local key data of the purchase, sale and storage three systems under a simple service scene can be achieved by utilizing a mechanism of the relational database, a short board with a first data center or a large data bus synchronization delay can be made up temporarily, and execution of important service rules is ensured.
Referring to fig. 2, another embodiment of the present invention provides a data processing method of a purchase-sale-stock management system, which is applied to a purchase-sale-stock management system, the purchase, sale and deposit management system comprises a purchase subsystem, a pin system, a deposit subsystem, a first data center, a security server and a virtual server, and the data processing method of the purchase, sale and deposit management system comprises the following steps:
Constructing the first data center, wherein the first data center collects, cleans, stores and manages first data of the purchase, sale and storage system through a data center technology; the first data center is used as a central data source, first data of the purchase-sale-storage system firstly arrives at the first data center, and the first data are respectively processed according to respective data processing requirements of each purchase-sale-storage subsystem; establishing and storing a mapping relation for each processed data, and then respectively transmitting the data with the established mapping relation to each subsystem according to different requirements of each subsystem; the three subsystems can also actively acquire required data from the first data center; the scheme can furthest reduce the data dependence and the coupling degree between the systems;
establishing a first data interaction bus, wherein the first data center adopts a big data flow technology and performs real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus; the first data center can send the latest data to the three subsystems in real time through the data flow, and the three subsystems can also send the locally updated data to the first data center in real time;
Establishing a first data service interface, and when the business functions of the other party are required to be called or data are required to be accessed among the entering subsystem, the pin system and the storing subsystem, carrying out business function calling and data acquisition through the first data service interface so as to avoid directly relying on the local application system interface of the other party;
when a first preset data synchronization condition is met (for example, when the synchronization mechanism of the first data center and/or the first data interaction bus is abnormal), the local database auxiliary management center synchronizes the key service data of the access subsystem, the pin system and the storage subsystem in the local databases respectively by utilizing a database mechanism; when the first data center or the first data interaction bus does not synchronize data in time, the execution of the business rule can be ensured temporarily through a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem so as to ensure high-quality and high-efficiency data interaction;
The first data center analyzes first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem; localizing interdependent business logic and data, reducing cross-system calls, which enables each system to run and evolve more independently;
the proposal of the invention develops a unified data service interface by constructing a data center and a data interaction bus, synchronizes partial database of key data, adds data interaction specification/protocol and system reconstruction if necessary, can realize intelligent and efficient fusion, synchronization and intercommunication of data among the input subsystem, the output subsystem and the storage subsystem, the business process of purchase, sale and storage can be optimized for a long time, and the data operation efficiency and quality of the business process can be improved.
In some possible embodiments of the present invention, the step of constructing the first data center, where the first data center collects, cleans, stores and manages the first data of the purchase-sale-storage system through a data center technology includes:
Acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
according to the first attribute characteristics of the second data, a first data standard, a first data dictionary and a first data mapping relation are established, standardized data management and management are carried out, and the method is a basis for effective fusion and utilization of data;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
acquiring first historical operation data (including but not limited to historical processing data, synchronization data, modification data, historical reporting data of each subsystem, historical subscription data and the like of the first data center on the first historical data) for operating/managing the first historical data;
Establishing a plurality of first type data open models (for different internal systems) and a plurality of second type data open models (for different external systems) according to the first historical data and the first historical operation data respectively;
opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
and carrying out operation and monitoring on the first data center according to a preset first monitoring model (so as to ensure high availability and high reliable operation, and quick positioning, solving and repairing can be carried out once faults occur).
By adopting the technical means of the embodiment of the invention, the enterprise data center with the functions of data acquisition, cleaning, storage, analysis, service, system integration and the like can be constructed, which is the basis for realizing the data fusion, optimization and innovation of the purchase-sale-storage system, and can provide powerful support for the data operation and decision of enterprises.
In some possible embodiments of the present invention, the step of establishing a first data interaction bus, where the first data center uses a big data stream technology and performs real-time data interaction with the access subsystem, the pin system, and the storage subsystem through the first data interaction bus includes:
determining one or more matched first data stream transmission schemes (such as distributed stream message queues like Kafka, pulsar, etc. with high throughput and low delay) according to first data transmission requirements (including but not limited to transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc.) of the purchase and sale storage system and first basic characteristics (including but not limited to transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc.) of the first historical data, wherein the first data stream transmission schemes (such as distributed stream message queues like Kafka, pulsar, etc. have the characteristics of high throughput and low delay) are suitable for transmitting a large amount of data between a data center and the system, and can generate the first data stream transmission schemes matched with each according to different requirements of each subsystem because the requirements of transmission protocol, throughput, data transmission rate, bit error rate, channel capacity, time delay, etc. are different in terms of data volume, data response requirements of each subsystem, etc.;
A first data stream transmission service is built in the first data center according to the first data stream transmission scheme, and is used for receiving data reported by the inlet subsystem, the pin system and the storage subsystem, and sending corresponding data to the inlet subsystem, the pin system and the storage subsystem in real time (the first data center can generate the first data stream transmission scheme matched with each subsystem according to different requirements of each subsystem, and the three subsystems are deployed with the corresponding data stream service for reporting and subscribing the data);
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, the data circulation process in the purchase-sale-storage system is controlled by utilizing the partition, batch processing and micro batch functions of the first data flow transmission service so as to improve the transmission efficiency and facilitate the filtering, conversion and arrangement of data;
according to the first data interaction specification and protocol, determining fields, update frequencies and transmission contents for carrying out data stream interaction through the first data stream transmission service so as to ensure consistent understanding of data definition and update mechanism of the two parties;
by the scheme of the embodiment, a data interaction bus based on a big data stream technology can be constructed, real-time and low-delay transmission of mass data between the first data center and the input, output and storage subsystems is realized, the data synchronization efficiency is further improved, and the system coupling degree is reduced.
In some possible embodiments of the present invention, when the service function or access data of the other party needs to be called mutually among the access subsystem, the pin system and the storage subsystem, the step of calling the service function and obtaining the data through the first data service interface includes:
Determining a first interface protocol (such as REST API, which is simple and easy to use, supports multiple data formats, and easily realizes cross-language and cross-system call) according to the first data interaction specification and protocol;
determining a first interface standard according to the first interface protocol and subsystem characteristic data of the entering subsystem, the pin system and the storing subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format (the compatibility of the existing interfaces of the three subsystems is required to be met according to the confirmation of the data characteristics, transmission requirements, system architecture characteristics and the like of the entering and storing subsystem);
the access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data;
the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the data service interfaces in the subsystem, which are opened by the access subsystem, the pin system and the storage subsystem, and returns the data to a calling party after integrating the data (the coupling degree among the three systems can be further reduced);
The access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or business service and perform local business processing (interface documents should be checked before calling interfaces to construct correct request parameters);
adding an interface security mechanism (such as request signature, token authentication, etc. to prevent an interface from being called by unauthorized) for the first data service interface and the subsystem internal data service interface according to the first interface standard;
monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data (including indexes such as response time, failure rate, flow limit and the like of an inspection interface, and timely finding out and solving problems);
perfecting and optimizing the first data service interface and the internal data service interface of the subsystem according to the first interface monitoring data so as to ensure that the increasing data intercommunication requirements of three subsystems are met;
by the scheme of the embodiment, a unified data service interface can be constructed, and the intercommunication of service data and service capacity among the purchase, sale and storage three systems is realized, which is a key link for connecting the purchase, sale and storage three subsystems and reducing system coupling.
In some possible embodiments of the present invention, the step of establishing a local database auxiliary management center, when a first preset data synchronization condition is met, the local database auxiliary management center uses a database mechanism to synchronize key service data of the access subsystem, the pin system, and the storage subsystem in respective local databases, includes:
the local database auxiliary management center respectively identifies the key business data of the entering subsystem, the pin system and the storing subsystem (the data directly affect the execution of business rules and the data consistency, such as an order master table, a commodity master table, a customer master table and the like);
the local database auxiliary management center respectively analyzes the business rules of the entering subsystem, the pin system and the storing subsystem, and determines the dependency/update relation between the key business data (such as the need of synchronously updating an order commodity table when an order is generated, the need of synchronously updating the customer information of historical orders when the customer information is changed, and the like);
the local database auxiliary management center determines a plurality of corresponding database synchronization mechanisms, such as commonly used triggers, storage processes, custom functions and the like, according to the subsystem characteristic data of the access subsystem, the pin system and the storage subsystem and the dependency/update relation respectively; a trigger: automatically executing an SQL procedure before or after data insertion, update and deletion; data update synchronization across tables can be achieved; the storage process comprises the following steps: a precompiled SQL statement block can contain logic control flow, and is more suitable for complex synchronous logic realization; custom functions: the method is used for updating other table data in batches, and synchronization is realized through function call during main table updating;
The local database auxiliary management center obtains a plurality of synchronous programs (synchronous triggers, storage processes or functions can be written according to the dependence/update relation determined by the business rules to realize the data synchronous update between the main table and the slave table) according to the database synchronous mechanisms;
the local database assists the management center to deploy and test the plurality of synchronous programs, check whether the synchronous result is correct or not and influence other business processes or not and generate a synchronous check result or not;
maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
and detecting whether a first preset data synchronization condition is met, and stopping corresponding database synchronization programs in the plurality of synchronization programs to avoid repeated operation when the first preset data synchronization condition is not met (after the synchronization mechanism of the first data center and/or the first data interaction bus is repaired).
By means of the scheme of the embodiment, the synchronization of local key data of the purchase, sale and storage three systems under a simple service scene can be achieved by utilizing a mechanism of the relational database, a short board with a first data center or a large data bus synchronization delay can be made up temporarily, and execution of important service rules is ensured.
It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, such as the above-described division of units, merely a division of logic functions, and there may be additional manners of dividing in actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, or may be in electrical or other forms.
The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a memory, including several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned method of the various embodiments of the present application. And the aforementioned memory includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Those of ordinary skill in the art will appreciate that all or a portion of the steps in the various methods of the above embodiments may be implemented by a program that instructs associated hardware, and the program may be stored in a computer readable memory, which may include: flash disk, read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic disk or optical disk.
The foregoing has outlined rather broadly the more detailed description of embodiments of the present application, wherein specific examples are provided herein to illustrate the principles and embodiments of the present application, the above examples being provided solely to assist in the understanding of the methods of the present application and the core ideas thereof; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Although the present invention is disclosed above, the present invention is not limited thereto. Variations and modifications, including combinations of the different functions and implementation steps, as well as embodiments of the software and hardware, may be readily apparent to those skilled in the art without departing from the spirit and scope of the invention.
Claims (8)
1. A purchase-sale-stock management system, comprising: the system comprises a feeding subsystem, a pin system, a storage subsystem and a first data center;
the first data center is configured to:
collecting, cleaning, storing and managing first data of the purchase, sale and storage system through a data center technology;
establishing a first data interaction bus, and carrying out real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus by adopting a big data flow technology;
establishing a first data service interface, and when the business functions of the other party or the access data are required to be called among the entering subsystem, the pin system and the storing subsystem, calling the business functions and acquiring the data through the first data service interface, wherein the method specifically comprises the following steps: determining a first interface protocol according to the first data interaction specification and protocol; determining a first interface standard according to subsystem characteristic data of the first interface protocol, the access subsystem, the pin system and the storage subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format; the access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data; the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the subsystem internal data service interfaces opened by the access subsystem, the pin system and the storage subsystem, integrates the data and returns the integrated data to a calling party; the access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or service and perform local service processing; adding an interface security mechanism for the first data service interface and the subsystem internal data service interface according to the first interface standard; monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data; perfecting and optimizing the first data service interface and the subsystem internal data service interface according to the first interface monitoring data;
Establishing a local database auxiliary management center, wherein when a first preset data synchronization condition is met, the local database auxiliary management center synchronizes key service data of the access subsystem, the pin system and the storage subsystem in respective local databases by utilizing a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem;
analyzing the first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem.
2. The in-stock and out-of-stock management system of claim 1, wherein the step of collecting, cleaning, storing, and managing first data of the in-stock and out-of-stock system by a data center technology, the first data center configured to:
Acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
establishing a first data standard, a first data dictionary and a first data mapping relation according to the first attribute characteristics of the second data, and carrying out standardized data management and management;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
acquiring first historical operation data for operating/managing the first historical data;
respectively establishing a plurality of first type data open models and a plurality of second type data open models according to the first historical data and the first historical operation data;
Opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
and carrying out operation and maintenance and monitoring on the first data center according to a preset first monitoring model.
3. The inventory management system of claim 2, wherein the step of establishing a first data interaction bus, using large data flow techniques, and interacting real-time data with the inventory subsystem, the pin system, and the inventory subsystem via the first data interaction bus, the first data center is configured to:
Determining one or more matched first data stream transmission schemes according to first data transmission requirements of the purchase and sale system and first basic characteristics of the first historical data;
a first data stream transmission service is built in the first data center according to the first data stream transmission scheme and is used for receiving data reported by the access subsystem, the pin system and the storage subsystem and sending corresponding data to the access subsystem, the pin system and the storage subsystem in real time;
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, utilizing the partition, batch processing and micro batch functions of the first data stream transmission service to control the data stream flowing process in the purchase-sale-storage system;
and determining fields, updating frequency and transmission contents of data stream interaction through the first data stream transmission service according to the first data interaction specification and protocol.
4. The purchase-sale-stock-management system of claim 3, wherein the establishing a local database auxiliary management center uses a database mechanism to synchronize the critical business data in the respective local databases when the first preset data synchronization condition is satisfied, wherein the step of synchronizing the critical business data in the respective local databases comprises:
the local database auxiliary management center is configured to:
identifying the key business data of the entering subsystem, the pin system and the storing subsystem respectively;
respectively analyzing business rules of the access subsystem, the pin system and the storage subsystem, and determining dependency/update relation among the key business data;
Determining a plurality of corresponding database synchronization mechanisms according to the subsystem characteristic data and the dependency/update relationship of the entering subsystem, the pin system and the storing subsystem respectively;
obtaining a plurality of synchronous programs according to the database synchronous mechanisms;
deploying and testing the plurality of synchronous programs, checking whether the synchronous result is correct or not, influencing other business processes or not and generating a synchronous checking result or not;
maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
and detecting whether a first preset data synchronization condition is met, and stopping the corresponding database synchronization program in the plurality of synchronization programs to avoid generating repeated operation when the first preset data synchronization condition is not met.
5. The data processing method of the purchase, sale and storage management system is characterized by being applied to the purchase, sale and storage management system, wherein the purchase, sale and storage management system comprises a purchase subsystem, a pin system, a storage subsystem and a first data center, and the data processing method of the purchase, sale and storage management system comprises the following steps:
constructing the first data center, wherein the first data center collects, cleans, stores and manages first data of the purchase, sale and storage system through a data center technology;
Establishing a first data interaction bus, wherein the first data center adopts a big data flow technology and performs real-time data interaction with the access subsystem, the pin system and the storage subsystem through the first data interaction bus;
establishing a first data service interface, and when the business functions of the other party or the access data are required to be called among the entering subsystem, the pin system and the storing subsystem, calling the business functions and acquiring the data through the first data service interface, wherein the method specifically comprises the following steps: determining a first interface protocol according to the first data interaction specification and protocol; determining a first interface standard according to subsystem characteristic data of the first interface protocol, the access subsystem, the pin system and the storage subsystem, wherein the first interface standard comprises an interface address, a request method, parameters and a response data format; the access subsystem, the pin system and the storage subsystem develop an internal data service interface of the subsystem according to the first interface standard respectively, acquire data by calling internal business service and return the data; the first data center develops the first data service interface according to the first interface standard, and the first data service interface acquires data by calling the subsystem internal data service interfaces opened by the access subsystem, the pin system and the storage subsystem, integrates the data and returns the integrated data to a calling party; the access subsystem, the pin system and the storage subsystem call an internal data service interface of the subsystem to acquire data or service and perform local service processing; adding an interface security mechanism for the first data service interface and the subsystem internal data service interface according to the first interface standard; monitoring the performance and calling conditions of the first data service interface and the internal data service interface of the subsystem to obtain first interface monitoring data; perfecting and optimizing the first data service interface and the subsystem internal data service interface according to the first interface monitoring data;
Establishing a local database auxiliary management center, wherein when a first preset data synchronization condition is met, the local database auxiliary management center synchronizes key service data of the access subsystem, the pin system and the storage subsystem in respective local databases by utilizing a database mechanism;
formulating a first data interaction specification and protocol, and determining the frequency, interface specification and field mapping rule of data communication among the access subsystem, the pin system and the storage subsystem;
the first data center analyzes first historical data of the feeding subsystem, the pin system and the storage subsystem to obtain a first analysis result;
when the first analysis result accords with a first preset modification condition, respectively carrying out system transformation and reconstruction on the feeding subsystem and/or the pin system and/or the storage subsystem so as to reduce the coupling degree among the feeding subsystem, the pin system and the storage subsystem.
6. The method for processing data of a purchase-sale-stock management system according to claim 5, wherein said constructing said first data center, said first data center collecting, cleaning, storing and managing first data of said purchase-sale-stock system by a data center technology comprises:
Acquiring the first data from the purchase-sale-storage system in real time or off-line by adopting a data warehouse technology, a message queue technology and a big data stream technology;
performing dirty data filtering, outlier restoration and standardization processing on the first data by using a first data processing model trained in advance through a machine learning technology to obtain second data;
adopting a relational database, a non-relational database, a data lake and a data warehouse technology to realize multi-layer data storage for the second data according to data types and service requirements;
establishing a first data standard, a first data dictionary and a first data mapping relation according to the first attribute characteristics of the second data, and carrying out standardized data management and management;
configuring a first user identity authentication mechanism and a first data authority scheme for preventing illegal access/tampering to the second data to enhance security management and monitoring of the first data center;
acquiring first historical operation data for operating/managing the first historical data;
respectively establishing a plurality of first type data open models and a plurality of second type data open models according to the first historical data and the first historical operation data;
Opening the data and services of the first data center to different internal systems and external systems according to the plurality of first type data opening models and the plurality of second type data opening models so as to realize shared exchange of data and creation of value;
OLAP analysis, data mining and predictive modeling are realized in the first data center based on big data and artificial intelligence technology, a first data report and a first predictive early warning report are obtained, and data management scheme modification and system upgrading are carried out on the purchase-sale-storage system according to the first data report and the first predictive early warning report;
adopting various system integration technologies to realize application integration and workflow cooperation among the first data center, the purchase, sale and storage system and other enterprise systems so as to ensure the safety of data flow;
and carrying out operation and maintenance and monitoring on the first data center according to a preset first monitoring model.
7. The method for processing data of a purchase, sale and inventory management system according to claim 6, wherein said establishing a first data interaction bus, said first data center adopting a big data stream technology and performing real-time data interaction with said purchase subsystem, said sale system and said inventory subsystem through said first data interaction bus comprises:
Determining one or more matched first data stream transmission schemes according to first data transmission requirements of the purchase and sale system and first basic characteristics of the first historical data;
a first data stream transmission service is built in the first data center according to the first data stream transmission scheme and is used for receiving data reported by the access subsystem, the pin system and the storage subsystem and sending corresponding data to the access subsystem, the pin system and the storage subsystem in real time;
the data reporting modes of the entering subsystem, the pin system and the storing subsystem comprise: directly reporting the locally updated data to the first data center through the API of the first data stream transmission service; reporting an update log of a local database to the first data center through the first data stream transmission service; adopting an incremental data acquisition method to periodically push newly added and updated data to the first data center through the first data stream transmission service;
the first data center pushes cleaned and processed data to the data flow services of the access subsystem, the pin system and the storage subsystem in real time through the first data flow transmission service, and the access subsystem, the pin system and the storage subsystem receive the data and synchronously update the local system;
The first data center monitors the data flow of the purchase, sale and storage system in real time to obtain first real-time monitoring data;
according to the first real-time monitoring data, utilizing the partition, batch processing and micro batch functions of the first data stream transmission service to control the data stream flowing process in the purchase-sale-storage system;
and determining fields, updating frequency and transmission contents of data stream interaction through the first data stream transmission service according to the first data interaction specification and protocol.
8. The method for processing data of a purchase, sale and sale storage management system according to claim 7, wherein said establishing a local database auxiliary management center, when a first preset data synchronization condition is satisfied, uses a database mechanism to synchronize key service data of said purchase subsystem, said sale system and said storage subsystem in respective local databases, comprises:
the local database auxiliary management center respectively identifies the key business data of the access subsystem, the pin system and the storage subsystem;
the local database auxiliary management center respectively analyzes the business rules of the access subsystem, the pin system and the storage subsystem and determines the dependency/update relationship between the key business data;
The local database auxiliary management center determines a plurality of corresponding database synchronization mechanisms according to the subsystem characteristic data of the access subsystem, the pin system and the storage subsystem and the dependence/update relation respectively;
the local database auxiliary management center obtains a plurality of synchronous programs according to the database synchronous mechanisms;
the local database assists the management center to deploy and test the plurality of synchronous programs, check whether the synchronous result is correct or not and influence other business processes or not and generate a synchronous check result or not;
maintaining and optimizing the plurality of synchronization programs according to the synchronization checking result to ensure normal operation and synchronize the latest business rules;
and detecting whether a first preset data synchronization condition is met, and stopping the corresponding database synchronization program in the plurality of synchronization programs to avoid generating repeated operation when the first preset data synchronization condition is not met.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310886746.6A CN116882931B (en) | 2023-07-18 | 2023-07-18 | Purchase, sale and deposit management system and data processing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310886746.6A CN116882931B (en) | 2023-07-18 | 2023-07-18 | Purchase, sale and deposit management system and data processing method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116882931A CN116882931A (en) | 2023-10-13 |
| CN116882931B true CN116882931B (en) | 2024-03-19 |
Family
ID=88265957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310886746.6A Active CN116882931B (en) | 2023-07-18 | 2023-07-18 | Purchase, sale and deposit management system and data processing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116882931B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0022858D0 (en) * | 2000-09-18 | 2000-11-01 | Oracle Corp | Transaction reconstruction |
| EP1507226A1 (en) * | 2003-08-13 | 2005-02-16 | Accenture Global Services GmbH | Fraud detection method and system |
| CN113849483A (en) * | 2021-09-29 | 2021-12-28 | 中国兵器装备集团自动化研究所有限公司 | Real-time database system architecture for intelligent factory |
| CN113991874A (en) * | 2021-12-27 | 2022-01-28 | 国家电网有限公司客户服务中心 | Power grid APP operation support system and method based on middle platform |
| CN115408004A (en) * | 2022-07-29 | 2022-11-29 | 清华大学 | Web application system storage and calculation integrated adaptation optimization method and device |
| CN115907439A (en) * | 2023-03-01 | 2023-04-04 | 广东赛博威信息科技有限公司 | Integrated marketing management system |
-
2023
- 2023-07-18 CN CN202310886746.6A patent/CN116882931B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0022858D0 (en) * | 2000-09-18 | 2000-11-01 | Oracle Corp | Transaction reconstruction |
| EP1507226A1 (en) * | 2003-08-13 | 2005-02-16 | Accenture Global Services GmbH | Fraud detection method and system |
| CN113849483A (en) * | 2021-09-29 | 2021-12-28 | 中国兵器装备集团自动化研究所有限公司 | Real-time database system architecture for intelligent factory |
| CN113991874A (en) * | 2021-12-27 | 2022-01-28 | 国家电网有限公司客户服务中心 | Power grid APP operation support system and method based on middle platform |
| CN115408004A (en) * | 2022-07-29 | 2022-11-29 | 清华大学 | Web application system storage and calculation integrated adaptation optimization method and device |
| CN115907439A (en) * | 2023-03-01 | 2023-04-04 | 广东赛博威信息科技有限公司 | Integrated marketing management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116882931A (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100449548C (en) | Method and system for synchronizing data base | |
| US8938421B2 (en) | Method and a system for synchronizing data | |
| CN103684828B (en) | A kind for the treatment of method and apparatus of telecommunication equipment fault | |
| CN101477543B (en) | System and method for automating ETL application | |
| RU2767149C2 (en) | Method and configuration for automated testing system | |
| CN107818431A (en) | A kind of method and system that order track data is provided | |
| US10817532B2 (en) | Scientific computing process management system | |
| CN101471819A (en) | Test system, test method, management domain and operation domain | |
| CN106878363A (en) | A kind of information processing method, apparatus and system | |
| CN108062231A (en) | A kind of cloud application automatic configuration method based on correlation analysis | |
| CN115374102A (en) | Data processing method and system | |
| CN109902919A (en) | Server assets management method, device, equipment and readable storage medium storing program for executing | |
| CN111538720B (en) | Method and system for cleaning basic data of power industry | |
| CN111181775B (en) | Integrated operation and maintenance management alarm method based on automatic host asset discovery | |
| CN114548833A (en) | Integrated intelligent operation and maintenance control method, system and operation and maintenance platform | |
| CN116882931B (en) | Purchase, sale and deposit management system and data processing method thereof | |
| CN115757552B (en) | Bank historical data management system based on distributed microservice | |
| CN108156061A (en) | Esb monitoring service platforms | |
| CN113886061A (en) | Industry thing networking operation and maintenance platform | |
| US20100156623A1 (en) | Method and Apparatus for Network Service Assurance | |
| CN109754131B (en) | SCD file configuration method and device based on NXD | |
| CN109921963A (en) | A kind of network state method for inspecting and system | |
| Zalewski | Beyond ATAM: Architecture analysis in the development of large scale software systems | |
| CN109726827A (en) | A kind of intelligent operation management method of security system equipment | |
| CN117349384B (en) | Database synchronization method, system and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |