CN108595639A - A kind of production real time mass data processing system based on NoSQL - Google Patents
A kind of production real time mass data processing system based on NoSQL Download PDFInfo
- Publication number
- CN108595639A CN108595639A CN201810380322.1A CN201810380322A CN108595639A CN 108595639 A CN108595639 A CN 108595639A CN 201810380322 A CN201810380322 A CN 201810380322A CN 108595639 A CN108595639 A CN 108595639A
- Authority
- CN
- China
- Prior art keywords
- data
- production
- nosql
- creation
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The production real time mass data processing system based on NoSQL that the invention discloses a kind of, including creation data import modul, creation data sharing module, bulletin policy management module and system management module；The creation data import modul provides data source for system；The creation data sharing module is used to complete the flow of entire data sharing and checking for every production information for ordinary user, and the production includes industry characteristics data, image data, video data and manufacturer's data；The bulletin policy management module is used to be directed to super keepe and common management administrator, completes the management function of production shared data；The system management module is used to complete the management function to system user addition or permission modification for super keepe；NoSQL databases MongoDB is replaced application of the traditional relational data in data sharing platform by present treatment system, the characteristics of using MongoDB without " relationship ", improves the efficiency of data sharing platform data inputting and inquiry.
Description
Technical field
The present invention relates to creation data process field, more particularly at a kind of production real time mass data based on NoSQL
Reason system.
Background technology
Tellurian information increases with surprising rapidity, and according to statistics, average each second has 200 general-purpose families using
Search engine, the thing that Facebook user shares daily are more than 4,000,000,000, and the information content that microblogging is handled daily is more than 3.4 hundred million.
The data of the shared 1.8ZB in the whole world in 2011 generate, and the CD data that can be filled up are enough to be connected to the moon from the earth, however, this is only
It is the beginning of information " explosion ", the common of scientific and technological progress and application allows information to result from various sources, social networks,
The information that mobile device, sensor, all trades and professions constantly generate so that data are increased with annual 50% amount.
Whether by data-optimized existing business and management, or emerging business model is excavated, big data and new
The unprecedented door that chance is opened for enterprise of data technique.More accurately administrative decision branch can be provided for manager
Data are supportted, we are in 21 century, each business is generating a large amount of data, and everywhere is it can be seen that data are
The immense value that we bring.
In the big data epoch, there are many new data processing techniques, such as distributed caching technology, NOSQL technologies, cloud meter
Calculate etc., the appearance of these technologies solves data in " high reliability, High Availabitity primarily directed to real-time, magnanimity data processing
The requirement of property and consistency ".Based on the real-time mass data processing model of production exactly under the background in current big data epoch,
It is innovated in traditional data processing based on real-time data base and on the basis of relational database, and is looked forward to according to production management
Industry is the processing system of a set of magnanimity product practice of enterprise development to the concentrations of data, shared and service requirement.It should
System combines state-of-the-art mass data processing technology instantly, by enterprise to the management processes of data (acquisition, processing, storage,
Analysis) it is integrated, provide integrated solution for the data utilization of enterprise.
Various Internet enterprises at present, such as Taobao, Tencent, large enterprise of Microsoft have been widely used distribution
Data processing, distributed data cache, cloud computing and NOSQL technologies come cope with mass data processing under the current information epoch and
Analysis, provide to the user faster, more accurately information service.Original technology has the following disadvantages：1. the acquisition number of targets supported
It is single according to source, it can only be acquired from fixed data source, and for each acquisition target source, be required for disposing different applications,
It is not easy to manage and safeguard.2. acquisition module cannot ensure the accuracy of data, and cannot provide effective means help and use
Family finds mistake or abnormal data.3. pair basic network is more demanding, in the case of unstable networks, data transmission cannot be true
The timing of data is protected, while not supporting to realize off-line data caching.4. the simple data collection and transmission of realization, right
In the pretreatment of data, calculating, storage and the additional information system support of needs is shared, and needs to dispose multiple and different answer
With.5. carrying out data storage using traditional relevant database or real time data, the access efficiency of data depends on database
Product and hardware process performance, data volume is bigger, and the access efficiency of data is lower.6. lacking in data analysis and application
The technical support of aspect cannot help user to be found from mass data well and find valuable information.
Invention content
In view of this, the purpose of the present invention is be total to NoSQL databases MongoDB in data instead of traditional relational data
The application in platform is enjoyed, the characteristics of using MongoDB without " relationship ", improves the effect of data sharing platform data inputting and inquiry
Rate.
The production real time mass data processing system based on NoSQL of the present invention, including creation data import modul, production
Data sharing module, bulletin policy management module and system management module；The creation data import modul provides number for system
According to source, by administrator by associated production data inputting to database, and by super keepe to the data of addition into
Row audit when the audit fails, needs common management administrator to re-type new data, after audit passes through, the data
It is retrieved；The creation data sharing module is used to complete the flow of entire data sharing and every production for ordinary user
Information is checked.The production includes industry characteristics data, image data, video data and manufacturer's data；The bulletin
Policy management module is used to be directed to super keepe and common management administrator, completes the management function of production shared data；The system
Management module of uniting is used to complete the management function to system user addition or permission modification for super keepe；
Further, the framework level of the system includes expression layer, Business Logic, data Layer and data resource layer；It is described
Expression layer is to use ExJS and JQuery class library informations to develop for interface, interacted with server end by Ajax；The industry
Business logical layer includes that web server and data sharing manage required data query and share component and user right component；Institute
State interface of the data Layer for providing underlying database uniform range to Business Logic；The data resource layer integrates isomery number
It is used to provide data to upper layer according to source；
Further, the creation data sharing module provides following four retrieval mode for ordinary user：Mechanical type
Mechanical equipment is retrieved, mechanical equipment is retrieved according to function, mechanical equipment and advanced search are retrieved according to manufacturer；
Further, the data of this system safeguard that mechanical data, backstage language use by uploading Excel file
Python, database use MongoDB.
Beneficial effects of the present invention：The production real time mass data processing system based on NoSQL of the present invention, by NoSQL
Database MongoDB replaces application of the traditional relational data in data sharing platform, utilizes spies of the MongoDB without " relationship "
Point improves the efficiency of data sharing platform data inputting and inquiry, while base the characteristics of having studied mechanical equipment data
On plinth, using the web.py frames and non-relational database MongoDB of Python, it is complicated to solve creation data format
The problem of, realize the publication of mechanical equipment and interactive use, and associated production data can be effectively integrated.
Description of the drawings
The invention will be further described with reference to the accompanying drawings and examples.
Fig. 1 is the functional module structure figure of the present invention；
Fig. 2 is the platform architecture figure of the present invention；
Specific implementation mode
Fig. 1 is the functional module structure figure of the present invention, as shown in the figure：The production based on NoSQL of the present embodiment is extra large in real time
Measure data processing system comprising four big function modules, creation data import modul, creation data sharing module, bulletin policy
Management module, system management module, Fig. 1 divide for detailed function module.Creation data import modul provides data for system
Source.Common management administrator mainly completes the addition, modification and deletion function of creation data.Common management administrator is firstly the need of will be related
Creation data is entered into database, and the mode for being manually entered and being imported with file may be used；It is exactly common management to be manually entered
After member clicks interpolation data button, addition item by item needs the information of typing, the situation for being suitble to data volume less；File importing side
Formula is, by uploading excel files, once to add mass data after common management administrator clicks importing excel file buttons, be somebody's turn to do
Method is suitble to data volume bigger, is manually entered the no longer applicable situation of mode.Super keepe main task is to adding
The data added are audited, and when the audit fails, common management administrator are needed to re-type new data, should after audit passes through
Data can be retrieved.Super keepe also has the right for creating user and modification authority simultaneously.Creation data shares mould
Block completes the flow of entire data sharing and checking for every terms of information mainly for ordinary user.Ordinary user's first step is to determine
The strategy of data retrieval, there are four types of search strategies altogether, retrieve mechanical equipment according to mechanical type, are set according to function retrieval machinery
It is standby, mechanical equipment and advanced search are retrieved according to manufacturer, after select search strategy, input correlation search condition it
Afterwards, it submits.Ordinary user can obtain the results list of the search condition, and list shows such general production information, clicks one
Creation data check in detail after, the details of this creation data, including mechanical property data, picture number can be obtained
According to, video data, manufacturer's data etc..
Policy management module is announced mainly for super keepe and common management administrator, main production shared data of completing is put down
The management function of the contents such as platform portal news.Super keepe or common management administrator log in management platform, in abstract generation column
Middle selection needs newer article's style, such as industry Zone Information, policies and regulations, after clicking addition button, addition article occurs
The page of information fills in the particular content and pictorial information of article title, addition date and article, rearmost point on this page
Submission is hit, is added among database.Super keepe or common management administrator can also delete and change to article simultaneously
Operation, specific steps are similar to addition and operate.
System management module is mainly for super keepe, the main management completed to system user addition or permission modification
Function.Super keepe can utilize the module, by logging in management platform, system administration column be selected, in the corresponding page
Selection creates user or change permission.
The platform of this system is developed using the web.py frames of python language, using with it is currently popular
The opposite MTC frameworks of MVC frameworks, MTC are Model layers, Template layers and Controller layers；Model layers are mainly
Related data CRUD operations；The Controller layers of operation flow primarily with regard to system；Template layers be primarily used into
Row displaying is used, and web.py frames simplify the exploitation of presentation layer by mould plate technique, while various template replacement can also be supported to draw
It holds up.
The design of system structure combines B/S patterns, Ajax, python, MongoDB and various Open Frameworks, level
Structure such as Fig. 2, respectively：Expression layer, Business Logic, data Layer and data resource layer.
1) expression layer is user interface layer, and WebUI is developed using ExtJS and JQuery class libraries, passes through Ajax and clothes
The interaction of business device end, has very strong interactivity and user friendly.User is interacted by Web browser with server,
It obtains the information needed for oneself and realizes the functions such as the management of fodder grass mechanical data, bulletin policy management.
2) Business Logic is the core level of system, completes all business functions of system, includes mainly Web service
Device, while further including that data sharing manages required specialized service and component, including：Data query is shared component, is used
Family permission component and other management components.
3) data Layer is abstracted to underlying database, is a logic level for being absorbed in data manipulation, is patrolled to business
It collects layer and the unified interface accessed underlying database is provided.
4) data resource layer integrates various heterogeneous data sources and provides data to upper layer, is the entity of Deta bearer.Including letter
Cease database, other information database and Miscellaneous Documents.
Finally illustrate, the above examples are only used to illustrate the technical scheme of the present invention and are not limiting, although with reference to compared with
Good embodiment describes the invention in detail, it will be understood by those of ordinary skill in the art that, it can be to the skill of the present invention
Art scheme is modified or replaced equivalently, and without departing from the objective and range of technical solution of the present invention, should all be covered at this
In the right of invention.
Claims (4)
1. a kind of production real time mass data processing system based on NoSQL, it is characterised in that：Mould is imported including creation data
Block, creation data sharing module, bulletin policy management module and system management module；
The creation data import modul provides data source for system, by administrator by associated production data inputting to data
In library, and the data of addition are audited by super keepe, when the audit fails, common management administrator is needed to re-type
New data, after audit passes through, which can be retrieved；
The creation data sharing module is used to complete the flow of entire data sharing and every production information for ordinary user
Check, the production includes industry characteristics data, image data, video data and manufacturer's data；
The bulletin policy management module is used to be directed to super keepe and common management administrator, completes the management of production shared data
Function；
The system management module is used to complete the management function to system user addition or permission modification for super keepe.
2. the production real time mass data processing system according to claim 1 based on NoSQL, it is characterised in that：This is
The framework level of system includes expression layer, Business Logic, data Layer and data resource layer；
The expression layer is to use ExJS and JQuery class library informations to develop for interface, is handed over by Ajax and server end
Mutually；
The Business Logic includes that web server and data sharing manage required data query and share component and user
Permission component；
The data Layer is used to provide the interface of underlying database uniform range to Business Logic；
The data resource layer integrates heterogeneous data source and is used to provide data to upper layer.
3. the production real time mass data processing system according to claim 2 based on NoSQL, it is characterised in that：It is described
Creation data sharing module provides following four retrieval mode for ordinary user：Mechanical type retrieves mechanical equipment, according to work(
Mechanical equipment can be retrieved, mechanical equipment and advanced search are retrieved according to manufacturer.
4. the production real time mass data processing system according to claim 3 based on NoSQL, it is characterised in that：This is
The data of system safeguard that mechanical data, backstage language use python, database to use MongoDB by uploading Excel file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810380322.1A CN108595639A (en) | 2018-04-25 | 2018-04-25 | A kind of production real time mass data processing system based on NoSQL |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810380322.1A CN108595639A (en) | 2018-04-25 | 2018-04-25 | A kind of production real time mass data processing system based on NoSQL |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108595639A true CN108595639A (en) | 2018-09-28 |
Family
ID=63609826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810380322.1A Pending CN108595639A (en) | 2018-04-25 | 2018-04-25 | A kind of production real time mass data processing system based on NoSQL |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108595639A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114518907A (en) * | 2022-01-27 | 2022-05-20 | 佰聆数据股份有限公司 | Extensible self-service data analysis method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202795370U (en) * | 2012-08-27 | 2013-03-13 | 湖南大唐先一科技有限公司 | Massive data processing system on basis of production real-time property |
| CN103763344A (en) * | 2013-12-31 | 2014-04-30 | 湖南大唐先一科技有限公司 | General data sharing platform based on service bus and message drive mode |
| CN104361424A (en) * | 2014-10-11 | 2015-02-18 | 中国电子科技集团公司第十研究所 | Master data system integration method based on enterprise service bus (ESB) |
| CN104657918A (en) * | 2015-01-21 | 2015-05-27 | 胡宝清 | Regional resource environmental data sharing and comprehensive service platform |
| CN104700227A (en) * | 2015-03-23 | 2015-06-10 | 云南电网有限责任公司信息中心 | Station affair management system platform of power supply station of county level power supply enterprise |
-
2018
- 2018-04-25 CN CN201810380322.1A patent/CN108595639A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202795370U (en) * | 2012-08-27 | 2013-03-13 | 湖南大唐先一科技有限公司 | Massive data processing system on basis of production real-time property |
| CN103763344A (en) * | 2013-12-31 | 2014-04-30 | 湖南大唐先一科技有限公司 | General data sharing platform based on service bus and message drive mode |
| CN104361424A (en) * | 2014-10-11 | 2015-02-18 | 中国电子科技集团公司第十研究所 | Master data system integration method based on enterprise service bus (ESB) |
| CN104657918A (en) * | 2015-01-21 | 2015-05-27 | 胡宝清 | Regional resource environmental data sharing and comprehensive service platform |
| CN104700227A (en) * | 2015-03-23 | 2015-06-10 | 云南电网有限责任公司信息中心 | Station affair management system platform of power supply station of county level power supply enterprise |
Non-Patent Citations (1)
| Title |
|---|
| 苏坡等: "基于NoSQL数据库的饲草料生产机械设备数据共享平台的设计与实现", 《软件工程与应用》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114518907A (en) * | 2022-01-27 | 2022-05-20 | 佰聆数据股份有限公司 | Extensible self-service data analysis method and system |
| CN114518907B (en) * | 2022-01-27 | 2023-11-03 | 佰聆数据股份有限公司 | Extensible self-service data analysis method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11080279B2 (en) | Dynamic data management | |
| US10592474B2 (en) | Processing log files using a database system | |
| US10776336B2 (en) | Dynamic creation and maintenance of multi-column custom indexes for efficient data management in an on-demand services environment | |
| Gousios | The GHTorent dataset and tool suite | |
| US10339126B2 (en) | Processing log files using a database system | |
| US9647922B2 (en) | Computer implemented methods and apparatus for trials onboarding | |
| US8407205B2 (en) | Automating sharing data between users of a multi-tenant database service | |
| US10200337B2 (en) | Combining updates of a social network feed | |
| US8880522B2 (en) | Generating reports in an online services system | |
| US20180025113A1 (en) | Event detail processing at run-time | |
| US20180096024A1 (en) | Release management in a content management system | |
| US10579692B2 (en) | Composite keys for multi-tenant non-relational platform objects | |
| US9547689B2 (en) | Security descriptors for record access queries | |
| US11714811B2 (en) | Run-time querying of multi-tenant non-relational platform objects | |
| US10469438B2 (en) | Facilitating integration of collaborative communication platform and document collaboration tool | |
| US11275806B2 (en) | Dynamic materialization of feeds for enabling access of the feed in an online social network | |
| US10599654B2 (en) | Method and system for determining unique events from a stream of events | |
| US10642862B2 (en) | System and method for referencing master data in a multi-tenant application-platform-as-a-service architecture | |
| US11003662B2 (en) | Trigger-free asynchronous maintenance of custom indexes and skinny performance meta-structures | |
| US20200136938A1 (en) | Generating events from host based logging for consumption by a network logging host | |
| US20240104630A1 (en) | Generation of recommendations from dynamically-mapped data | |
| CN108595639A (en) | A kind of production real time mass data processing system based on NoSQL | |
| Delchev et al. | Big Data Analysis Architecture | |
| US10545962B2 (en) | Relational database instruction validation | |
| Alvord et al. | Big data fabric architecture: How big data and data management frameworks converge to bring a new generation of competitive advantage for enterprises |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180928 |
|
| RJ01 | Rejection of invention patent application after publication |