CN111428062A - Data processing system, method, terminal and medium of view library system - Google Patents
Data processing system, method, terminal and medium of view library system Download PDFInfo
- Publication number
- CN111428062A CN111428062A CN202010219385.6A CN202010219385A CN111428062A CN 111428062 A CN111428062 A CN 111428062A CN 202010219385 A CN202010219385 A CN 202010219385A CN 111428062 A CN111428062 A CN 111428062A
- Authority
- CN
- China
- Prior art keywords
- unit
- data processing
- data
- interface
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/71—Indexing; Data structures therefor; Storage structures
Abstract
The invention discloses a data processing system of a view library system, which comprises: the interface system comprises an interface application unit, an interface service unit, a middle data processing unit and a bottom database unit, wherein the interface application unit is used for providing application service; the interface service unit is used for providing interface service and transmitting data; the intermediate data processing unit is used for counting the data sources in storage in real time and providing data subscription service for the outside; and the bottom-layer database unit stores data by adopting a column type storage method. The invention has the beneficial effects that: the view library system meets the industrial standard, an index unit is not required to be arranged in the system, the hardware resource requirement is greatly reduced, the efficiency is high under the condition of large data volume, and the distributed deployment is stable.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data processing system, a data processing method, a data processing terminal and a data processing medium of a view library system meeting 1400 standards.
Background
The GA/T1400 public safety industry standard of the people's republic of China, which specifies the technical requirements of 4 parts of a public security video image information application system: the method comprises the following steps: general technical requirements, application platform technical requirements, database technical requirements and interface protocol requirements. A Video Image Information Database (VIID) conforms to standards such as a GA/T1399 public security video image analysis system and a GA/T1400 public security video image information application system, and based on technologies such as Internet of things multi-dimensional sensing acquisition, view intelligent analysis, mixed cloud computing and view big data analysis, original videos and pictures are intelligently analyzed and stored in a classified mode, and are combined with other sensing data (RFID and GPS), so that the Video Image Information Database (VIID) is applied to public security upper-layer services and provides unified data service support capacity. At present, under the condition of large data volume, an index layer is generally established on the basis of a database, for example: es, satisfy high performance queries but consume more hardware resources and do not maintain data synchronization well.
Disclosure of Invention
Aiming at the defects in the prior art, the embodiment of the invention provides a data processing system, a data processing method, a data processing terminal and a data processing medium of a view library system, which do not need an index layer, greatly reduce the hardware resource requirement, have high efficiency under the condition of large data volume and are stable in distributed deployment.
In a first aspect, an embodiment of the present invention provides a data processing system of a view library system, including: an interface application unit, an interface service unit, an intermediate data processing unit and a bottom database unit,
the interface application unit is used for providing application services;
the interface service unit is used for providing interface service and transmitting data;
the intermediate data processing unit is used for counting the data sources in storage in real time and providing data subscription service for the outside;
and the bottom-layer database unit stores data by adopting a column type storage method.
In a second aspect, a data processing method of a view library system provided in an embodiment of the present invention is applicable to the system described in the foregoing embodiment, and the method includes:
the interface application unit displays the application service;
the interface service unit provides interface service for the outside and performs data transmission;
the intermediate data processing unit counts the data sources in storage in real time and provides data subscription service for the outside;
the bottom database unit stores data in a column type storage mode.
In a third aspect, an intelligent terminal provided in an embodiment of the present invention includes a processor, an input device, an output device, and a memory, where the processor, the input device, the output device, and the memory are connected to each other, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method steps described in the foregoing embodiment.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored, the computer program comprising program instructions, which, when executed by a processor, cause the processor to perform the method steps described in the above embodiments.
The invention has the beneficial effects that:
the view library system meets the industrial standard, an index unit is not required to be arranged in the data processing process, the hardware resource requirement is greatly reduced, the efficiency is high under the condition of large data volume, and the distributed deployment is stable.
Drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below. Throughout the drawings, like elements or portions are generally identified by like reference numerals. In the drawings, elements or portions are not necessarily drawn to scale.
FIG. 1 is a block diagram of a data processing system of a view library system according to a first embodiment of the present invention;
FIG. 2 is a flow chart illustrating a data processing method of a view library system according to another embodiment of the present invention;
fig. 3 shows a schematic structural diagram of an intelligent terminal according to another embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which the invention pertains.
As shown in fig. 1, there is a schematic structural diagram illustrating a data processing system of a view library system according to a first embodiment of the present invention, where the system includes: the interface application unit is used for providing application service; the interface application unit is mainly used for providing a visual display interface for a user. Firstly, when nginx receives an http request, finding a corresponding server through a configuration file, then matching all locations in the server, giving a request of suffix of a file by php to php-fpm service for processing through a location instruction, selecting and connecting php-fpm to a fastcgi subprocess, sending an environment variable and a standard input to the fastcgi subprocess, returning a standard output and error information after the fastgi subprocess finishes processing, and when the fastcgi subprocess closes the connection, finishing the processing of the current request. The interface service unit is used for providing interface service and transmitting data. After receiving a request sent from the interface application unit or the third-party application, the flash service of the interface service unit judges the type of the request, if the request is a query request, the interface service unit communicates with the bottom database unit, and corresponding data is returned to the interface application unit or the third-party application; if the request is a data submission request, the submitted data is subjected to cleaning and formatting and then forwarded to the intermediate data processing unit. The intermediate data processing unit is used for counting the data sources in storage in real time and providing data subscription service for the outside; the intermediate processing unit mainly adopts kafka and redis as message middleware, a series of python services are used for consuming and maintaining the data received and processed by the whole system, the data are efficiently stored in a database, and data subscription and distribution services are provided for the outside. And the bottom layer data unit stores data by adopting a column type storage method. The bottom layer data unit adopts srv _ db of the company as a database to store data sent by various services of the intermediate message processing unit, and maintains (inquires, deletes and updates) the stored historical data according to instructions sent by the interface service unit.
In this embodiment, the interface application unit adopts nginx + php to build a web system, the interface service unit adopts python flash framework for development, and is started by gunitron, gunicorn is a high-performance python wsgi Unix Http Server of Unix, and provides high-performance and high-concurrency interface service. The intermediate data processing unit adopts kafka and redis as message middleware, wherein the redis mainly performs real-time statistics, and the kafka is used as a warehousing data source inside the program to provide data subscription service for the outside. The underlying database unit is stored by srv _ db, and is a database stored in a columnar mode, so that high-performance query and mass data storage are met. In the embodiment, an index layer is not needed, hardware resource requirements are greatly reduced, efficiency is not lower than that of es under the condition of large data volume, database falling is fast, distributed deployment is stable, stable interface service can be provided for the outside by means of flash frameworks of kafka and python, and internal data processing is efficient.
The data processing system of the view library system provided by the embodiment of the invention meets the industrial standard, the system does not need to be provided with an index unit, the hardware resource requirement is greatly reduced, the efficiency is high under the condition of large data volume, and the distributed deployment is stable.
In the first embodiment described above, a data processing system of a view library system is provided, and correspondingly, the present application also provides a data processing method of the view library system. Please refer to fig. 2, which is a flowchart illustrating a data processing method of a view library system according to a second embodiment of the present invention. Since the method embodiment is basically similar to the device embodiment, the description is simple, and the relevant points can be referred to the partial description of the device embodiment.
As shown in fig. 2, there is shown a flowchart of a data processing method of a view library system according to another embodiment of the present invention, which is applied to the data processing system of the view library system described in the first embodiment, and the method includes the following steps:
s1, displaying application service by the interface application unit;
s2, the interface service unit provides interface service for external use to transmit data;
s3, the intermediate data processing unit counts the warehousing data sources in real time and provides data subscription service for the outside;
and S4, storing the data by the bottom database unit in a columnar storage mode.
In this embodiment, the interface application unit adopts nginx + php to build a web system, the interface service unit adopts python flash framework for development, and is started by gunitron, gunicorn is a high-performance python wsgi Unix Http Server of Unix, and provides high-performance and high-concurrency interface service. The intermediate data processing unit adopts kafka and redis as message middleware, wherein the redis mainly performs real-time statistics, and the kafka is used as a warehousing data source inside the program to provide data subscription service for the outside. The underlying database unit is stored by srv _ db, and is a database stored in a columnar mode, so that high-performance query and mass data storage are met. In the embodiment, an index layer is not needed, hardware resource requirements are greatly reduced, efficiency is not lower than that of es under the condition of large data volume, database falling is fast, distributed deployment is stable, stable interface service can be provided for the outside by means of flash frameworks of kafka and python, and internal data processing is efficient.
According to the data processing method of the view library system, the view library system meets the industrial standard, an index unit is not required to be arranged in the data processing process, the hardware resource requirement is greatly reduced, the efficiency is high under the condition of large data volume, and the distributed deployment is stable.
As shown in fig. 3, it is shown that another embodiment of the present invention provides an intelligent terminal structure schematic diagram, the terminal includes a processor, an input device, an output device, and a memory, the processor, the input device, the output device, and the memory are connected with each other, the memory is used for storing a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the first embodiment.
It should be understood that in the embodiments of the present invention, the Processor may be a Central Processing Unit (CPU), and the Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input devices may include a touch pad, a fingerprint sensor (for collecting fingerprint information of a user and direction information of the fingerprint), a microphone, etc., and the output devices may include a display (L CD, etc.), a speaker, etc.
The memory may include both read-only memory and random access memory, and provides instructions and data to the processor. The portion of memory may also include non-volatile random access memory. For example, the memory may also store device type information.
In a specific implementation, the processor, the input device, and the output device described in the embodiments of the present invention may execute the implementation described in the method embodiments provided in the embodiments of the present invention, and may also execute the implementation described in the system embodiments in the embodiments of the present invention, which is not described herein again.
The invention also provides an embodiment of a computer-readable storage medium, in which a computer program is stored, which computer program comprises program instructions that, when executed by a processor, cause the processor to carry out the method described in the above embodiment.
The computer readable storage medium may be an internal storage unit of the terminal described in the foregoing embodiment, for example, a hard disk or a memory of the terminal. The computer readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the terminal. The computer-readable storage medium is used for storing the computer program and other programs and data required by the terminal. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the terminal and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed terminal and method can be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.
Claims (8)
1. A data processing system of a view library system, comprising: an interface application unit, an interface service unit, an intermediate data processing unit and a bottom database unit,
the interface application unit is used for providing application services;
the interface service unit is used for providing interface service and transmitting data;
the intermediate data processing unit is used for counting the data sources in storage in real time and providing data subscription service for the outside;
and the bottom-layer database unit stores data by adopting a column type storage method.
2. The data processing system of a view library system of claim 1, wherein the interface service unit is developed using a flash framework.
3. The data processing system of the view library system of claim 1, wherein the intermediate data processing unit employs kafka and redis as message middleware.
4. A data processing method of a view library system, which is applied to the system of any one of claims 1 to 3, the method comprising:
the interface application unit displays the application service;
the interface service unit provides interface service for the outside and performs data transmission;
the intermediate data processing unit counts the data sources in storage in real time and provides data subscription service for the outside;
the bottom database unit stores data in a column type storage mode.
5. The data processing method of the view library system of claim 4, wherein the interface service unit is developed using a flash framework.
6. The data processing method of the view library system of claim 4, wherein the intermediate data processing unit employs kafka and redis as message middleware.
7. An intelligent terminal comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, the memory being adapted to store a computer program comprising program instructions, characterized in that the processor is configured to invoke the program instructions to perform the method steps according to any of claims 4 to 6.
8. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps according to any one of claims 4 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010219385.6A CN111428062A (en) | 2020-03-25 | 2020-03-25 | Data processing system, method, terminal and medium of view library system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010219385.6A CN111428062A (en) | 2020-03-25 | 2020-03-25 | Data processing system, method, terminal and medium of view library system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111428062A true CN111428062A (en) | 2020-07-17 |
Family
ID=71548853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010219385.6A Pending CN111428062A (en) | 2020-03-25 | 2020-03-25 | Data processing system, method, terminal and medium of view library system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111428062A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193669A (en) * | 2017-05-09 | 2017-09-22 | 千寻位置网络有限公司 | The system and design method of maintenance interface based on mixed cloud or large-scale cluster |
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN110110002A (en) * | 2019-05-13 | 2019-08-09 | 江南大学 | Big data virtual interactive interface system |
| CN110535909A (en) * | 2019-07-29 | 2019-12-03 | 广东电网有限责任公司信息中心 | Big data and cloud computing unified platform system towards energy Internet application |
| CN110716744A (en) * | 2019-10-21 | 2020-01-21 | 中国科学院空间应用工程与技术中心 | Data stream processing method, system and computer readable storage medium |
-
2020
- 2020-03-25 CN CN202010219385.6A patent/CN111428062A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804237A (en) * | 2017-05-05 | 2018-11-13 | 北京京东尚科信息技术有限公司 | Data real-time statistical method, device, storage medium and electronic equipment |
| CN107193669A (en) * | 2017-05-09 | 2017-09-22 | 千寻位置网络有限公司 | The system and design method of maintenance interface based on mixed cloud or large-scale cluster |
| CN110110002A (en) * | 2019-05-13 | 2019-08-09 | 江南大学 | Big data virtual interactive interface system |
| CN110535909A (en) * | 2019-07-29 | 2019-12-03 | 广东电网有限责任公司信息中心 | Big data and cloud computing unified platform system towards energy Internet application |
| CN110716744A (en) * | 2019-10-21 | 2020-01-21 | 中国科学院空间应用工程与技术中心 | Data stream processing method, system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200159702A1 (en) | Method, apparatus, and computer program product for data quality analysis | |
| CN111198859B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| WO2021056737A1 (en) | Data compression method and apparatus for high-frequency service data, device, and storage medium | |
| CN111273999B (en) | Data processing method and device, electronic equipment and storage medium | |
| US10007562B2 (en) | Business transaction context for call graph | |
| CN114244595A (en) | Method and device for acquiring authority information, computer equipment and storage medium | |
| CN110598093A (en) | Business rule management method and device | |
| CN111428062A (en) | Data processing system, method, terminal and medium of view library system | |
| US20140089525A1 (en) | Compressed analytics data for multiple recurring time periods | |
| CN110022343B (en) | Adaptive event aggregation | |
| US11954630B2 (en) | Real time method and system for analyzing data streams | |
| CN116028696A (en) | Resource information acquisition method and device, electronic equipment and storage medium | |
| WO2023066258A1 (en) | Data processing method and apparatus for private data, computer device and medium | |
| CN115187364A (en) | Method and device for monitoring deposit risk under bank distributed scene | |
| CN111680754B (en) | Image classification method, device, electronic equipment and computer readable storage medium | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| US20180107763A1 (en) | Prediction using fusion of heterogeneous unstructured data | |
| CN113434572A (en) | Data query method, query system, device, storage medium and program product | |
| CN113468342A (en) | Data model construction method, device, equipment and medium based on knowledge graph | |
| CN112182008A (en) | System, method, terminal and medium for analyzing face picture data acquired by mobile terminal | |
| CN114510299A (en) | Method, device and storage medium for processing artificial intelligence service | |
| CN111222048A (en) | User number query calculation method and device, electronic equipment and storage medium | |
| CN111258670B (en) | Method and device for managing component data, electronic equipment and storage medium | |
| US11087103B2 (en) | Adaptive spatial granularity based on system performance | |
| CN111930704B (en) | Service alarm equipment control method, device, equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 266400 No. 77, Lingyan Road, LINGSHANWEI sub district office, Huangdao District, Qingdao City, Shandong Province Applicant after: Issa Technology Co.,Ltd. Applicant after: QINGDAO YISA DATA TECHNOLOGY Co.,Ltd. Address before: 266400 No. 77, Lingyan Road, LINGSHANWEI sub district office, Huangdao District, Qingdao City, Shandong Province Applicant before: Qingdao Issa Technology Co.,Ltd. Applicant before: QINGDAO YISA DATA TECHNOLOGY Co.,Ltd. Address after: 266400 No. 77, Lingyan Road, LINGSHANWEI sub district office, Huangdao District, Qingdao City, Shandong Province Applicant after: Qingdao Issa Technology Co.,Ltd. Applicant after: QINGDAO YISA DATA TECHNOLOGY Co.,Ltd. Address before: 100020 room 108, 1 / F, building 17, yard 6, Jingshun East Street, Chaoyang District, Beijing Applicant before: BEIJING YISA TECHNOLOGY Co.,Ltd. Applicant before: QINGDAO YISA DATA TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information |