CN113051957A - Data analysis method and system - Google Patents
Data analysis method and system Download PDFInfo
- Publication number
- CN113051957A CN113051957A CN201911362443.4A CN201911362443A CN113051957A CN 113051957 A CN113051957 A CN 113051957A CN 201911362443 A CN201911362443 A CN 201911362443A CN 113051957 A CN113051957 A CN 113051957A
- Authority
- CN
- China
- Prior art keywords
- analysis
- analyzed
- data
- node
- analysis node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000007405 data analysis Methods 0.000 title claims abstract description 26
- 238000004458 analytical method Methods 0.000 claims abstract description 314
- 238000004891 communication Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/08—Detecting or categorising vehicles
Landscapes
- Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The application provides a data analysis method and system, and relates to the technical field of data analysis. The analysis system comprises at least two analysis nodes, the at least two analysis nodes are communicated through a bus, and the data volume to be analyzed of each analysis node is acquired; and then, each analysis node allocates the data volume to be analyzed through the cache of the bus according to the data volume to be analyzed. The data analysis method and the data analysis system have the advantages that the data volume to be analyzed can be balanced effectively, and the maximum calculation power of the whole system can be exerted.
Description
Technical Field
The present application relates to the field of data analysis technologies, and in particular, to a data analysis method and system.
Background
The AI (Artificial Intelligence) technology is advanced rapidly, the computing power and the algorithm are greatly improved and enriched, security manufacturers have greatly developed the AI technology in the field of video monitoring, and the method plays a great role in the field of AI landing and social stability development.
In order to increase the processing speed of images or videos, a plurality of analysis nodes are generally required to simultaneously analyze the video stream or the image stream.
However, due to the imbalance of the target object in the actual use process, some analysis nodes are wasted, or the analysis amount of some analysis nodes is too large, so that the maximum calculation power of the whole analysis system cannot be exerted.
Disclosure of Invention
The present application aims to provide a data analysis method and system to solve the problem that the maximum computational power of the whole analysis system cannot be exerted when data analysis is performed in the prior art.
In order to achieve the above purpose, the embodiments of the present application employ the following technical solutions:
in one aspect, an embodiment of the present application provides a data analysis method, where the method is applied to an analysis system, where the analysis system includes at least two analysis nodes, and the at least two analysis nodes communicate with each other through a bus, and the method includes:
each analysis node acquires the data volume to be analyzed of the analysis node;
and allocating the data volume to be analyzed of each analysis node through the cache of the bus according to the data volume to be analyzed of each analysis node.
In another aspect, an analysis system is provided in an embodiment of the present application, where the analysis system includes at least two analysis nodes, and the at least two analysis nodes communicate with each other through a bus;
each analysis node is used for acquiring the data volume to be analyzed of the analysis node;
each analysis node is further configured to allocate the data volume to be analyzed of the analysis node through the cache of the bus according to the data volume to be analyzed of the analysis node.
Compared with the prior art, the method has the following beneficial effects:
the application provides a data analysis method, which is applied to an analysis system, wherein the analysis system comprises at least two analysis nodes, the at least two analysis nodes are communicated through a bus, and the data volume to be analyzed of each analysis node is acquired; and then, each analysis node allocates the data volume to be analyzed through the cache of the bus according to the data volume to be analyzed. According to the method and the device, each analysis node can allocate the data volume to be analyzed of the analysis node through the cache of the bus, so that the balance of the data volume to be analyzed can be effectively realized for each analysis node, and the maximum calculation power of the whole system can be exerted.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and it will be apparent to those skilled in the art that other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a schematic flow chart of a data analysis method provided in an embodiment of the present application.
Fig. 2 is a schematic block diagram of analysis nodes connected by a bus according to an embodiment of the present disclosure.
Fig. 3 is a schematic diagram of another module in which the analysis nodes are connected through a bus according to an embodiment of the present application.
Fig. 4 is a block diagram of an analysis system according to an embodiment of the present disclosure.
Fig. 5 is a first schematic flowchart of S104 in fig. 1 according to an embodiment of the present disclosure.
Fig. 6 is a second schematic flowchart of S104 in fig. 1 according to an embodiment of the present disclosure.
Fig. 7 is a third schematic flowchart of S104 in fig. 1 according to an embodiment of the present disclosure.
Fig. 8 is a fourth schematic flowchart of S104 in fig. 1 according to an embodiment of the present disclosure.
Fig. 9 is another schematic flow chart of a data analysis method provided in an embodiment of the present application.
In the figure: 200-an analysis system; 210-a first analysis node; 220-a second analysis node; 230-a management node; 240-data access node; 250-data center.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
First embodiment
As described in the background art, for an analysis system, due to the imbalance of target objects in the actual use process, some analysis nodes are wasted or the analysis amount of some analysis nodes is too large, and the maximum computation power of the whole analysis system cannot be exerted.
For example, a hierarchical system includes an analysis node a and an analysis node B, where the analysis node a and the analysis node B are both used for vehicle identification and analysis, for example, to identify and analyze a license plate of a vehicle, and the analysis node a and the analysis node B are both in communication with a management node, the management node configures the analysis node a to acquire a video stream captured by a camera a for analysis, and configures the analysis node B to acquire a video stream captured by a camera B for analysis. Due to uncertainty in the driving process of the vehicle, it may occur that within a certain time period, the camera a does not shoot any vehicle, and the camera B shoots consecutive vehicles, and it can be understood that at this time, the analysis node a is in an idle state, and the analysis node B is in a full-load working state, that is, at this time, there is a problem that the computational power (i.e., analysis capability) of the analysis node a is wasted, and the analysis amount of the analysis node B is too large.
In view of this, the present application provides a data analysis method, which implements allocation of each analysis node and a cache of a bus in a bus manner, thereby exerting the maximum computation power of the entire analysis system.
The following is an exemplary illustration of the data analysis method provided in the present application:
referring to fig. 1, as a possible implementation manner provided by the present application, the method includes:
and S102, each analysis node acquires the data volume to be analyzed.
And S104, allocating the data volume to be analyzed of each analysis node through the cache of the bus according to the data volume to be analyzed of each analysis node.
It should be noted that, in the present application, each analysis node can obtain its own data volume to be analyzed, and for each analysis node, the data volume to be analyzed of its own node may be the same or different, for example, the data volume to be analyzed of the analysis node a is 100, that is, there are also 100 data to be analyzed in the analysis node a for analysis, and the data volume to be analyzed of the analysis node B is 0, that is, there is no data to be analyzed in the analysis node B at this time, and the analysis node B is in an idle state.
After each analysis node obtains the data volume to be analyzed, the analysis node allocates the data volume to be analyzed through the cache of the bus. The allocation described in the application refers to that for an analysis node, when the amount of data to be processed is large, the redundant amount of data to be processed can be sent to a cache of a bus; and when the data volume to be processed of the analysis node is small, the analysis node can acquire the data to be analyzed from the buffer of the bus to process the data.
It should be noted that, since each analysis node is allocated with the cache of the bus, the present application essentially uses the cache of the bus as a medium to implement allocation among the analysis nodes.
As a possible implementation manner of the present application, please refer to fig. 2, the analysis nodes described in the present application may include a bus module for each analysis node through bus communication, and the bus module in each analysis node is connected to the bus modules in other analysis nodes, so as to implement communication between the analysis nodes. In addition, in the implementation manner, each analysis node includes a bus cache, so that allocation between the analysis nodes and the bus caches can be realized.
As another possible implementation manner of the present application, please refer to fig. 3, where the analysis nodes described in the present application communicate through a bus, that means that each analysis node is connected through the bus and communicates with the other analysis node, and in the implementation manner, the cache of the bus may be a fixed cache, for example, the cache of the bus is implemented in the form of a common pool, that is, each analysis node is connected with the common pool through the bus, so as to implement allocation between each analysis node and the common pool.
In addition, in order to implement management of the analysis nodes, please refer to fig. 4, the analysis system 200 further includes a management node 230, wherein the management node 230 is connected to each analysis node to further manage the work of each analysis node. For example, in a certain area, the analysis node includes a and B, and the source of the data source includes a, B, c, and d, the management node 230 may configure the analysis node a to receive the data source from a and B, and configure the analysis node B to receive the data source from c and d, thereby implementing analysis of all data in the whole area.
It should be noted that, in the present application, the data source of the analysis node is not fixed, for example, when the analysis node a fails, the management node 230 may control the analysis node B to receive the data source from a, B, c, d.
Meanwhile, it should be noted that the present application may also utilize a bus to implement the connection between the management node 230 and the analysis node, for example, a bus module is also installed in the management node 230, and then the connection between the management node 230 and the analysis node is implemented through the bus module.
By the analysis system 200 provided by the application, the management node 230 and the analysis node can form a bus cluster similar to a distributed bus cluster, and the cluster bus is integrally formed by the management node and the analysis node. Meanwhile, load balance of each analysis node is realized through the cache of the bus, and further system analysis resources can be utilized to the maximum extent.
As a first implementation manner of the present application, the analysis nodes include a first analysis node 210 and at least one second analysis node 220, please refer to fig. 5, and S104 includes:
s104-1, the first analysis node judges whether the data volume to be analyzed is larger than a first threshold value, if so, S104-2 is executed.
And S104-2, the first analysis node sends redundant data to be analyzed to a cache of the bus.
And S104-3, judging whether the data volume to be analyzed of the first analysis node and/or the second analysis node is lower than a second threshold value, and if so, executing S104-4.
And S104-4, the first analysis node and/or the second analysis node acquire the data to be analyzed from the cache of the bus and analyze the data.
That is, for any analysis node, when the amount of data to be analyzed is large, the redundant data to be analyzed can be sent to the buffer of the bus, so that when the amount of data to be analyzed is small, other analysis nodes or the analysis node itself can obtain the data to be analyzed from the buffer of the bus for analysis.
For example, if the first threshold is set to 100, the first analysis node 210 will send more data to be processed to the cache on the bus when the amount of data to be processed of the first analysis node 210 exceeds 100. If the data source continuously sends data to the first analysis node 210, and the analysis speed of the first analysis node 210 is lower than the speed of the data input from the data source to the first analysis node 210, the data to be processed by the first analysis node 210 will be accumulated. Once the amount of the to-be-processed data exceeds the first threshold, it indicates that the analysis pressure of the first analysis node 210 is large at this time, and at this time, the extra to-be-processed data needs to be sent to the cache of the bus.
When a certain second analysis node 220 is idle, for example, when the second threshold is set to 20, if the amount of data to be processed of the second analysis node 220 is only 5 at this time, the second analysis node 220 can obtain the data to be processed sent by the first analysis node 210 from the cache of the bus, and process the data to be processed through the second analysis node 220, so that load balancing of different analysis nodes is achieved, and the analysis capability of the analysis nodes in the system is utilized to the maximum. Optionally, when the amount of data to be processed of the second analysis node 220 is greater than or equal to the second threshold, the second analysis node 220 processes its own data preferentially.
Of course, the first analysis node 210 may also retrieve the data to be analyzed from the cache of the bus and analyze the data when it is relatively idle. That is, when the processing amount is large, the first analysis node 210 sends the redundant amount of data to be processed to the cache of the bus, and when the first analysis node 210 is in the idle state again, the first analysis node 210 acquires the data put in before from the cache of the bus again.
For example, when more data is input into the first analysis node 210 from the data source in a certain period of time, the data to be processed in the first analysis node 210 is accumulated, and the first analysis node 210 sends the redundant data to be processed to the cache of the bus. If the other second analysis node 220 is also in a state with a large processing amount, the second analysis node 220 does not acquire and analyze the data in the bus. On this basis, if the data input to the first analysis node 210 by the data source is decreased, and when the amount of the data to be processed of the first analysis node 210 is decreased to the second threshold, the first analysis node 210 will pull the previously stored data to be processed from the cache of the bus.
It should be noted that, in the present application, the first threshold and the second threshold of each analysis node may be the same or different, and the present application is not limited in any way. For example, for some analysis nodes, if the analysis capability is poor, the first threshold is relatively low; if the analysis capability is better, the first threshold is relatively higher.
As a possible implementation manner of the present application, the data source may be a video captured by a video capture device. That is, the analysis system 200 provided in the present application further includes a video acquisition device, such as a camera. Of course, the data source may also be other sources, such as a buffered video source, and the like, which is not limited in this application.
On this basis, referring to fig. 6, S104 includes:
and S104-a, the first analysis node acquires the video stream to be analyzed and preprocesses the video stream to be analyzed to acquire the preprocessed picture information to be analyzed.
And S104-b, the first analysis node stores the picture information to be analyzed into a local cache.
S104-c, when the number of the picture information to be analyzed in the local cache reaches a first threshold value, the first analysis node sends redundant picture information to be analyzed to the cache of the bus.
That is, in the present application, the first analysis node 210 can directly obtain the video stream to be analyzed, which is sent by the camera. And the video stream to be analyzed is converted into the picture information to be analyzed after being preprocessed.
Optionally, the first analysis node 210 may include a bus module, an analysis module, a decoding module, and a local cache module, where the decoding module is configured to pre-process a received video stream to be analyzed. Including but not limited to decoding, matting, etc. of the video stream. Namely, converting the video flow into a picture, and carrying out cutout on the picture, and the like.
For example, the first analysis node 210 is used to analyze a vehicle type condition, such as whether a vehicle violation is present. When the decoding module decodes, the video stream is converted into a picture, and the part of the picture where the motor vehicle appears is subjected to matting. In this case, there may be a case where no vehicle passes through a certain period of time, or there may be a case where there are many vehicles in a certain period of time. Or when the human face is needed to be identified, the decoding module can decode the video stream and simultaneously perform the matting on the human face. Meanwhile, the decoding module can also perform matting and the like on various types of pictures at the same time, for example, the decoding module can simultaneously identify a vehicle, a human face and the like in the pictures.
After preprocessing is performed and the preprocessed to-be-analyzed picture information is obtained, an analysis module in the analysis node analyzes the to-be-analyzed picture information according to a certain rule, for example, whether a red light running operation exists in a vehicle is analyzed.
As a possible implementation manner, after the analysis module in the first analysis node 210 preprocesses the video stream, the to-be-analyzed picture information acquired after preprocessing is stored in the local cache module. Moreover, it can be understood that the local cache module has a certain upper storage limit, for example, the upper storage limit of the local cache module is 1000 pieces of picture information to be analyzed. Meanwhile, the first threshold and the second threshold described in this embodiment are actually thresholds set for the local cache module, for example, the first threshold is set to 80% of the upper limit of the local cache module, that is, the first threshold is 800, and the second threshold is set to 20% of the upper limit of the local cache module, that is, the second threshold is 200.
When the number of the to-be-analyzed picture information in the local cache reaches the first threshold, the first analysis node 210 sends the redundant to-be-analyzed picture information to the cache of the bus. As an optional implementation manner of the present application, the local cache module actively pulls the picture to be analyzed after being processed by the decoding module. When the picture information to be analyzed in the local cache module reaches the first threshold value, the local cache module suspends the pulling of the picture information to be analyzed.
At this time, if the decoding matting module further analyzes the to-be-analyzed picture information, the first analysis node 210 sends the to-be-analyzed picture information analyzed by the decoding matting module at this time to the cache of the bus. In this period of time, the picture to be analyzed, which is generated after being processed by the decoding matting module, can directly enter the cache of the bus, and cannot enter the local cache module.
As a possible implementation manner of the present application, after the first analysis node 210 sends the information of the picture to be analyzed to the cache of the bus, there is a second analysis node 220 in an idle state, for example, the number of the information of the picture to be analyzed of the second analysis node 220 at this time is 5, and at this time, the second analysis node 220 obtains the picture to be analyzed from the cache of the bus and analyzes the picture.
As another possible implementation manner of the present application, after the first analysis node 210 sends the information of the picture to be analyzed to the cache of the bus, the amount of the data to be analyzed in the second analysis node 220 is large, and therefore the second analysis node 220 does not obtain the data from the cache of the bus, on this basis, if the information of the picture to be analyzed in the local cache module of the first analysis node 210 is continuously consumed and is lower than the second picture to be analyzed, the local cache module of the first analysis node 210 may take the picture to be analyzed again from the cache of the bus and analyze the picture.
It should be noted that, in the present application, the analysis module of the first analysis node 210 only pulls the data in the local cache module and performs analysis, and when the data in the local cache is lower than the second threshold, the local cache module of the first analysis node 210 acquires the data from the cache of the bus.
In addition to the above-described implementation, the first threshold and the second threshold satisfy a relationship:
LT_high–LT_low≥n*m；
n*m＝x+△；
wherein LT _ high represents a first threshold value, and LT _ low represents a second threshold value; n represents the time period for calling the picture information to be analyzed from the cache of the bus; m represents the number of the picture information to be analyzed called from the cache of the bus in unit time, x represents the number of the picture information to be analyzed in unit time by the first analysis node 210, and Δ represents a preset adjustment factor.
For example, the time period for the first analysis node 210 to call the information of the picture to be analyzed from the bus cache is 2S, that is, when the picture to be analyzed in the local cache module of the first analysis node 210 is lower than the second threshold, the picture to be analyzed is obtained from the cache of the bus every 2S, and 200 pieces of information of the picture to be processed are obtained every second.
Of course, when there are other services in the first analysis node 210 consuming the capabilities of the analysis module, then the above formula is satisfied:
n*m＝x+△-y；
wherein y is the analysis capability occupied by other services.
And, as a possible implementation manner, the cache of the bus includes a plurality of sets to store different types of picture information. For example, one of the sets is used to store the to-be-analyzed picture information related to face recognition, and the other set is used to store the to-be-analyzed picture information related to a vehicle, which is not limited in this application.
Thus, in the present application, S104-c comprises:
and determining a target set according to the category of the picture information to be analyzed, and storing the picture information to be analyzed into the target set.
That is, when the number of the to-be-analyzed picture information in the local cache reaches the first threshold, the first analysis node 210 stores the to-be-analyzed picture information processed by the decoding matting module into the target set of the bus cache according to the category.
It will be appreciated that the analysis nodes may operate by category of data being analyzed, for example, there are 10 analysis nodes total, of which 5 are used to analyze face data and the other 5 are used to analyze vehicle data. In actual use, when the analysis amount of the analysis node for analyzing the face data is lower than the second threshold, it may acquire the information to be analyzed from the set stored in association with the face data.
Meanwhile, as another implementation manner of the present application, please refer to fig. 7, and S104 further includes:
and S104-e, when the data volume to be analyzed of the first analysis node is smaller than a second threshold, the first analysis node acquires the data to be analyzed from the cache of the bus and analyzes the data to be analyzed, wherein the data to be analyzed is data sent by any one second analysis node 220 when the data volume to be analyzed is larger than the first threshold.
That is, in the present application, when the amount of data to be analyzed is greater than the first threshold, any one of the second analysis nodes 220 may also send the data to be analyzed to the cache of the bus. Further, when the data volume to be analyzed is smaller than the second threshold, the first analysis node 210 can also obtain the data volume to be analyzed from the cache of the bus, and store the data volume to be analyzed in the local cache module for processing.
In other words, in the present application, when the amount of data to be analyzed of any analysis node is large, the data to be analyzed may be sent to the cache of the bus, and then when the analysis node that processes the same type of data is relatively idle, the data to be analyzed may be obtained from the cache of the bus for analysis.
In addition, as another implementation manner of the present application, the analysis system 200 further includes a data access node 240, and the data access node 240 communicates with the first analysis node 210 and the second analysis node 220 through a bus respectively.
It should be noted that there is no precedence order between S104-e and S104-a to S104-c.
The data access node 240 is mainly used for analyzing the image data uploaded and collected by the system 200 access front end, other video image collection systems and the like, and is used for secondary intelligent analysis in the system 200.
On this basis, referring to fig. 8, S104 further includes:
and S104-f, when the data volume to be analyzed of the first analysis node and/or the second analysis node is smaller than a second threshold value, the first analysis node and/or the second analysis node acquires the data to be analyzed from the cache of the bus and analyzes the data to be analyzed, wherein the data to be analyzed is the data sent to the cache of the bus by the data access node.
That is, after the data access node 240 receives the picture information to be analyzed, the picture information to be analyzed is sent to the cache of the bus, and when the data amount to be analyzed of the first analysis node 210 is smaller than the second threshold, or the data amount to be analyzed of the second analysis node 220 is smaller than the second threshold, or the data amounts to be analyzed of the first analysis node 210 and the second analysis node 220 are simultaneously smaller than the second threshold, the data sent to the cache of the bus by the data access node 240 can be obtained from the cache of the bus and analyzed.
Meanwhile, it should be noted that, as another possible implementation manner in the present application, the analysis node may not include a decoding matting module, and on this basis, the analysis node can only obtain the data to be analyzed from the cache of the bus and perform analysis.
For example, if the first analysis node 210 does not include a decoding matting module, as a possible implementation manner, the second threshold is set to 1, so that the first analysis node 210 can always obtain data to be analyzed from the cache of the bus for processing.
It should be noted that there is no precedence between S104-f and S104-a to S104-c and S104-e.
It should be noted that, as a possible implementation manner, the analysis system 200 further includes a data center 250, please refer to fig. 9, and the method further includes:
and S106, each analysis node sends the analyzed data to the data center 250 through the bus.
In this application, each analysis node may directly place the result after the analysis is completed in the bus by way of the bus, for example, in a cache of the bus, and then the data center 250 pulls the relevant data in the bus and completes the processing such as storage. The data center 250 may be a collection of databases, big data service clusters or cloud storage, local hard disk storage, and the like.
In summary, the present application implements full-service load balancing of the analysis system 200 by setting a bus, and by means of the bus, a local cache, and the like. Meanwhile, the decoding module and the analysis module are separated and decoupled, so that decoding and analysis capacity resources of the analysis node are utilized to the maximum extent. And the picture cache after local decoding can be preferentially processed after the cache reaches the threshold value of the analysis node, so that the cost of pulling pictures from other parts of the bus is reduced.
Second embodiment
Referring to fig. 5, an analysis system 200 is further provided in the embodiment of the present application, where the analysis system 200 includes at least two analysis nodes, and the at least two analysis nodes communicate with each other through a bus; wherein the content of the first and second substances,
each analysis node is used for acquiring the data volume to be analyzed of the analysis node.
Each analysis node is also used for allocating the data volume to be analyzed through the cache of the bus according to the data volume to be analyzed.
And, the first analysis node 210 is configured to send the redundant data to be analyzed to the cache of the bus when the amount of the data to be analyzed is greater than the first threshold.
Meanwhile, the first analysis node 210 and/or the second analysis node 220 are configured to obtain the data to be analyzed from the cache of the bus and analyze the data when the amount of the data to be analyzed is lower than a second threshold.
Of course, the analysis system 200 may further include a management node 230, a data access node 240, and a data center 250, wherein the management node 230 is configured to connect with each analysis node and configure each analysis node. The data access node 240 is used for analyzing the image data uploaded and collected by the access front end and other video image collection systems in the system 200, and for secondary intelligent analysis in the system 200. The data center 250 may be a collection of databases, big data service clusters, and cloud storage, where the databases and big data service clusters are mainly used for data record storage. Since the first embodiment has already described the data analysis method in detail, the present application is not repeated herein.
In summary, the present application provides a data analysis method, which is applied to an analysis system, where the analysis system includes at least two analysis nodes, the at least two analysis nodes communicate via a bus, and each analysis node obtains its own data volume to be analyzed; and then, each analysis node allocates the data volume to be analyzed through the cache of the bus according to the data volume to be analyzed. According to the method and the device, each analysis node can allocate the data volume to be analyzed of the analysis node through the cache of the bus, so that the balance of the data volume to be analyzed can be effectively realized for each analysis node, and the maximum calculation power of the whole system can be exerted.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Claims (10)
1. A method of data analysis, the method being applied to an analysis system comprising at least two analysis nodes communicating via a bus, the method comprising:
each analysis node acquires the data volume to be analyzed of the analysis node;
and allocating the data volume to be analyzed of each analysis node through the cache of the bus according to the data volume to be analyzed of each analysis node.
2. The data analysis method of claim 1, wherein the at least two analysis nodes include at least a first analysis node and a second analysis node, and each analysis node allocates the own data volume to be analyzed through the cache of the bus according to the own data volume to be analyzed, including:
when the data volume to be analyzed of the first analysis node is larger than a first threshold value, the first analysis node sends redundant data to be analyzed to a cache of the bus;
and when the data volume to be analyzed of the first analysis node and/or the second analysis node is lower than a second threshold value, the first analysis node and/or the second analysis node acquires the data to be analyzed from the cache of the bus and analyzes the data.
3. The data analysis method of claim 2, wherein when the amount of data to be analyzed of the first analysis node is greater than a first threshold, the step of the first analysis node sending excess data to be analyzed to the cache of the bus comprises:
the first analysis node acquires a video stream to be analyzed and preprocesses the video stream to be analyzed to acquire preprocessed picture information to be analyzed;
the first analysis node stores the picture information to be analyzed into a local cache;
and when the number of the picture information to be analyzed in the local cache reaches a first threshold value, the first analysis node sends redundant picture information to be analyzed to the cache of the bus.
4. The data analysis method of claim 3, wherein the cache of the bus includes a plurality of sets to store different categories of picture information; when the number of the to-be-analyzed picture information in the local cache reaches a first threshold value, the step of sending the redundant to-be-analyzed picture information to the cache of the bus by the first analysis node comprises the following steps:
and determining a target set according to the category of the picture information to be analyzed, and storing the picture information to be analyzed into the target set.
5. The data analysis method of claim 2, wherein the first threshold and the second threshold satisfy a relationship:
LT_high–LT_low≥n*m；
n*m＝x+△；
wherein LT _ high represents a first threshold value, and LT _ low represents a second threshold value; n represents the time period for calling the picture information to be analyzed from the cache of the bus; m represents the number of the picture information to be analyzed called from the cache of the bus in unit time, x represents the number of the picture information to be analyzed of the first analysis node in unit time, and delta represents a preset adjusting factor.
6. The data analysis method of claim 1, wherein the at least two analysis nodes include at least a first analysis node and a second analysis node, and each analysis node allocates the own data volume to be analyzed through the cache of the bus according to the own data volume to be analyzed, including:
when the data volume to be analyzed of the first analysis node is smaller than a second threshold, the first analysis node acquires the data to be analyzed from the cache of the bus and analyzes the data to be analyzed, wherein the data to be analyzed is data sent by any one of the second analysis nodes when the data volume to be analyzed is larger than the first threshold.
7. The data analysis method of claim 1, wherein the analysis system further includes a data access node, the at least two analysis nodes include at least a first analysis node and a second analysis node, the data access node is respectively in communication with the first analysis node and the second analysis node through the bus, and the step of allocating, by each of the analysis nodes, the own data volume to be analyzed through the cache of the bus according to the own data volume to be analyzed includes:
when the data volume to be analyzed of the first analysis node and/or the second analysis node is smaller than a second threshold value, the first analysis node and/or the second analysis node obtains the data to be analyzed from the cache of the bus and analyzes the data to be analyzed, wherein the data to be analyzed is the data sent to the cache of the bus by the data access node.
8. The data analysis method of claim 1, wherein the analysis system further comprises a data center, the method further comprising:
and each analysis node sends the analyzed data to the data center through a bus.
9. An analysis system, comprising at least two analysis nodes, the at least two analysis nodes communicating over a bus;
each analysis node is used for acquiring the data volume to be analyzed of the analysis node;
each analysis node is further configured to allocate the data volume to be analyzed of the analysis node through the cache of the bus according to the data volume to be analyzed of the analysis node.
10. The analytical system of claim 9, wherein the at least two analytical nodes include a first analytical node and at least one second analytical node;
the first analysis node is used for sending redundant data to be analyzed to the cache of the bus when the data volume to be analyzed is larger than a first threshold value;
the first analysis node and/or the second analysis node are/is used for acquiring the data to be analyzed from the cache of the bus and analyzing the data when the data volume to be analyzed of the first analysis node and/or the second analysis node is lower than a second threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911362443.4A CN113051957A (en) | 2019-12-26 | 2019-12-26 | Data analysis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911362443.4A CN113051957A (en) | 2019-12-26 | 2019-12-26 | Data analysis method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113051957A true CN113051957A (en) | 2021-06-29 |
Family
ID=76505935
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911362443.4A Pending CN113051957A (en) | 2019-12-26 | 2019-12-26 | Data analysis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113051957A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741912A (en) * | 2009-12-30 | 2010-06-16 | 中兴通讯股份有限公司 | Method, network apparatus and distributed network system for processing computation task |
| US20110286743A1 (en) * | 2010-05-24 | 2011-11-24 | Mclaren Moray | Flow-control methods and systems for multibus systems |
| CN102906726A (en) * | 2011-12-09 | 2013-01-30 | 华为技术有限公司 | Co-processing accelerating method, device and system |
| US20140033220A1 (en) * | 2011-05-10 | 2014-01-30 | International Business Machines Corporation | Process grouping for improved cache and memory affinity |
| CN110333945A (en) * | 2019-05-09 | 2019-10-15 | 成都信息工程大学 | A kind of dynamic load balancing method, system and terminal |
-
2019
- 2019-12-26 CN CN201911362443.4A patent/CN113051957A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741912A (en) * | 2009-12-30 | 2010-06-16 | 中兴通讯股份有限公司 | Method, network apparatus and distributed network system for processing computation task |
| US20110286743A1 (en) * | 2010-05-24 | 2011-11-24 | Mclaren Moray | Flow-control methods and systems for multibus systems |
| US20140033220A1 (en) * | 2011-05-10 | 2014-01-30 | International Business Machines Corporation | Process grouping for improved cache and memory affinity |
| CN102906726A (en) * | 2011-12-09 | 2013-01-30 | 华为技术有限公司 | Co-processing accelerating method, device and system |
| CN110333945A (en) * | 2019-05-09 | 2019-10-15 | 成都信息工程大学 | A kind of dynamic load balancing method, system and terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11132224B2 (en) | Fine granularity real-time supervision system based on edge computing | |
| CN102222213B (en) | Distributed vision computing method based on open type Web Service framework | |
| US20240119089A1 (en) | Cascaded video analytics for edge computing | |
| CN114255432A (en) | Video stream processing method, device, electronic equipment, storage medium and system | |
| CN112446395A (en) | Network camera, video monitoring system and method | |
| CN109919033B (en) | Self-adaptive city people searching method based on edge calculation | |
| CN110740293A (en) | data transmission method, system, device and computer storage medium | |
| CN117437579A (en) | Video processing method, device, equipment and medium | |
| CN114584758A (en) | City-level monitoring video quality assessment method and system | |
| CN106372171A (en) | Real-time data processing method of monitoring platform | |
| CN113051957A (en) | Data analysis method and system | |
| CN112988337A (en) | Task processing system, method, device, electronic equipment and storage medium | |
| CN106791648B (en) | Monitoring video analysis method and device | |
| CN117492989A (en) | AI calculates power adjustable intelligent video analysis system | |
| CN115567563B (en) | Comprehensive transportation hub monitoring and early warning system based on end edge cloud and control method thereof | |
| CN112418062A (en) | Face recognition method, face recognition system, electronic equipment and storage medium | |
| KR102477845B1 (en) | System and method of processing image data for batch processing of big data and computer program for the same | |
| CN110543497B (en) | High-real-time deployment and control solution method and system | |
| CN112669353B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN110321857B (en) | Accurate passenger group analysis method based on edge calculation technology | |
| CN113438417A (en) | Method, system, medium and device for capturing object to be identified by video | |
| CN116132623A (en) | Intelligent analysis method, system and equipment based on video monitoring | |
| CN112948779A (en) | Front-end-acquisition-based multi-stage shared portrait big data system | |
| CN113536878B (en) | Intelligent analysis system and method | |
| CN112925741B (en) | Heterogeneous computing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |