CN112733901A - Structured action classification method and device based on federal learning and block chain - Google Patents

Structured action classification method and device based on federal learning and block chain Download PDF

Info

Publication number
CN112733901A
CN112733901A CN202011621661.8A CN202011621661A CN112733901A CN 112733901 A CN112733901 A CN 112733901A CN 202011621661 A CN202011621661 A CN 202011621661A CN 112733901 A CN112733901 A CN 112733901A
Authority
CN
China
Prior art keywords
action classification
classification result
structured
request
action
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011621661.8A
Other languages
Chinese (zh)
Other versions
CN112733901B (en
Inventor
蔡亮
李伟
匡立中
邱炜伟
张帅
李吉明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Qulian Technology Co Ltd
Original Assignee
Hangzhou Qulian Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Qulian Technology Co Ltd filed Critical Hangzhou Qulian Technology Co Ltd
Priority to CN202011621661.8A priority Critical patent/CN112733901B/en
Publication of CN112733901A publication Critical patent/CN112733901A/en
Application granted granted Critical
Publication of CN112733901B publication Critical patent/CN112733901B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/94—Hardware or software architectures specially adapted for image or video understanding
    • G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Bioethics (AREA)
  • Medical Informatics (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)

Abstract

The invention belongs to the technical field of block chains, and provides a structured action classification method and a device based on federal learning and block chains, which synchronize equipment classified for the same image characteristic request in the action classification request by receiving image characteristics of a plurality of edge computing devices of an information uploading block chain and an action classification request aiming at the image characteristics to obtain an image characteristic training loss function model, obtain an action classification result of a single image characteristic close to a real value, use a boundary structured output learning framework to model the structured output of the action classification result so as to obtain a fusion action classification result, send the fusion action classification result to a request device, thereby only uploading the image characteristics and downloading the identification result between the device and a server, ensuring privacy, reducing transmission cost and collecting training data on different edge computing devices, the accuracy of the model is improved, and the block chain is used for protecting the model from being abused by unverified equipment.

Description

Structured action classification method and device based on federal learning and block chain
Technical Field
The present application relates to the field of blockchain technology, and in particular, to a method, a system, an apparatus, a computer device, and a computer-readable storage medium for structured action classification based on federated learning and blockchain.
Background
Currently, an action classification method based on a two-dimensional image is of great interest, for example, human action classification is applicable to the fields of security protection and the like. However, motion-emitting subjects (e.g., human bodies) are non-rigid bodies and are often in complex scenes. Therefore, a plurality of devices are used for acquiring images from different angles to obtain a comprehensive classification result, and the method is an important method for improving the classification effect.
However, in the prior art, the original information such as the main image is transmitted between the edge computing device and the server, so that privacy is easily revealed, and the transmission cost is high; because no proper training model is available, the precision of action classification is not high, no consensus mechanism is established among devices through a block chain, action classification can be requested without classification authority, the load of a server is increased, and the server is easily abused.
In summary, the prior art has the technical problems of low security, low classification accuracy, high transmission cost, easy abuse of the server, and the like.
Disclosure of Invention
In order to solve the technical problems, the invention provides a structured action classification method, a system, a device, a computer device and a computer readable storage medium based on federal learning and a blockchain.
A structured action classification method based on federated learning and blockchain comprises the following steps:
receiving image characteristics of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image characteristics;
synchronizing the devices which are classified aiming at the same image feature request in the action classification request;
obtaining the image feature training loss function model to obtain a motion classification result of a single image feature close to a real value;
and modeling the structured output of the action classification result by using a boundary structured output learning framework so as to obtain a fusion action classification result and sending the fusion action classification result to the request equipment.
A structured action classification device based on federal learning and block chain comprises:
the receiving module is used for receiving image features of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image features;
the synchronization module is used for synchronizing the devices which are classified according to the same image feature request in the action classification request;
the training module is used for obtaining the image feature training loss function model and obtaining a motion classification result of a single image feature close to a real value;
and the modeling module is used for modeling the structured output of the action classification result by using a boundary structured output learning frame so as to obtain a fusion action classification result and sending the fusion action classification result to the request equipment.
A structured action classification system based on federated learning and blockchains, comprising: a plurality of edge computing devices, servers, and blockchains;
the plurality of edge computing devices communicate over the blockchain; a receiving module, a synchronization module, a training module and a modeling module are operated in the server;
the receiving module receives image features of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image features;
the synchronization module synchronizes the devices which are classified according to the same image feature request in the action classification request;
the training module acquires the image feature training loss function model to obtain a motion classification result of a single image feature close to a real value;
and the modeling module uses a boundary structured output learning framework to model the structured output of the action classification result so as to obtain a fusion action classification result and send the fusion action classification result to the request equipment.
A computer device comprising a memory and a processor, the memory storing a computer program, the computer program performing an implementable method in the processor.
A computer-readable storage medium storing a computer program which, when executed in a processor, implements the above-described method.
The invention provides a structured action classification method, a device and a system based on federal learning and block chains, which synchronize equipment for classifying the same image feature request in an action classification request by receiving image features of a plurality of edge computing devices of an information uploading block chain and the action classification request aiming at the image features, acquire an image feature training loss function model, obtain an action classification result of a single image feature close to a real value, use a boundary structured output learning framework to model the structured output of the action classification result, so as to obtain a fused action classification result and send the fused action classification result to a request device, thereby only uploading the image features and downloading the identification result between the edge computing devices and a server, not transmitting information of original images and the like, ensuring privacy, and ensuring that the data volume of the features is smaller than that of the original images, the method has the advantages of reducing transmission cost, collecting training data on different edge computing devices, improving the precision of the model, protecting the model by using a block chain, and ensuring that the model is not abused by unverified devices.
Drawings
FIG. 1 is a block diagram of a structured action classification system based on federated learning and blockchains according to an embodiment;
fig. 2 is a schematic flowchart of a structured action classification method based on federal learning and a blockchain according to an embodiment;
FIG. 3 is a schematic diagram of a similar image frame;
fig. 4 is a schematic structural diagram of a structured action classification apparatus based on federal learning and a blockchain according to an embodiment;
fig. 5 is a schematic block diagram of a computer device according to an embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it should be understood that the described embodiments are a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The method for classifying structured actions based on federated learning and blockchain provided by the present embodiment can be applied to the application environment shown in fig. 1, in which the edge computing device communicates with a server through a network, and the server operates in a blockchain system. Among other things, edge computing devices may be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server may be implemented as a stand-alone server or as a server cluster of multiple servers.
In an embodiment, as shown in fig. 2, a structured action classification method based on federal learning and blockchain is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:
s1, receiving image features of a plurality of edge computing devices of the information uploading block chain and action classification requests aiming at the image features;
s2, synchronizing the devices which are classified according to the same image feature request in the action classification request;
s3, obtaining an image feature training loss function model, and obtaining a motion classification result of a single image feature close to a real value;
and S4, modeling the structural output of the action classification result by using a boundary structural output learning framework to obtain a fusion action classification result, and sending the fusion action classification result to the request device.
In the embodiment, by receiving image features of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image features, synchronizing the devices which are classified aiming at the same image feature request in the action classification requests to obtain an image feature training loss function model, obtaining an action classification result of a single image feature close to a real value, and modeling structured output of the action classification result by using a boundary structured output learning framework to obtain a fused action classification result and send the fused action classification result to a request device, so that only image features are uploaded and recognition results are downloaded between the edge computing devices and a server, information such as original images and the like is not transmitted, privacy is guaranteed, the data volume of the features is smaller than that of the original images, the transmission cost is reduced, training data on different edge computing devices are collected, and the precision of the model is improved, the use of a blockchain protection model ensures that the model is not abused by unauthorized devices.
In step S1, the edge computing device sends its own device serial number to the server, where the serial number is verified on the edge computing device information chain, and the verified device may send an image feature that needs to be identified by an action and a request for identification to the server.
It should be noted that, the edge computing device may upload information to the blockchain through the server, where the uploaded information includes, but is not limited to, the device serial number and the contribution data amount. In this way, when the image features and the identification request which need to be subjected to action identification are sent to the server, the server firstly initiates information verification to the block chain, confirms the identity and data volume contribution of the request equipment, and responds to the request of the request equipment or rejects the request of the request equipment. Therefore, authenticity and privacy of the uplink information can be guaranteed through the block chain, and abuse of the server can be avoided.
It should be noted that the image data provided by the edge computing device to the server is image feature data, and the image feature data may be feature-extracted data corresponding to the image data one to one, and has an advantage of a small data amount compared to an original copy of the image data, so that transmission cost can be effectively reduced in transmission. Wherein the feature extraction may be performed by prior art means, e.g. by a gradient direction histogram algorithm.
It is understood that the data source provided by the edge computing device may be captured using any camera, or may be directly input into an existing picture, video screen, etc.
In step S2, a federate learning method is used to synchronize information of a plurality of requesting devices that initiated identification requests for the same object in step S1, for example, by device time stamps, indicating that the identification requests were initiated for the same identification object. Therefore, the technical effects of ensuring the privacy of the training data and collecting more training data can be achieved.
It should be noted that federal learning is a prior art, and the main idea thereof is to construct a machine learning model based on a data set distributed on a plurality of devices, and simultaneously prevent data leakage.
In step S3, an image feature training loss function model is obtained by using a loss function model technique, and a motion classification result of a single image feature close to a real value is obtained. The loss function model can be specifically characterized as:
Figure BDA0002874095070000065
wherein the function
Figure BDA0002874095070000066
Representing classification output
Figure BDA0002874095070000067
And real value
Figure BDA0002874095070000068
Loss in between; assuming that an action is represented by a frame or a series of key frames containing action capture information, assuming a query set Q and a category set C; for each query sample qi＝xi，yi∈Q，xie.X is the image feature related to the query sample, XiIs D, i.e. xi∈RD；
Figure BDA0002874095070000061
Is an output list containing the | C | category and its confidence.
It should be noted that the data result output by the loss function model processing is the action classification result of a single image feature, the image feature is related to the query sample, the set of the query sample represents the image feature for model training, and the larger the data size is, the more beneficial to reducing the classification output
Figure BDA0002874095070000062
And real value
Figure BDA0002874095070000063
Loss between, realize classification output
Figure BDA0002874095070000064
And real value
Figure BDA0002874095070000071
The method has the advantages of infinite approximation and more accurate technical effect of action classification result of single image characteristics.
It should be noted that the real value is a preset value or an ideal value of the output data of the loss function model, and the training of the loss function model aims to make the motion classification result of the image feature closer to the real value.
In step S4, the structured output of the action classification result is modeled by using a boundary structured output learning framework to obtain a fused action classification result, which is sent to the requesting device. Wherein the boundary structured output learning framework can be characterized as:
Figure RE-GDA0002982733740000072
wherein w is a model parameter vector; f (x)i，yi(ii) a w) is defined as a linear function of w; f (x)i，yi；w)＝wTΨ(xi，yi)， Ψ(xi，yi) Is a joint feature map, maps feature xiAnd confidence prediction yiMapping to a real value; Ψ (x)i,yi) Is defined as
Figure RE-GDA0002982733740000073
Is deformed into
Figure RE-GDA0002982733740000074
Figure BDA0002874095070000075
Is a query sample qiList of action confidences.
It should be noted that, on the basis of the motion classification result of a single image feature, the boundary structured output learning framework may implement fusion of the classification result to process continuous image feature classification, for example, processing a video image feature.
Further, referring to fig. 3, since in the learning process of the conventional method, a training sample is generally assigned a single label. In this way, the different classes are trained separately. However, some training samples in different classes may be very similar. Taking the image frame illustrated in fig. 3 as an example, where two frames, one from thread and the other from Walking, look very similar in both raw video and motion data, simply dividing the training samples by category may result in a cluttered classifier. To avoid such clutter, the boundary structured output learning framework may be enhanced by applying local constraints. Wherein, in order to preserve locality, a hypergraph can be constructed to describe the correlation of samples. The specific implementation can be as follows:
redefining the boundary structured output learning framework as:
Figure BDA0002874095070000081
where α is used to balance the structured dependence wTΨ(xi，yi) And local constraint terms
Figure BDA0002874095070000082
The trade-off parameter of (1); minimizing the local constraint term is considered as a hypergraph based on manifold regularization, so that similar training samples obtain similar confidence.
It should be noted that, based on the regularization theory, the definition of the laplacian matrix L may be introduced, and the redefined boundary structured output learning framework is characterized as:
Figure BDA0002874095070000083
wherein c ═ z ((z)i-yi) /(2. alpha.). The solution can be divided into calculation and optimization. The laplacian matrix L can be calculated in two steps.
Firstly, optimizing the parts: in the subdivision optimization stage, a slice of a hypergraph may be defined as being bounded by vertices associated by hyper-edges. Thus, the slice of the hypergraph is defined in this learning process as:
Figure BDA0002874095070000084
for a slice of a hypergraph, the calculation is needed:
Figure BDA0002874095070000085
this means that two vertices in the set of vertices contained by a super-edge e are randomly selected and summed:
Figure BDA0002874095070000086
by the expansion formula:
Figure BDA0002874095070000087
and combining terms to obtain a slice optimization for each excess edge:
Figure BDA0002874095070000088
wherein matrix E is defined as:
Figure BDA0002874095070000089
i is an nxn identity matrix.
Second, overall alignment: in the hypergraph, the weight of a hyperedge is computed by summing the similarity scores of all pairs of vertices contained in the hyperedge. The similarity score of any pair of vertices is defined as the distance of the image features:
Figure BDA0002874095070000091
where feat (u) represents the image feature vector for vertex u, dist (x, y) is typically set to a distance of L2. In the super-edge weighting matrix, the Laplace matrix of the multi-view super graph can be formed by pairing
Figure BDA0002874095070000092
The image slice optimization summation of all the superedges defined in (1) to calculate:
Figure BDA0002874095070000093
to solve the optimization
Figure RE-GDA0002982733740000093
Lagrange multiplier lambda can be introduced to eliminate equality constraint and obtain
Figure RE-GDA0002982733740000094
Wherein I is Ni×NiAn identity matrix. λ can be selected from
Figure RE-GDA0002982733740000095
The solution in the dual problem of (2) is:
Figure RE-GDA0002982733740000096
wherein eta is UTc and Li＝U∑UTL with U as a feature vectoriThe characteristic of (A) is decomposed,
Figure RE-GDA0002982733740000097
is the corresponding characteristic value λ, noted
Figure RE-GDA0002982733740000098
The analysis (2) is common knowledge, and an optimum can be obtained by using, for example, a binary search technique. Specific procedures can be found in documents B.Geng, Y.Yang, C.xu, and X. -S.Hua, "Content-aware linking for visual search," in IEEE International Conference on Computer Vision and Pattern recognition. IEEE Press, June 2010, pp.3400-3407.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, referring to fig. 4, a structured action classification device based on federated learning and blockchain is provided, and the device corresponds to the structured action classification method based on federated learning and blockchain in one-to-one correspondence in the above embodiment. As shown in fig. 4, the apparatus includes a receiving module 1, a synchronization module 2, a training module 3, and a modeling module 4.
The system comprises a receiving module 1, a judging module and a judging module, wherein the receiving module is used for receiving image characteristics of a plurality of edge computing devices for establishing block uplink and action classification requests aiming at the image characteristics;
the synchronization module 2 is used for synchronizing the devices which are classified according to the same image feature request in the action classification request;
the training module 3 is used for obtaining an image feature training loss function model and obtaining a motion classification result of a single image feature close to theoretical output;
and the modeling module 4 is used for modeling the structured output of the action classification result by using a boundary structured output learning framework so as to obtain a fusion action classification result and sending the fusion action classification result to the request equipment.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules expressly listed, but may include other steps or modules not expressly listed or inherent to such process, method, article, or apparatus, and that the division of the blocks presented herein is merely a logical division and may be implemented in a practical application in a further manner.
It should be further noted that, for specific definitions and descriptions of the above device, reference may be made to the definitions and descriptions of the federal learning and block chain based structured action classification method in the foregoing, and details are not repeated here. The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, referring to fig. 4 and 5, a computer device is provided, which may be a server running the receiving module 1, the synchronization module 2, the training module 3, and the modeling module 4. Wherein the internal structure of the computer device can be as shown in figure 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data involved in a structured action classification method based on federal learning and block chains. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a structured action classification method based on federated learning and blockchains.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor when executing the computer program implements the steps of the federated learning and blockchain-based structured action classification method in the above-described embodiments, such as step 1 to step 4 shown in fig. 2 and an extension of other extensions and related steps of the method. Alternatively, the processor, when executing the computer program, implements the functions of each module/unit of the structured action classification apparatus based on the federal learning and block chain in the above embodiment, for example, the functions of the modules 1 to 4 shown in fig. 4. To avoid repetition, further description is omitted here.
The Processor may be a Central Processing Unit (CPU), or may be other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center of the computer device and the various parts of the overall computer device being connected by various interfaces and lines.
The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the computer device by running or executing the computer programs and/or modules stored in the memory, as well as by invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, video data, etc.) created according to the use of the cellular phone, etc.
The memory may be integrated in the processor or may be provided separately from the processor.
In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the steps of the federated learning and blockchain based structured action classification method in the above-described embodiments, such as step 1 through step 4 and other extensions of the method and extensions of related steps shown in fig. 2. Alternatively, the computer program, when executed by the processor, implements the functions of each module/unit of the structured action classification apparatus based on the federal learning and blockchain in the above embodiments, such as the functions of the modules 1 to 4 shown in fig. 4. To avoid repetition, further description is omitted here.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and simplicity of description, the foregoing functional units and modules are merely illustrated in terms of their division, and in practical applications, the foregoing functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the device may be divided into different functional units or modules to implement all or part of the above-described functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims (10)

1. A structured action classification method based on federated learning and block chains is characterized by comprising the following steps:
receiving image characteristics of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image characteristics;
synchronizing the devices which are classified aiming at the same image feature request in the action classification request;
obtaining the image feature training loss function model to obtain a motion classification result of a single image feature close to a real value;
and modeling the structured output of the action classification result by using a boundary structured output learning framework so as to obtain a fusion action classification result and sending the fusion action classification result to the request equipment.
2. The method of claim 1, further comprising:
applying local constraints to the boundary structured output learning framework to enhance the boundary structured output learning framework.
3. The method of claim 1, wherein the information of the upload block chain comprises a device serial number and a contribution data amount.
4. The method of claim 2, wherein the loss function model is characterized by:
Figure FDA0002874095060000011
wherein the function
Figure FDA0002874095060000012
Representing classification output
Figure FDA0002874095060000013
And real value
Figure FDA0002874095060000014
Loss in between; assuming that an action is represented by a frame or a series of key frames containing action capture information, assuming a query set Q and a category set C; for each query sample qi＝xi，yi∈Q，xie.X is the image feature related to the query sample, XiIs D, i.e. xi∈RD；
Figure FDA0002874095060000015
Is an output list containing the | C | category and its confidence.
5. The method of claim 4, wherein the boundary structured output learning framework is characterized by:
Figure RE-FDA0002982733730000016
wherein W is a model parameter vector; f (x)i，yi(ii) a w) is defined as a linear function of w; f (x)i，yi；w)＝wTΨ(xi，yi)，Ψ(xi，yi) Is a joint feature map, maps feature xiAnd confidence prediction yiMapping to a real value; Ψ (x)i,yi) Is defined as
Figure RE-FDA0002982733730000021
Is deformed into
Figure RE-FDA0002982733730000022
Is a query sample qiList of action confidences.
6. The method of claim 5, comprising redefining:
Figure FDA0002874095060000024
where α is used to balance the structured dependence wTΨ(xi，yi) And local constraint terms
Figure FDA0002874095060000025
The trade-off parameter of (1); minimizing the local constraint term is considered as a hypergraph based on manifold regularization, so that similar training samples obtain similar confidence.
7. A structured action classification device based on federal learning and block chain is characterized by comprising:
the receiving module is used for receiving image characteristics of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image characteristics;
the synchronization module is used for synchronizing the devices which are classified according to the same image feature request in the action classification request;
the training module is used for obtaining the image feature training loss function model and obtaining a motion classification result of a single image feature close to a real value;
and the modeling module is used for modeling the structured output of the action classification result by using a boundary structured output learning frame so as to obtain a fusion action classification result and sending the fusion action classification result to the request equipment.
8. A structured action classification system based on federated learning and blockchains, comprising: a plurality of edge computing devices, servers, and blockchains;
the plurality of edge computing devices communicate over the blockchain; a receiving module, a synchronization module, a training module and a modeling module are operated in the server;
the receiving module receives image characteristics of a plurality of edge computing devices of an information uploading block chain and action classification requests aiming at the image characteristics;
the synchronization module synchronizes the devices which are classified aiming at the same image characteristic request in the action classification request;
the training module acquires the image feature training loss function model to obtain a motion classification result of a single image feature close to a real value;
and the modeling module uses a boundary structured output learning framework to model the structured output of the action classification result so as to obtain a fusion action classification result and send the fusion action classification result to the request equipment.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the computer program is operative to perform the method of any of claims 1-6 in the processor.
10. A computer-readable storage medium storing a computer program, characterized in that the computer program is executed in a processor to implement the method of any of claims 1-6.
CN202011621661.8A 2020-12-30 2020-12-30 Structured action classification method and device based on federal learning and blockchain Active CN112733901B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011621661.8A CN112733901B (en) 2020-12-30 2020-12-30 Structured action classification method and device based on federal learning and blockchain

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011621661.8A CN112733901B (en) 2020-12-30 2020-12-30 Structured action classification method and device based on federal learning and blockchain

Publications (2)

Publication Number Publication Date
CN112733901A true CN112733901A (en) 2021-04-30
CN112733901B CN112733901B (en) 2024-01-12

Family

ID=75608246

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011621661.8A Active CN112733901B (en) 2020-12-30 2020-12-30 Structured action classification method and device based on federal learning and blockchain

Country Status (1)

Country Link
CN (1) CN112733901B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115424079A (en) * 2022-09-30 2022-12-02 深圳市大数据研究院 Image classification method based on federal edge learning and related equipment
CN115701071A (en) * 2021-07-16 2023-02-07 中移物联网有限公司 Model training method and device, electronic equipment and storage medium
CN116502732A (en) * 2023-06-29 2023-07-28 杭州金智塔科技有限公司 Federal learning method and system based on trusted execution environment

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020199693A1 (en) * 2019-03-29 2020-10-08 中国科学院深圳先进技术研究院 Large-pose face recognition method and apparatus, and device
CN111931242A (en) * 2020-09-30 2020-11-13 国网浙江省电力有限公司电力科学研究院 Data sharing method, computer equipment applying same and readable storage medium
CN112100295A (en) * 2020-10-12 2020-12-18 平安科技（深圳）有限公司 User data classification method, device, equipment and medium based on federal learning

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020199693A1 (en) * 2019-03-29 2020-10-08 中国科学院深圳先进技术研究院 Large-pose face recognition method and apparatus, and device
CN111931242A (en) * 2020-09-30 2020-11-13 国网浙江省电力有限公司电力科学研究院 Data sharing method, computer equipment applying same and readable storage medium
CN112100295A (en) * 2020-10-12 2020-12-18 平安科技（深圳）有限公司 User data classification method, device, equipment and medium based on federal learning

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
徐梦炜;刘渊强;黄康;刘?哲;黄罡;: "面向移动终端智能的自治学习系统", 软件学报, no. 10 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115701071A (en) * 2021-07-16 2023-02-07 中移物联网有限公司 Model training method and device, electronic equipment and storage medium
CN115424079A (en) * 2022-09-30 2022-12-02 深圳市大数据研究院 Image classification method based on federal edge learning and related equipment
CN115424079B (en) * 2022-09-30 2023-11-24 深圳市大数据研究院 Image classification method based on federal edge learning and related equipment
CN116502732A (en) * 2023-06-29 2023-07-28 杭州金智塔科技有限公司 Federal learning method and system based on trusted execution environment
CN116502732B (en) * 2023-06-29 2023-10-20 杭州金智塔科技有限公司 Federal learning method and system based on trusted execution environment

Also Published As

Publication number Publication date
CN112733901B (en) 2024-01-12

Similar Documents

Publication Publication Date Title
WO2019228317A1 (en) Face recognition method and device, and computer readable medium
WO2021135499A1 (en) Damage detection model training and vehicle damage detection methods, device, apparatus, and medium
CN110580482B (en) Image classification model training, image classification and personalized recommendation method and device
CN112733901B (en) Structured action classification method and device based on federal learning and blockchain
WO2021022521A1 (en) Method for processing data, and method and device for training neural network model
CN112115783A (en) Human face characteristic point detection method, device and equipment based on deep knowledge migration
WO2021114612A1 (en) Target re-identification method and apparatus, computer device, and storage medium
CN111091075B (en) Face recognition method and device, electronic equipment and storage medium
WO2022179581A1 (en) Image processing method and related device
US11816877B2 (en) Method and apparatus for object detection in image, vehicle, and robot
CN111680675B (en) Face living body detection method, system, device, computer equipment and storage medium
WO2021031704A1 (en) Object tracking method and apparatus, computer device, and storage medium
WO2022052782A1 (en) Image processing method and related device
CN112926654A (en) Pre-labeling model training and certificate pre-labeling method, device, equipment and medium
Lu et al. Rethinking prior-guided face super-resolution: A new paradigm with facial component prior
CN112561973A (en) Method and device for training image registration model and electronic equipment
WO2022179603A1 (en) Augmented reality method and related device thereof
CN111652245B (en) Vehicle contour detection method, device, computer equipment and storage medium
CN112801134A (en) Gesture recognition model training and distributing method and device based on block chain and image
CN112348008A (en) Certificate information identification method and device, terminal equipment and storage medium
CN115115552B (en) Image correction model training method, image correction device and computer equipment
CN111291611A (en) Pedestrian re-identification method and device based on Bayesian query expansion
WO2022257433A1 (en) Processing method and apparatus for feature map of image, storage medium, and terminal
CN115995079A (en) Image semantic similarity analysis method and homosemantic image retrieval method
Fernandes et al. Matching images captured from unmanned aerial vehicle

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant