CN117742974A - Data processing method, device, storage medium, electronic equipment and system - Google Patents
Data processing method, device, storage medium, electronic equipment and system Download PDFInfo
- Publication number
- CN117742974A CN117742974A CN202410186426.4A CN202410186426A CN117742974A CN 117742974 A CN117742974 A CN 117742974A CN 202410186426 A CN202410186426 A CN 202410186426A CN 117742974 A CN117742974 A CN 117742974A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- processing device
- determining
- memory module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title description 3
- 238000012545 processing Methods 0.000 claims abstract description 394
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000004590 computer program Methods 0.000 claims description 14
- 238000011161 development Methods 0.000 abstract description 11
- 238000004891 communication Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 12
- 210000002569 neuron Anatomy 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 238000013136 deep learning model Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Landscapes
- Stored Programmes (AREA)
Abstract
The present disclosure relates to a method, apparatus, storage medium, electronic device, and system for data processing; determining the data type of the operation data of a preset network model; determining a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device; processing the operational data by the target processing device; through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a method, an apparatus, a storage medium, an electronic device, and a system for data processing.
Background
At present, the large computer model has good development prospect in various industries. Large models refer to deep learning models with tens or even hundreds of millions of parameters, which present significant challenges in data operation and processing because their data magnitude can reach the GB level or TB level.
In the related art, the GPU (Graphic Process Unit, image processor) and the CPU (Central Process Unit, central processing unit) can jointly perform data processing of a large model, so as to reduce the running cost. However, in the process of data processing, the GPU and the CPU need to run different types of computer programs, resulting in poor flexibility in the data processing process, improving development cost, and reducing work efficiency.
Disclosure of Invention
The disclosure aims to provide a data processing method, a data processing device, a storage medium, electronic equipment and a data processing system.
To achieve the above object, in a first aspect, the present disclosure provides a method of data processing, the method comprising:
determining the data type of operation data of a preset network model;
determining a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
and processing the operation data through the target processing device.
Optionally, determining the data type of the operation data of the preset network model includes:
and determining the data type of the operation data through a preset data processing algorithm.
Optionally, the determining, according to the data type, a target processing device corresponding to the data type from a plurality of processing devices includes:
determining a candidate processing device from a plurality of processing devices according to the data type;
the target processing device is determined from the candidate processing devices.
Optionally, the data type includes hot spot data or non-hot spot data; the determining candidate processing means from the plurality of processing means according to the data type comprises:
in the case that the data type is determined to be the hot spot data, taking the first processing device of the plurality of processing devices as the candidate processing device; or,
and in the case that the data type is determined to be the non-hot spot data, taking the second processing device in the plurality of processing devices as the candidate processing device.
Optionally, the determining the target processing device from the candidate processing devices includes:
determining the storage capacity of each candidate processing device according to a preset processing sequence;
and taking the candidate processing device with the storage capacity larger than or equal to a preset capacity threshold as the target processing device.
In a second aspect, the present disclosure provides an apparatus for data processing, the apparatus comprising:
the first determining module is used for determining the data type of the operation data of the preset network model;
a second determining module, configured to determine a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
and the processing module is used for processing the operation data through the target processing device.
Optionally, the first determining module is configured to determine the data type of the operation data through a preset data processing algorithm.
Optionally, the second determining module is configured to determine a candidate processing device from a plurality of processing devices according to the data type; the target processing device is determined from the candidate processing devices.
Optionally, the data type includes hot spot data or non-hot spot data; the second determining module is configured to, when determining that the data type is the hotspot data, take the first processing device of the plurality of processing devices as the candidate processing device; or in the case that the data type is determined to be the non-hot spot data, the second processing device of the plurality of processing devices is taken as the candidate processing device.
Optionally, the second determining module is configured to determine a storage capacity of each candidate processing device according to a preset processing sequence; and taking the candidate processing device with the storage capacity larger than or equal to a preset capacity threshold as the target processing device.
In a third aspect, the present disclosure provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of data processing of the first aspect described above.
In a fourth aspect, the present disclosure provides an electronic device comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the method for data processing according to the first aspect.
In a fifth aspect, the present disclosure provides a system for data processing, the system comprising a controller and a plurality of processing devices; the controller is respectively connected with the plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
the controller is configured to perform the method for data processing described in the first aspect.
Optionally, the second processing device includes a video memory module and a memory module; the controller is respectively connected with the video memory module and the memory module; the video memory module is connected with the memory module; the operation data comprises processing data and data to be processed; the data volume of the processing data is the same as the storage capacity of the video memory module; the data to be processed is data except the processing data in the operation data;
the video memory module is used for receiving the processing data sent by the controller; and processing the processing data;
the memory module is used for receiving the data to be processed sent by the controller; and storing the data to be processed.
Through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Additional features and advantages of the present disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings are included to provide a further understanding of the disclosure, and are incorporated in and constitute a part of this specification, illustrate the disclosure and together with the description serve to explain, but do not limit the disclosure.
FIG. 1 is a flow chart illustrating a method of data processing according to an exemplary embodiment.
Fig. 2 is a flow chart of a method of data processing according to the exemplary embodiment of fig. 1.
FIG. 3 is a flowchart illustrating another method of data processing according to an exemplary embodiment.
Fig. 4 is a block diagram illustrating an apparatus for data processing according to an exemplary embodiment.
FIG. 5 is a block diagram of a system for data processing, according to an exemplary embodiment.
FIG. 6 is a block diagram of a system for data processing shown in accordance with the exemplary embodiment of FIG. 5.
Fig. 7 is a block diagram of an electronic device, according to an example embodiment.
Fig. 8 is a block diagram of another electronic device, shown in accordance with an exemplary embodiment.
FIG. 9 is a block diagram of another system for data processing, according to an exemplary embodiment.
Detailed Description
Specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating and illustrating the disclosure, are not intended to limit the disclosure.
First, an application scenario of the present disclosure will be described, which is applied in a scenario in which a large model is run by a processor. Large models refer to deep learning models with tens or even hundreds of millions of parameters, which present significant challenges in data operation and processing because their data magnitude can reach the GB level or TB level.
In the related art, the data processing of the large model can be performed through the GPU, but the GPU has higher cost; in order to reduce the cost, the GPU and the CPU can also jointly process data of a large model. However, in the process of data processing, the GPU and the CPU need to run different types of computer programs, resulting in poor flexibility in the data processing process, improving development cost, and reducing work efficiency.
In order to solve the above problems, the present disclosure provides a method, an apparatus, a storage medium, an electronic device, and a system for data processing; determining the data type of the operation data of a preset network model; determining a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device; processing the operational data by the target processing device; through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
FIG. 1 is a flow chart illustrating a method of data processing according to an exemplary embodiment. As shown in fig. 1, the method may be applied to a controller, and the method may include the following steps.
S101, determining the data type of operation data of a preset network model.
Illustratively, the preset network model may be a large language model, a computer vision model, an audio model, a multi-modal large model, or the like, which is not limited herein; the data type may include hotspot data or non-hotspot data.
In some embodiments, the step S101 may include: and determining the data type of the operation data through a preset data processing algorithm. The preset data processing algorithm may be, for example, an algorithm that accelerates the speed of reasoning of a preset network model, such as the powerlner algorithm. For example, the operation data may be divided into hot spot data and non-hot spot data by a preset data processing algorithm; the hot spot data refers to data with higher frequency of use in the preset network model, and the non-hot spot data refers to data with lower frequency of use in the preset network model.
It should be noted that, since the preset data processing algorithm may be combined with a preset network model, the data type may also be a hot-spot neuron and a non-hot-spot neuron; the hot-spot neuron refers to data with higher use frequency in the preset network model and operation corresponding to the data with higher use frequency; the non-hotspot neurons refer to data with lower use frequency in the preset network model and operations corresponding to the data with lower use frequency.
S102, determining a target processing device corresponding to the data type from a plurality of processing devices.
Wherein the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device.
By way of example, the processing device may be a processor, such as a GPU; the first processing device may be a high performance GPU and the second processing device may be a low performance GPU, the first processing device having a higher performance than the second processing device. The performance parameter may include, but is not limited to, an operating speed, a data operand, or a power consumption. For example, it may be that the operating speed of the first processing device is greater than the operating speed of the second processing device; the data operation amount of the first processing device is larger than that of the second processing device; the power consumption of the first processing device is smaller than the power consumption of the second processing device, etc.
S103, processing the operation data through the target processing device.
Through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Fig. 2 is a flow chart of a method of data processing according to the exemplary embodiment of fig. 1. As shown in fig. 2, the above step S102 may include the following steps.
S1021, according to the data type, determining a candidate processing device from a plurality of processing devices.
In some embodiments, the data type includes hotspot data or non-hotspot data; the S1021 may include: and in the case that the data type is determined to be the hot spot data, taking the first processing device as the candidate processing device in the plurality of processing devices. Or in the case that the data type is determined to be the non-hot spot data, the second processing device of the plurality of processing devices is taken as the candidate processing device.
For example, each processing device comprises an identification by which the processing device can be determined to be either the first processing device or the second processing device. In this way, it is possible to determine that different processing apparatuses perform data processing based on the type of the operation data. To improve the efficiency of data processing.
S1022, determining the target processing device from the candidate processing devices.
In some embodiments, the step S1022 may include: determining the storage capacity of each candidate processing device according to a preset processing sequence; and taking the candidate processing device with the storage capacity larger than or equal to a preset capacity threshold as the target processing device.
For example, the preset processing sequence may be set by the user according to actual operation conditions; the preset capacity threshold may be set by the user according to actual operation conditions, and is not limited herein. For example, a first storage capacity of the first candidate processing device may be determined, and in a case where the first storage capacity is greater than or equal to a preset capacity threshold, the first candidate processing device is taken as a candidate processing device; determining a second storage capacity of a second candidate processing device if the storage capacity of the first candidate processing device is less than a preset capacity threshold, and taking the second candidate processing device as the candidate processing device if the second storage capacity is greater than or equal to the preset capacity threshold; and if the second storage capacity is smaller than the preset capacity threshold, repeating the step of determining the storage capacity of the processing device until the candidate processing device is determined.
The first candidate processing device may be a processing device in any one of the preset processing sequences, and the second candidate processing device may be a next processing device in the preset processing sequence, which is immediately adjacent to the first candidate processing device. Therefore, the operation data can be sequentially stored into the processing devices according to the preset sequence for processing, so that each processing device can perform data processing in a full load mode, omission is avoided, and the working efficiency is improved.
FIG. 3 is a flowchart illustrating a method of data processing according to an exemplary embodiment. As shown in fig. 3, the method may include the following steps.
S301, determining the data type of the operation data through a preset data processing algorithm.
The data type may include hot spot data or non-hot spot data, among others.
S302, determining candidate processing devices from a plurality of processing devices according to the data type.
S303, determining the storage capacity of each candidate processing device according to a preset processing sequence.
S304, the candidate processing device with the storage capacity being larger than or equal to a preset capacity threshold is used as the target processing device.
S305, processing the operation data through the target processing device.
Through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Fig. 4 is a block diagram illustrating an apparatus for data processing according to an exemplary embodiment. As shown in fig. 4, the apparatus 400 may include a first determination module 410, a second determination module 420, and a processing module 430;
the first determining module 410 is configured to determine a data type of operation data of a preset network model;
the second determining module 420 is configured to determine a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
the processing module 430 is configured to process the operation data by the target processing device.
Through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Optionally, the first determining module 410 is configured to determine the data type of the operation data by using a preset data processing algorithm.
Optionally, the second determining module 420 is configured to determine a candidate processing device from the plurality of processing devices according to the data type; the target processing device is determined from the candidate processing devices.
Optionally, the data type includes hotspot data or non-hotspot data; the second determining module 420 is configured to, in a case where the data type is determined to be the hotspot data, take the first processing device of the plurality of processing devices as the candidate processing device; or in the case that the data type is determined to be the non-hot spot data, the second processing device of the plurality of processing devices is taken as the candidate processing device.
Optionally, the second determining module 420 is configured to determine a storage capacity of each of the candidate processing devices according to a preset processing sequence; and taking the candidate processing device with the storage capacity larger than or equal to a preset capacity threshold as the target processing device.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
FIG. 5 is a block diagram of a system for data processing, according to an exemplary embodiment. As shown in fig. 5, the system 500 includes a controller 510 and a plurality of processing devices; the controller 510 is connected to the plurality of processing devices, respectively; the plurality of processing means comprises a first processing means 520 and a second processing means 530; the first processing device 520 and the second processing device 530 are the same device; the performance parameter of the first processing device 520 is greater than the performance parameter of the second processing device 530;
the controller 510 is configured to perform the method of data processing described above in relation to the first aspect.
It should be noted that the connection sequence between the controller 510 and the first processing device 520 and the second processing device 530 may be various, the number of the first processing device 520 and the second processing device 530 may be various, the connection sequence shown in fig. 5 is only one of various connection sequences, and other sequences and numbers are not shown in fig. 5.
Through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
FIG. 6 is a block diagram of a system for data processing shown in accordance with the exemplary embodiment of FIG. 5. As shown in fig. 6, the second processing device 530 includes a memory module 531 and a memory module 532; the controller 510 is connected to the video memory module 531 and the memory module 532, respectively; the video memory module 531 is connected to the memory module 532; the operation data comprises processing data and data to be processed; the data size of the processing data is the same as the storage capacity of the video memory module 531; the data to be processed is the data except the processing data in the operation data;
the video memory module 531 is configured to receive the processing data sent by the controller 510; and processing the processed data;
the memory module 532 is configured to receive the data to be processed sent by the controller 510; and stores the data to be processed.
The memory module may be, for example, a GPU for processing data; the memory module may be a memory of a server for storing data, thereby increasing a storage capacity of the processing device.
In some embodiments, the controller 510 may be configured to determine the processing data and the data to be processed according to the storage capacity of the memory module 531; and sends the processed data to the memory module 531 and the data to be processed to the memory module 532.
In other embodiments, the controller 510 may be configured to use the operation data having the same data amount as the storage capacity as the processing data and use data other than the processing data in the operation data as the data to be processed according to a predetermined data sequence. The preset data sequence may be the sequence in which the controller obtains the operation data, or may be set by a user, which is not limited herein. Therefore, the video memory module can be operated in a full load state, so that data processing is realized, and the processing efficiency of the video memory module is improved.
In some embodiments, the controller 510 may be configured to obtain, by the video memory module 531, the data to be processed stored in the memory module 532 when it is determined that the first storage capacity of the video memory module 531 is greater than or equal to the first preset capacity threshold; the memory module 531 is used for processing the data to be processed. Therefore, after the video memory module processes part of data, the data to be processed of the memory module can be stored into the video memory module in time, the data processing is continued, the interruption of the processing process is avoided, and the processing efficiency is improved.
In other embodiments, the controller 510 may be configured to determine the remaining storage capacity of the video memory module 531 and obtain, by the video memory module 531, the first data to be processed stored in the memory module 532, if it is determined that the first storage capacity of the video memory module 531 is greater than or equal to the first preset capacity threshold; the data amount of the first data to be processed is the same as the remaining storage capacity.
The first preset capacity threshold may be, for example, a maximum storage capacity of the video memory module, and may be set by a user, which is not limited herein. The remaining storage capacity may be determined from the first preset capacity threshold and the first storage capacity. For example, a difference between the first preset capacity threshold and the first storage capacity may be used as the remaining storage capacity. Thus, the data to be processed, which is the same as the data quantity of the residual storage capacity of the video memory module, can be obtained, and the data quantity is prevented from exceeding the storage capacity of the video memory module.
In some embodiments, the controller 510 may control the memory module 531 to obtain the data to be processed stored in the memory module 532 through a preset processing algorithm. For example, the preset control algorithm may be a Managed Memory algorithm, which is disclosed in related art documents and will not be described herein. Therefore, the mobilization between the processed data and the data to be processed can be realized, and the working efficiency is improved.
In summary, the present disclosure provides a method, an apparatus, a storage medium, an electronic device, and a system for data processing; determining the data type of the operation data of a preset network model; determining a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device; processing the operational data by the target processing device; through the technical scheme, the data processing can be carried out through the processing devices with different performance parameters, so that the running cost of the data processing is effectively reduced, and the flexibility of the data processing is improved; in addition, different running programs are not required to be developed for different processing devices, so that development cost is greatly reduced, and working efficiency is improved.
Fig. 7 is a block diagram of an electronic device 700, according to an example embodiment. As shown in fig. 7, the electronic device 700 may include: a processor 701, a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input/output interface 704, and a communication component 705.
Wherein the processor 701 is configured to control the overall operation of the electronic device 700 to perform all or part of the steps of the method for data processing described above. The memory 702 is used to store various types of data to support operation on the electronic device 700, which may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data, such as contact data, messages sent and received, pictures, audio, video, and so forth. The Memory 702 may be implemented by any type or combination of volatile or non-volatile Memory devices, such as static random access Memory (Static Random Access Memory, SRAM for short), electrically erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), erasable programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM for short), programmable Read-Only Memory (Programmable Read-Only Memory, PROM for short), read-Only Memory (ROM for short), magnetic Memory, flash Memory, magnetic disk, or optical disk. The multimedia component 703 can include a screen and an audio component. Wherein the screen may be, for example, a touch screen, the audio component being for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 702 or transmitted through the communication component 705. The audio assembly further comprises at least one speaker for outputting audio signals. The input/output interface 704 provides an interface between the processor 701 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, bluetooth, near field communication (Near Field Communication, NFC for short), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or one or a combination of more of them, is not limited herein. The corresponding communication component 705 may thus comprise: wi-Fi module, bluetooth module, NFC module, etc.
In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (Application Specific Integrated Circuit, abbreviated ASIC), digital signal processor (Digital Signal Processor, abbreviated DSP), digital signal processing device (Digital Signal Processing Device, abbreviated DSPD), programmable logic device (Programmable Logic Device, abbreviated PLD), field programmable gate array (Field Programmable Gate Array, abbreviated FPGA), controller, microcontroller, microprocessor, or other electronic component for performing the methods of data processing described above.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the method of data processing described above is also provided. For example, the computer readable storage medium may be the memory 702 including program instructions described above, which are executable by the processor 701 of the electronic device 700 to perform the method of data processing described above.
Fig. 8 is a block diagram of an electronic device 800, according to an example embodiment. For example, the electronic device 800 may be provided as a server. Referring to fig. 8, the electronic device 800 includes a processor 822, which may be one or more in number, and a memory 832 for storing computer programs executable by the processor 822. The computer program stored in memory 832 may include one or more modules each corresponding to a set of instructions. Further, the processor 822 may be configured to execute the computer program to perform the methods of data processing described above.
In addition, the electronic device 800 may further include a power supply component 826 and a communication component 850, the power supply component 826 may be configured to perform power management of the electronic device 800, and the communication component 850 may be configured to enable communication of the electronic device 800, such as wired or wireless communication. In addition, the electronic device 800 may also include an input/output interface 858. The electronic device 800 may operate an operating system based on storage 832.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the method of data processing described above is also provided. For example, the non-transitory computer readable storage medium may be the memory 832 including program instructions described above that are executable by the processor 822 of the electronic device 800 to perform the method of data processing described above.
In another exemplary embodiment, a computer program product is also provided, comprising a computer program executable by a programmable apparatus, the computer program having code portions for performing the method of data processing described above when executed by the programmable apparatus.
FIG. 9 is a block diagram of another system for data processing, according to an exemplary embodiment. As shown in fig. 9, the system 900 may include an electronic device 700.
The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solutions of the present disclosure within the scope of the technical concept of the present disclosure, and all the simple modifications belong to the protection scope of the present disclosure.
In addition, the specific features described in the foregoing embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, the present disclosure does not further describe various possible combinations.
Moreover, any combination between the various embodiments of the present disclosure is possible as long as it does not depart from the spirit of the present disclosure, which should also be construed as the disclosure of the present disclosure.
Claims (10)
1. A method of data processing, the method comprising:
determining the data type of operation data of a preset network model;
determining a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
and processing the operation data through the target processing device.
2. The method of claim 1, wherein determining the data type of the operational data of the predetermined network model comprises:
and determining the data type of the operation data through a preset data processing algorithm.
3. The method of claim 1, wherein determining, from the plurality of processing devices, a target processing device corresponding to the data type based on the data type comprises:
determining a candidate processing device from a plurality of processing devices according to the data type;
the target processing device is determined from the candidate processing devices.
4. A method according to claim 3, wherein the data type comprises hotspot data or non-hotspot data; the determining candidate processing means from the plurality of processing means according to the data type comprises:
in the case that the data type is determined to be the hot spot data, taking the first processing device of the plurality of processing devices as the candidate processing device; or,
and in the case that the data type is determined to be the non-hot spot data, taking the second processing device in the plurality of processing devices as the candidate processing device.
5. The method of claim 3, wherein said determining the target processing device from the candidate processing devices comprises:
determining the storage capacity of each candidate processing device according to a preset processing sequence;
and taking the candidate processing device with the storage capacity larger than or equal to a preset capacity threshold as the target processing device.
6. An apparatus for data processing, the apparatus comprising:
the first determining module is used for determining the data type of the operation data of the preset network model;
a second determining module, configured to determine a target processing device corresponding to the data type from a plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
and the processing module is used for processing the operation data through the target processing device.
7. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the program when executed by a processor realizes the steps of the method according to any of claims 1-5.
8. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the method of any one of claims 1-5.
9. A system for data processing, the system comprising a controller and a plurality of processing means; the controller is respectively connected with the plurality of processing devices; the plurality of processing devices includes a first processing device and a second processing device; the first processing device and the second processing device are the same device; the performance parameter of the first processing device is greater than the performance parameter of the second processing device;
the controller for performing the method of data processing of any of claims 1-5.
10. The system of claim 9, wherein the second processing device comprises a video memory module and a memory module; the controller is respectively connected with the video memory module and the memory module; the video memory module is connected with the memory module; the operation data comprises processing data and data to be processed; the data volume of the processing data is the same as the storage capacity of the video memory module; the data to be processed is data except the processing data in the operation data;
the video memory module is used for receiving the processing data sent by the controller; and processing the processing data;
the memory module is used for receiving the data to be processed sent by the controller; and storing the data to be processed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410186426.4A CN117742974A (en) | 2024-02-19 | 2024-02-19 | Data processing method, device, storage medium, electronic equipment and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410186426.4A CN117742974A (en) | 2024-02-19 | 2024-02-19 | Data processing method, device, storage medium, electronic equipment and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117742974A true CN117742974A (en) | 2024-03-22 |
Family
ID=90281759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410186426.4A Pending CN117742974A (en) | 2024-02-19 | 2024-02-19 | Data processing method, device, storage medium, electronic equipment and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117742974A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111382850A (en) * | 2018-12-28 | 2020-07-07 | 上海寒武纪信息科技有限公司 | Operation method, device and related product |
| CN114755993A (en) * | 2022-06-13 | 2022-07-15 | 深圳市玄羽科技有限公司 | Cooperative control method and system applied to industrial Internet and related equipment |
| CN115437778A (en) * | 2021-06-03 | 2022-12-06 | Oppo广东移动通信有限公司 | Kernel scheduling method and device, electronic equipment and computer readable storage medium |
| CN116229233A (en) * | 2023-02-01 | 2023-06-06 | 北京地平线机器人技术研发有限公司 | Image data processing method, device, electronic equipment and storage medium |
| CN117271121A (en) * | 2023-09-14 | 2023-12-22 | 中国平安财产保险股份有限公司 | Task processing progress control method, device, equipment and storage medium thereof |
-
2024
- 2024-02-19 CN CN202410186426.4A patent/CN117742974A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111382850A (en) * | 2018-12-28 | 2020-07-07 | 上海寒武纪信息科技有限公司 | Operation method, device and related product |
| CN115437778A (en) * | 2021-06-03 | 2022-12-06 | Oppo广东移动通信有限公司 | Kernel scheduling method and device, electronic equipment and computer readable storage medium |
| CN114755993A (en) * | 2022-06-13 | 2022-07-15 | 深圳市玄羽科技有限公司 | Cooperative control method and system applied to industrial Internet and related equipment |
| CN116229233A (en) * | 2023-02-01 | 2023-06-06 | 北京地平线机器人技术研发有限公司 | Image data processing method, device, electronic equipment and storage medium |
| CN117271121A (en) * | 2023-09-14 | 2023-12-22 | 中国平安财产保险股份有限公司 | Task processing progress control method, device, equipment and storage medium thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107957976B (en) | Calculation method and related product | |
| KR101529802B1 (en) | Method and apparatus for providing efficient context classification | |
| EP3855362A1 (en) | Convolution processing method, apparatus, and storage medium of convolutional neural network | |
| US11086668B2 (en) | Method, electronic device and computer program product for processing task | |
| CN114610650A (en) | Memory compression method and device, storage medium and electronic equipment | |
| CN110688160B (en) | Instruction pipeline processing method, system, equipment and computer storage medium | |
| CN117149665B (en) | Continuous integration method, control device, continuous integration system, and storage medium | |
| CN117742974A (en) | Data processing method, device, storage medium, electronic equipment and system | |
| CN110009100B (en) | Calculation method of user-defined operator and related product | |
| CN113747423B (en) | Cloud mobile phone state synchronization method, device, equipment, storage medium and program product | |
| CN116451174A (en) | Task execution device, method, electronic device, and storage medium | |
| CN107301017B (en) | Data storage method and device | |
| CN114879832A (en) | Power consumption control method, device, chip, apparatus, and medium for arithmetic device | |
| CN106507268B (en) | Method and device for dynamic verification in mobile phone carriers | |
| CN114780501A (en) | Data processing method, electronic device and computer program product | |
| CN109408242B (en) | Server resource online and offline method and device | |
| CN107610035B (en) | Method and system for processing image | |
| CN112230964A (en) | Application program development method, application program running method, device, equipment and medium | |
| CN105468603A (en) | Data selection method and apparatus | |
| CN105988804B (en) | Software transfer method and terminal | |
| CN107562442B (en) | Method and device for reading data | |
| CN113590263B (en) | Method and device for obtaining virtual machine scheduling scheme, terminal equipment and storage medium | |
| CN110825606B (en) | Data operation method, device and central station system | |
| CN114490515A (en) | Method, device, equipment and storage medium for determining data type in file | |
| US11431349B2 (en) | Method, electronic device and computer program product for processing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |