CN110569329B

CN110569329B - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN110569329B
Application number: CN201911031910.5A
Authority: CN
Inventors: 王若晖; 胡颖凡
Original assignee: Shenzhen Sensetime Technology Co Ltd
Current assignee: Shenzhen Sensetime Technology Co Ltd
Priority date: 2019-10-28
Filing date: 2019-10-28
Publication date: 2022-08-02
Anticipated expiration: 2039-10-28
Also published as: KR20210096230A; TWI755890B; CN110569329A; TW202117707A; JP2022510660A; WO2021082463A1

Abstract

The present disclosure relates to a data processing method and apparatus, an electronic device, and a storage medium, the method including: determining a sample list comprising a plurality of samples according to the fields of the samples and the parameter values corresponding to the fields; determining description information corresponding to the sample list according to the data type of the samples in the sample list; and determining a sample set according to the sample list and the description information. According to the data processing method disclosed by the embodiment of the disclosure, the sample set for storing each sample and the corresponding description information can be determined, a unified mode can be used for storing a plurality of types of samples, the description information describing the data type is included in the sample set, the storage and reading of the sample data are facilitated, the storage and reading of the data of the plurality of types can be supported, and the understanding is easy.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.

Background

In machine learning, particularly in the case where a plurality of data sets need to be processed, the difficulty of reading data increases. In the related art, there is no uniform way for storing data sets, and the data sets are stored to adapt to the structure of the data sets, which cannot adapt to each other, and the data are not easy to read. For example, although the DataFrame type data storage method in the pandas library of python language is powerful, the supported data types are limited, and the reading process is usually complex, so that it is difficult for professional users to learn. In addition, the reading operation mainly focuses on the data itself, often ignores the description information describing the data, and thus is difficult to be applied in some scenarios for beginners (e.g., entrance level artificial intelligence education).

Disclosure of Invention

The disclosure provides a data processing method and device, electronic equipment and a storage medium.

According to an aspect of the present disclosure, there is provided a data processing method including:

determining a sample list comprising a plurality of samples according to the fields of the samples and the parameter values corresponding to the fields;

determining description information corresponding to the sample list according to the data type of the samples in the sample list;

and determining a sample set according to the sample list and the description information.

According to the data processing method disclosed by the embodiment of the disclosure, the sample set for storing each sample and the corresponding description information can be determined, a unified mode can be used for storing a plurality of types of samples, the description information describing the data type is included in the sample set, the storage and reading of the sample data are facilitated, the storage and reading of the data of the plurality of types can be supported, and the understanding is easy.

In one possible implementation manner, determining a sample list including a plurality of samples according to a field of a sample and a parameter value corresponding to the field includes:

determining the field as column coordinates of the sample list, and determining a sample index of a sample as row coordinates of the sample list;

objectifying the parameter values of the samples to obtain objects corresponding to the samples;

and adding the object to a storage area corresponding to the column coordinate and the row coordinate to obtain the sample list.

By the method, parameter values and/or labels of the samples can be objectified, the samples of various data types can be stored through the sample list, the samples can be stored and read quickly, resource occupation during storage and reading is reduced, various types of data storage can be supported, and the method is suitable for storage of more complex types of data sets.

In one possible implementation, the description information includes description information of a reading manner of the sample,

determining description information corresponding to the sample list according to the data type of the sample in the sample list, wherein the description information comprises:

determining a reading mode for reading the samples according to the data types of the samples in the sample set;

and obtaining the description information according to the data type of the sample and the corresponding reading mode.

By the method, the reading modes of various samples can be determined, the storage and the access of various types of data can be supported, and the storage and the access of more complex types of data sets can be adapted.

In a possible implementation manner, the description information further includes description information of a storage location of the sample, where the storage location includes a storage path of the sample set and an index position of the sample in the sample set.

In this way, the exact location of the sample can be determined, facilitating the reading of the sample.

In one possible implementation, the method further includes:

determining index range information for dividing categories of samples in a sample set;

and generating the description information corresponding to the sample set according to the description information of the samples in the sample set and the index range information of the sample category.

In this way, the index range of the sample can be determined, facilitating the search for the sample.

In one possible implementation, the method further includes:

and reading parameter values of the samples in the sample list according to the description information and the sample list.

In a possible implementation manner, reading parameter values of samples in the sample list according to the description information and the sample list includes:

obtaining objects in the sample list;

according to the sample index, obtaining a first object of a first sample corresponding to the sample index from the obtained objects;

determining the data type and the corresponding reading mode of the first sample according to the description information;

and reading the parameter value of the first sample from the first object according to the reading mode.

By the method, the resource consumption in the process of reading the sample can be reduced, the sample data can be stored in a unified mode, and the sample data type can be stored by utilizing the sample description information, so that the data reading mode can be conveniently obtained according to the description information, the samples of various data types can be conveniently read, and the rapid reading of the sample is facilitated.

acquiring a sample index;

and acquiring the parameter value of the sample corresponding to the sample index from the sample list according to the sample index.

obtaining an object corresponding to a target field in the sample list;

determining the data type and the reading mode of a second sample corresponding to the obtained object according to the description information;

and reading the parameter value of the second sample from the obtained object according to the reading mode.

In this way, the parameter value of one or more samples in the field is read by accessing the field, so that the flexibility of sample reading is improved, and the rapid reading of the samples is facilitated.

acquiring a field name of a target field;

and acquiring the parameter value of the target field corresponding to the field name from a sample list according to the field name.

In one possible implementation, the method further includes:

and at least one of inquiring, modifying and deleting the read parameter values of the samples.

According to an aspect of the present disclosure, there is provided a data processing apparatus including:

the device comprises a first determining module, a second determining module and a control module, wherein the first determining module is used for determining a sample list comprising a plurality of samples according to fields of the samples and parameter values corresponding to the fields;

the second determining module is used for determining the description information corresponding to the sample list according to the data type of the samples in the sample list;

and the third determining module is used for determining a sample set according to the sample list and the description information.

In one possible implementation, the first determining module is further configured to:

the second determination module is further configured to:

In one possible implementation, the method further includes:

the fourth determining module is used for determining index range information for dividing the categories of the samples in the sample set;

and the first generation module is used for generating the description information corresponding to the sample set according to the description information of the samples in the sample set and the index range information of the sample category.

In one possible implementation, the method further includes:

and the reading module is used for reading the parameter values of the samples in the sample list according to the description information and the sample list.

In one possible implementation, the reading module is further configured to:

obtaining objects in the sample list;

and reading the parameters of the first sample from the first object according to the reading mode.

In one possible implementation, the reading module is further configured to:

acquiring a sample index;

In one possible implementation, the reading module is further configured to:

obtaining an object corresponding to a target field in the sample list;

In one possible implementation, the reading module is further configured to:

acquiring a field name of a target field;

In one possible implementation, the apparatus further includes:

and the operation module is used for performing at least one operation of inquiry, modification and deletion on the read parameter values of the samples.

According to an aspect of the present disclosure, there is provided an electronic device including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: the above-described data processing method is performed.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described data processing method.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

FIG. 1 shows a flow diagram of data processing according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a sample list according to an embodiment of the present disclosure;

FIG. 3 illustrates an application diagram of a storage method according to an embodiment of the present disclosure;

FIG. 4 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure;

FIG. 5 shows a block diagram of an electronic device according to an embodiment of the disclosure;

fig. 6 illustrates a block diagram of an electronic device in accordance with an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

Fig. 1 shows a flow chart of a data processing method according to an embodiment of the present disclosure, as shown in fig. 1, the method comprising:

in step S11, a sample list including a plurality of samples is determined according to the fields of the samples and the parameter values corresponding to the fields;

in step S12, determining description information corresponding to the sample list according to the data type of the sample in the sample list;

in step S13, a sample set is determined according to the sample list and the description information.

In one possible implementation manner, the main body of the data processing method may be a terminal device or a server or other processing device, where the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the method may be implemented by a processor calling computer readable instructions stored in a memory or executed by a server, for example, data in the server may be stored by the storage method.

In one possible implementation, the sample may include a plurality of data types, such as image, video, and/or general numerical or text data, and the like, and the data types of the sample are not limited by the present disclosure. The sample may include one or more fields, each of which may correspond to a certain characteristic or class of characteristics of the sample, e.g., a sample is profile data of a certain person, the fields of the sample may include height, weight, school calendar, nationality, etc. of the person, and the sample may have data corresponding to the fields. The present disclosure does not limit the fields.

In one possible implementation manner, in step S11, the plurality of samples may be saved through a sample list (datatable), and operations such as adding, deleting, and the like of data are facilitated by storing in a list manner. Step S11 may include: determining the field as column coordinates of the sample list, and determining a sample index of a sample as row coordinates of the sample list; objectifying the parameter values of the samples to obtain objects corresponding to the samples; and adding the object to a storage area corresponding to the column coordinate and the row coordinate to obtain the sample list.

Fig. 2 shows a schematic diagram of a sample list according to an embodiment of the present disclosure, which may have a row coordinate of the index of the sample and a column coordinate of the field, as shown in fig. 2. The fields of each sample may be aggregated as column coordinates of a sample list, and the sample may be a tuple, such as a two-dimensional tuple, a data set composed of data of a plurality of fields. For example, if a sample (e.g., sample 1) is profile data of a person, fields of the sample may include height (e.g., field 1), weight (e.g., field 2), academic calendar (e.g., field 3), and the like of the person, fields of the sample may include identity information and the like of the person, fields of samples may be aggregated, and column coordinates of a sample list may include fields of height, weight, academic calendar, nationality, and identity information and the like. Each field corresponds to a column of data in the sample list, and in some rows of the column of data, the data may be empty. For example, in the sample list, a "nationality" field is included, in the sample of the data of the material of a certain person, the field may have data, such as "china", "usa", etc., and in the sample of the face image, the field may have no data, that is, in the sample list, in the area corresponding to the row where the face image sample is located, the column where the "nationality" field is located, the data is empty.

In one possible implementation, the sample index of a sample may be determined as the row coordinate of the sample list, e.g., each sample may have an index, e.g., sample 1, sample 2, sample 3, etc., and the sample index may be determined as the row coordinate of the sample list.

In one possible implementation, the samples may be training samples used in training a neural network, and each sample may have a label, which may be one of the fields describing the sample. For example, the samples may include face image samples, and when performing authentication through a neural network, the neural network compares each face image sample with a reference image (e.g., a face image of a certain target person) and determines the similarity between the face image sample and the reference image. When the similarity is greater than or equal to the similarity threshold (for example, 0.5), it may be determined that the similarity between the face image sample and the reference image is high, and the authentication passes, otherwise, the authentication does not pass. When the neural network is trained, each face image sample needs to be labeled, and parameters are provided for a loss function of the neural network. For example, if a certain face image sample is a face image of a target person, it may be labeled as 1, indicating that the face in the face image sample is the face of the target person, and if a certain face image sample is not a face image of a target person, it may be labeled as 0, indicating that the face in the face image sample is not the face of the target person. The sample may have parameter values for fields and labels, for example, a field of a certain face image sample is identity information (e.g., name or number, etc.) of a target object in the image, and is labeled as 0 or 1.

In one possible implementation, the label and the parameter value of the sample may be objectified, for example, a storage area corresponding to the sample may be established, for example, a storage space may be established in a memory, the label and the parameter of the sample are stored in the storage area, and an object (object) corresponding to the sample is obtained for storage and reading. Wherein, the storage area storing the parameters is the object.

In one possible implementation, an object corresponding to a sample may be added to or associated with the sample list to initialize the sample list. For example, the storage area in which the parameter is stored may correspond to a storage area in the sample list (e.g., a certain cell in the sample list). In an example, the object corresponding to sample 1 may be added to row 1 of the sample list, i.e., the data of each field in the object corresponding to sample 1 is added to the storage area corresponding to each field in row 1, e.g., the height data is added to the storage area having row coordinates of row 1 and column coordinates of the "height" field, the weight data is added to the storage area having row coordinates of row 1 and column coordinates of the "weight" field. Each sample can be objectified and filled in the storage area of the sample list in the above manner, and a sample list in which a plurality of samples are stored can be obtained.

In an example, the sample list may be initialized according to the sample index, for example, after data of each field of one sample is added to a corresponding storage area, data of each field of a next sample may be added, that is, data of each sample is added in a row direction in the list, so as to initialize the sample list.

In an example, the sample list may be initialized according to fields, that is, after all data of one field is added to the corresponding storage area, data of the next field is added, that is, data is added according to the column direction in the list, so as to initialize the sample list.

Specifically, the field name of the character type and the value list or array of the sample under the field may be defined in a two-dimensional tuple manner, or the sample list may be initialized in a key parameter manner, or a mixture of the two manners.

In one possible implementation manner, in step S12, the sample may include multiple data types, each type of sample may have a specific storage and reading manner, and when multiple types of samples are stored, each sample may be described by the description information so as to access (e.g., store or read) the multiple types of samples. In an example, the sample may include a video, the description information may include a description of a storage or reading manner of the video, the sample may include an image, and the description information may include a description of a storage or reading manner of the image. The description information may be description information in a metadata meta format, for example, a dictionary type meta, which facilitates to increase data query speed, and the disclosure does not limit the type of the description information.

In a possible implementation manner, the description information includes description information of a reading manner of the sample, and step S12 may include: determining a reading mode for reading each sample according to the data type of the sample in the sample set; and obtaining the description information according to the data type of the sample and the corresponding reading mode.

In a possible implementation, the description information may be information describing a reading manner of the sample. The reading mode of each sample can be determined according to the data type of the sample, for example, if the data type of a certain sample is an image, the sample can be read by using an image reading function, and if the data type of a certain sample is a video, the sample can be read by using a video reading function, the reading mode of each sample can be determined respectively. The data types of the samples may also be summarized, for example, the sample set includes a plurality of samples, the data types of the plurality of samples are three, and are respectively image, video and common data, and it may be determined to read the sample of the image type through the image reading function, read the sample of the video type through the video reading function, and read the sample of the common data type through the data reading function. The present disclosure does not limit the data type and reading manner of the sample.

In one possible implementation, the description information may be obtained according to a data type of the sample and a corresponding reading manner, and in an example, the description information may be configured as word-typical data, for example, a key of a dictionary is the data type, a value of the dictionary is the reading manner corresponding to the data type, for example, the key of the dictionary is an image, and the value of the dictionary is an image reading function. The present disclosure does not limit the data type of the description information.

In one possible implementation, the description information may include description information of storage locations of samples, including storage paths of the sample sets and index locations of samples in the sample sets. For example, the storage path of the sample set is that the sample set is stored in a "sample set" folder in a C-disk of the computer.

In a possible implementation manner, the index position of the sample in the sample set is a specific position of the sample. In an example, the data for each sample may be stored in the sample set in the form of an object (object), i.e., occupying some storage area in the sample set. The sample set may store each sample in a list form, and then the data of the sample may occupy a certain row in the sample set, the index position may represent a row number, for example, the 1 st sample is located in the 1 st row of the sample set list, the 2 nd sample is located in the 2 nd row … of the sample set list, the field information may include the storage path of the sample set and the index position of each sample, and the specific position of each sample may be determined according to the field information, for example, the 2 nd sample is located in the 2 nd row of the sample set list in the "sample set" folder in the C-disk of the computer. For another example, some samples may have only some fields, or multiple samples belong to the same type or the same sample file, and the index position may be a range of rows and columns of samples in the sample set, or a range of rows and columns of multiple samples. The present disclosure does not limit the storage location.

In a possible implementation manner, the sample set may include samples of multiple Data types, such as images, videos, and/or common numerical or text Data, and the like, the storage Format of the samples may include CSV (Comma-Separated Values), HDF5(Hierarchical Data Format 5), and the like, and the index position of the sample in the sample set may be a position range of the same type of sample generated after summarizing the same type of sample. For example, a sample file stored in CSV format would store the index location of the CSV file in the description information.

In one possible implementation, the method further includes: determining index range information for dividing categories of samples in a sample set; and generating the description information corresponding to the sample set according to the description information of the samples in the sample set and the index range information of the sample category.

In a possible implementation manner, the description information further includes index range information for dividing types of the samples in the sample set. The method for determining the description information further comprises the following steps: determining index range information for dividing the category of each sample in the sample set; and generating the description information corresponding to the sample set according to the description information of the samples in the sample set and the index range information of the sample category. For example, classification may be based on labeling of samples. For example, the sample set may be divided into a training set, a validation set, and a test set, where 5000 samples are included in the training set, where 3000 samples are labeled 1 and 2000 samples are labeled 0. Alternatively, the labels may not be distinguished, and the samples in the sample set may be directly divided according to the number and the index of the samples in the sample set, for example, the samples in the sample set may be training samples used in training the neural network, the training samples may be divided into a plurality of categories, for example, 40% of the samples may be divided into a training set, 30% of the samples may be divided into a test set, and 30% of the samples may be divided into a verification set. Or may be divided according to the index of each sample and determine the index range of each category. For example, the sample set stores samples in a list form, 10000 samples are stored in the list, the 1 st to 4000 th samples in the list can be divided into a training set, the 4001 st to 7000 th samples in the list can be divided into a test set, and the 7001 st to 10000 th samples in the list can be divided into a verification set. The description information may include index range information (e.g., 1 st-4000 th, 4001 st-7000 th, 7001 st-10000 th, etc.) for dividing sample categories. The present disclosure does not limit the manner of partitioning and the scope of indexing.

The sample set may include other forms of data subsets, such as image subsets, video subsets, etc., besides the sample subsets of the three sample categories of the training set, the test set, and the verification set. The description information of the sample set may include description information of other forms of data subsets in the sample set, in addition to the description information of the sample subsets of the three sample categories of the training set, the test set, and the verification set and their corresponding index ranges.

In one possible implementation manner, in step S13, the sample set (dataset) may be obtained according to the sample list and the description information, in an example, the sample list and the description information may be written into the sample set to obtain a file of the sample set, or the description information and the sample list may be packaged to obtain the sample set, and the present disclosure does not limit the manner of obtaining the sample set.

In a possible implementation manner, the sample set includes a sample list storing a plurality of samples, and further includes description information recording a reading manner of each sample, so as to facilitate storage and reading of samples of multiple data types, and provide a basis for storing samples of multiple data types. The description information can record the data type of each sample, the reading mode and other information. The samples or data of the samples in the sample set may be read based on the description information. The present disclosure does not limit the content of the description information.

In one possible implementation, the method further includes: and reading parameter values of the samples in the sample list according to the description information and the sample list.

In a possible implementation manner, the samples in the sample list may be read in an appropriate manner according to the description of the reading manner of the samples in the description information, and the reading process may include: obtaining objects in the sample list; according to the sample index, obtaining a first object of a first sample corresponding to the sample index from the obtained objects; determining the data type and the corresponding reading mode of the first sample according to the description information; and reading the parameter value of the first sample from the first object according to the reading mode.

In one possible implementation, the sample list includes a plurality of samples, and the samples are stored in the sample list in the format of an object. When a sample is read, a storage space corresponding to each object in the sample list may be obtained, that is, the object in the sample list is obtained (for example, by accessing the 'samples' attribute, that is, accessing the sample object, all objects of the list including all samples may be obtained). Further, the sample index in the sample list may be accessed, and the first sample to be read is selected in the sample index, that is, the first object corresponding to the first sample is selected, for example, a row in the sample list where the first sample is located is selected by the sample index number.

In a possible implementation manner, the description information may include a reading manner of each sample, and a corresponding reading manner may be selected from the description information according to a data type of the first sample, for example, the data type of the first sample is an image, an image reading function in the description information may be selected to read the first sample, the data type of the first sample is a video, and a video reading function in the description information may be selected to read the first sample.

In a possible implementation manner, the parameter value of the first sample may be read in the storage space corresponding to the first object by a reading manner corresponding to the data type of the first sample. For example, the first sample is data of a person, the first sample may include height data, weight data, academic data, national data, and the like of the person, the data of the person may be read by a data reading function, the data of the first sample may be extracted from the storage space of the first object, and the parameter values of the first sample may be read by the data reading function. In an example, parameter values of one or more fields of the first sample data may also be read, for example, height data and weight data in the profile data may be selected and read by a data reading function. The whole data of the first sample can not be required to be read, and the flexibility of sample reading is improved.

In a possible implementation manner, reading parameter values of samples in the sample list according to the description information and the sample list includes: acquiring a sample index; and acquiring the parameter value of the sample corresponding to the sample index from the sample list according to the sample index.

In an example, an index may be utilized to access a sample list, read a sample corresponding to the index, and obtain a parameter value for the sample. For example, when sample data needs to be read, a reading instruction input by a user or a sample obtaining device may be received, an index of a sample that needs to be read in the reading instruction may be set to have an index value of 100, and when reading, a sample in a 100 th row in the list, that is, a sample in a 100 th row in the list may be obtained, and a parameter value of the sample is read according to a reading manner of the sample. For example, the parameter value of the sample may be read according to a reading method in the description information, or the data type of the sample may be determined according to a suffix of the sample, and then the corresponding reading method may be selected to read the sample, so as to obtain the parameter value of the sample.

In a possible implementation manner, a plurality of parameter values corresponding to a certain field or parameter values of the field of a certain sample may also be read by accessing the field in the sample list.

In a possible implementation manner, reading parameter values of samples in the sample list according to the description information and the sample list includes: obtaining an object corresponding to a target field in the sample list; determining the data type and the reading mode of a second sample corresponding to the obtained object according to the description information; and reading the parameter value of the second sample from the obtained object according to the reading mode.

In one possible implementation, the target field in the sample list may be accessed, the object corresponding to the target field may be obtained, for example, by accessing the attributes of 'fields', that is, accessing the field, the entire object containing all sample data fields may be obtained, that is, the storage space (such as the field list) where the parameters of the field of the multiple samples are located may be obtained, for example, "age", "height", "nationality", etc. may be included in the field list, the corresponding field may also be accessed by the keywords in the field, for example, the parameter values in the "nationality" field may be accessed by the keywords "nationality", or the parameter values in the field may be directly accessed by the field name or the keywords, without accessing the field list, for example, the parameter values in the "nationality" field may be directly accessed by the field name "nationality". In an example, each field corresponds to a column of data in the sample list, and in some areas of the column of data, the data may be null. For example, in the list of samples, a "nationality" field is included, in the sample of the data of the material of a certain person, there may be data, such as "china", "usa", etc., and in the sample of the face image, there may be no data in the field, that is, in the list, in the area corresponding to the row where the sample of the face image is located, the column where the "nationality" field is located, the data is null. When the object corresponding to the target field is obtained, the empty area can be ignored or not ignored, and the disclosure does not limit whether to ignore or not. When the object corresponding to the target field is obtained, all the objects in the column where the target field is located, that is, all the storage spaces storing the data of the column, may be obtained.

In a possible implementation manner, the reading manner of the second sample to be read may be determined according to the description information, and a storage space of the parameter value of the target field of the second sample, that is, an object of the parameter value of the target field of the second sample, may also be determined in an object corresponding to the target field according to an index of the second sample.

In a possible implementation manner, the parameter value of the target field of the second sample may be read in the object of the parameter value of the target field of the second sample by the reading manner. For example, the second sample is the data of a person, and the second sample may include height data, weight data, academic data, national data, and the like of the person. In the object for obtaining all the parameter values corresponding to the "height" field, the object with the second sample in the field can be determined, and the parameter values in the object can be read by the reading mode corresponding to the material data.

In a possible implementation manner, reading parameter values of samples in the sample list according to the description information and the sample list includes: acquiring a field name of a target field; and acquiring the parameter value of the target field corresponding to the field name from a sample list according to the field name.

In an example, the field name of the target field may be utilized to access the target field and thereby obtain the parameter value of the target field. For example, when a user or device needs to read sample data, the storage device or processing device may receive a read instruction that includes a field name of a target field of the sample that needs to be read, the target field may be set to "height," and when read, a parameter value of the "height" field may be accessed by the field name "height. For example, the parameter value of the field may be read by using a reading method corresponding to the data type of the field in the description information, or the data type of the parameter of the field may be determined according to a suffix of the parameter of the field, and then the parameter value of the field may be read by determining the corresponding reading method. Further, any parameter value under this field may also be read, for example, the parameter value stored in the storage location with index value of 100 under the "height" field may be obtained.

When sample data is read, including the reading of sample or field parameter values, the storage device or the data processing device receives a reading instruction, the reading instruction includes sample data information to be read, such as a sample index, a field name, and the like, and according to the reading instruction, the data type, the storage position, and the corresponding reading mode of the sample are acquired from the sample set description information, so that the acquired reading mode can be adopted to read the required sample data at the storage position. Therefore, the sample set containing the description information can conveniently store and read various types of sample data, and the user can conveniently understand and view the data stored in the sample set through the description information, so that the method is suitable for scenes facing beginners.

In one possible implementation, the method further includes: and at least one of inquiring, modifying and deleting the read parameter values of the samples. In an example, after reading a parameter value in an object, the parameter value may be at least one of queried, modified, deleted, and the like. For example, a parameter value in the object may be queried and copied. Or the parameter value may be modified and the modified parameter value saved in the object. Alternatively, the parameter values in the object may be deleted, leaving the data in the object empty. The present disclosure does not limit the way in which the parameter values are processed.

Fig. 3 is a schematic diagram illustrating an application of the storage method according to an embodiment of the present disclosure, and as shown in fig. 3, a plurality of samples may be stored in a sample list, and the data types of the plurality of samples may include image, audio, video, and/or general data (e.g., numeric data, category data, character data), and the like. Each sample may occupy a row of storage area in the sample list. Each sample may include one or more fields, for example, a sample may be profile data of a person, fields of the sample may include height, weight, school calendar, nationality, etc. of the person, and the sample may have data corresponding to the fields.

In one possible implementation, fields of each sample may be collected as column coordinates of a sample list, and a sample index of a sample is determined as row coordinates of the sample list, so that an empty sample list may be obtained.

In one possible implementation, the labels and parameter values of the samples may be objectified, that is, a storage area corresponding to each parameter value of the samples is established, the labels and parameters of the samples are stored in the storage area, an object corresponding to the samples is obtained, and the object is added to the sample list, so as to obtain the sample list storing the plurality of samples.

In one possible implementation, the sample may include a plurality of data types, each type of sample may have a specific storage and reading manner, and when a plurality of types of samples are stored, each sample may be described by the description information in the meta data meta format so as to store or read the plurality of types of samples.

In a possible implementation manner, the data types of each sample in the sample list can be summarized, and the reading manner of each data type is judged, the data types of the multiple samples in the sample list are three, namely, image, video and common data, so that the sample of the image type can be read through an image reading function, the sample of the video type can be read through the video reading function, and the sample of the common data type can be read through the data reading function. And configuring the description information into word typical data, for example, the key of the dictionary is a data type, and the value of the dictionary is a reading mode corresponding to the data type.

In one possible implementation, the sample list and the description information may be written into the sample set, and a file of the sample set is obtained. And may read the sample or a value of a parameter in the sample by accessing the sample index or field.

In one possible implementation manner, a storage space corresponding to each object in the sample list may be obtained, that is, the object in the sample list is obtained, and a sample index in the sample list may be accessed to select the first sample to be read, that is, the first object corresponding to the first sample is selected. Further, a corresponding reading mode may be selected in the description information, for example, the data type of the first sample is an image, and an image reading function in the description information may be selected to read the first sample. And the first sample may be read by the reading means, or only the parameter values of one or more fields in the first sample may be read.

In one possible implementation, the target field in the sample list may be accessed, and all objects corresponding to the target field may be obtained (the empty area in the target field may be ignored). The parameter value of the second sample in the target field can be read in all objects corresponding to the target field. In an example, a data type of the second sample may be determined, and a reading manner corresponding to the data type of the second sample may be determined in the description information. Further, among the objects of all the parameter values of the target field, the object of the second sample in the field may be determined, and the parameter value of the second sample in the target field may be read in the object in a reading manner corresponding to the data type of the second sample.

In a possible implementation manner, the storage method can be used for storing multiple types of samples, for example, in the field of computer education, multiple types of samples can be stored according to a uniform storage method by using the storage method, the program interface for storing and reading is convenient and concise, and the various sample storage and reading manners are uniform and easy to understand, so that students can learn and operate conveniently.

It is understood that the above-mentioned embodiments of the method of the present disclosure can be combined with each other to form a combined embodiment without departing from the principle logic, which is limited by the space, and the detailed description of the present disclosure is omitted.

In addition, the present disclosure also provides a data processing apparatus, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any image processing method provided by the present disclosure, and the corresponding technical solutions and descriptions and corresponding descriptions in the methods section are not repeated.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

Fig. 4 shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure, which, as shown in fig. 4, includes:

a first determining module 11, configured to determine a sample list including multiple samples according to a field of a sample and a parameter value corresponding to the field;

a second determining module 12, configured to determine, according to the data type of the sample in the sample list, description information corresponding to the sample list;

and a third determining module 13, configured to determine a sample set according to the sample list and the description information.

the second determination module is further configured to:

In one possible implementation, the method further includes:

In one possible implementation, the reading module is further configured to:

obtaining objects in the sample list;

In one possible implementation, the reading module is further configured to:

acquiring a sample index;

In one possible implementation, the reading module is further configured to:

obtaining an object corresponding to a target field in the sample list;

In one possible implementation, the reading module is further configured to:

acquiring a field name of a target field;

In one possible implementation, the apparatus further includes:

In some embodiments, functions of or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and for specific implementation, reference may be made to the description of the above method embodiments, and for brevity, details are not described here again

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-mentioned method. The computer readable storage medium may be a volatile computer readable storage medium or a non-volatile computer readable storage medium.

An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured as the above method.

The electronic device may be provided as a terminal, server, or other form of device.

Fig. 5 is a block diagram illustrating an electronic device 800 in accordance with an example embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.

Referring to fig. 5, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described method.

The disclosed embodiments also provide a computer program product comprising computer readable code which, when run on a device, executes instructions for implementing a method as provided by any of the above embodiments.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

Fig. 6 is a block diagram illustrating an electronic device 1900 according to an example embodiment. For example, the electronic device 1900 may be provided as a server. Referring to fig. 6, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, comprising:

determining description information corresponding to the sample list according to the data type of the samples in the sample list, wherein the description information comprises description information of a reading mode of the samples and/or description information of storage positions of the samples;

2. The method of claim 1, wherein determining a sample list comprising a plurality of samples according to a field of the sample and a parameter value corresponding to the field comprises:

3. The method according to claim 1 or 2,

4. The method of claim 1 or 2, wherein the storage locations comprise storage paths of the sample set and index locations of samples in the sample set.

5. The method according to claim 1 or 2, characterized in that the method further comprises:

6. The method according to claim 1 or 2, characterized in that the method further comprises:

7. The method of claim 6, wherein reading parameter values of samples in the sample list according to the description information and the sample list comprises:

obtaining objects in the sample list;

8. The method of claim 6, wherein reading parameter values of samples in the sample list according to the description information and the sample list comprises:

acquiring a sample index;

9. The method of claim 6, wherein reading parameter values of samples in the sample list according to the description information and the sample list comprises:

obtaining an object corresponding to a target field in the sample list;

10. The method of claim 6, wherein reading parameter values of samples in the sample list according to the description information and the sample list comprises:

acquiring a field name of a target field;

11. The method of claim 6, further comprising:

12. A data processing apparatus, comprising:

the second determining module is used for determining the description information corresponding to the sample list according to the data type of the samples in the sample list, wherein the description information comprises the description information of the reading mode of the samples and/or the description information of the storage positions of the samples;

13. The apparatus of claim 12, wherein the first determining module is further configured to:

14. The apparatus of claim 12 or 13,

the second determination module is further configured to:

15. The apparatus of claim 12 or 13, wherein the storage location comprises a storage path of the sample set and an index position of a sample in the sample set.

16. The apparatus of claim 12 or 13, further comprising:

17. The apparatus of claim 12 or 13, further comprising:

18. The apparatus of claim 17, wherein the reading module is further configured to:

obtaining objects in the sample list;

19. The apparatus of claim 17, wherein the reading module is further configured to:

acquiring a sample index;

20. The apparatus of claim 17, wherein the reading module is further configured to:

obtaining an object corresponding to a target field in the sample list;

21. The apparatus of claim 17, wherein the reading module is further configured to:

acquiring a field name of a target field;

22. The apparatus of claim 17, further comprising:

23. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any of claims 1 to 11.

24. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 11.