JP2022510643A - Storage and reading methods, devices, electronic devices and storage media - Google Patents

Abstract

The embodiments of the present application relate to storage and reading methods, devices, electronic devices and storage media. The storage method determines the descriptive information corresponding to the sample set based on the label of the sample in the sample set, and the field information corresponding to the sample set based on the field and storage position of the sample in the sample set. And to determine the configuration file of the sample set based on the descriptive information and the field information.

Description

(Mutual reference of related applications)
The present application claims priority based on the Chinese patent application filed October 28, 2019, with application number 201911031019.1. The entire contents of the Chinese patent application are incorporated herein by reference.

The present application relates to the field of computer technology and relates to storage and reading methods, devices, electronic devices and storage media.

In machine learning, the storage method of a data set is extremely important, especially when it is necessary to process a plurality of types of data sets. In related arts, the storage of a data set is generally to adapt to the structure of the data set itself, for example, CSV (Comma-Separated Values: values separated by commas), HDF5 (Higherarchical Data Format Version 5 :). There are data storage methods such as hierarchical data format version 5) and file storage (image audio).

The embodiments of the present application provide storage and reading methods, devices, electronic devices and storage media.

According to one aspect of the present application, a storage method is provided. The method determines the descriptive information corresponding to the sample set based on the label of the sample in the sample set, and the field corresponding to the sample set based on the field and storage position of the sample in the sample set. It includes determining the information and determining the configuration file of the sample set based on the descriptive information and the field information.

According to the reading method of the embodiment of the present application, the description information of the sample set and the field information of the sample can be determined, the configuration file can be determined based on the description information and the field information, and the configuration file can be generated. It supports storing samples of multiple data types in one sample set and can be adapted to store more complex types of sample sets. According to the configuration file, information on each type of sample data in the sample set can be obtained, which contributes to reading the sample data.

In a possible implementation, the descriptive information includes descriptive information for the label of the sample, where determining the descriptive information corresponding to the sample set based on the label of the sample in the sample set is the sample. To obtain a label list based on the labels of the samples in the set, the label list includes the categories of the labels of the sample and adds a description to the labels in the label list of the sample. Includes getting descriptive information on the label. In such a method, the description information of the label of the sample is obtained for the label of the sample, the meaning represented by the label of each sample is matched, and it contributes to the storage and reading of the sample.

In a possible implementation, the sample set includes samples of at least one data type.

In a possible embodiment, the field information further includes reading method information for the sample, wherein the method determines the reading method corresponding to each data type based on the data type of the sample in the sample set. It further includes obtaining reading method information for the sample based on the data type of the sample and the corresponding reading method.

In this way, it is possible to determine how to read multiple types of samples, which contributes to the storage of multiple types of data, adapts to the storage of more complex types of data sets, and at the same time, when reading, the configuration file. It contributes to reading the sample by acquiring an appropriate reading method based on.

In a possible implementation, determining the descriptive information corresponding to the sample set based on the label of the sample in the sample set is to determine the index range information for separating the categories of the samples in the sample set. Further includes generating descriptive information corresponding to the sample set based on the descriptive information of the sample in the sample set and the index range information of the sample category.

When reading a sample in such a manner, it contributes to looking up the target sample in a specific index range and improves reading efficiency.

In a possible implementation, the field information comprises a field list of a plurality of fields of the sample, wherein the field information corresponding to the sample set is determined based on the field and storage position of the sample in the sample set. That includes performing a grouping process on the fields of a plurality of samples in the sample set to obtain the field list.

In this way, each field can be grouped into a field list, which contributes to reading.

In a possible implementation, the storage location includes the storage path of the sample set and the index position of the sample within the sample set. With such a method, an accurate storage position of each sample can be obtained, which contributes to reading.

In a possible implementation, the field information includes instructional information indicating whether to allow reading of multiple samples.

In such a method, it is possible to instruct whether or not to allow reading of a plurality of samples at once, which contributes to reading the samples.

According to one aspect of the present application, a reading method is provided. The method obtains the storage position and target field of the target sample in the sample set based on the configuration file corresponding to the sample set, and the target parameter value of the target field of the target sample based on the storage position. The configuration file is determined based on the description information of the sample set and the field information of the sample.

In a possible implementation, the method further comprises acquiring a method of reading the target sample based on the configuration file.

In a possible implementation, reading the target parameter value of the target field of the target sample based on the storage location is based on the storage location of the target sample and the storage path of the sample set and the target sample in the sample set. Based on the storage path of the sample set and the index position of the target sample in the sample set, look up the target sample, and based on the reading method of the target sample. It includes reading a target sample to obtain parameter values for one or more fields of the target sample, and determining the target parameter values from the parameter values of one or more fields of the target sample.

In a possible implementation, reading the target parameter value of the target field of the target sample based on the storage location is the storage path of the sample set and the target sample in the sample set based on the storage position of the target sample. To obtain the index position of, and to obtain the storage position of the parameter value of the target field of a plurality of samples in the sample set based on the storage path of the sample set and the target field, and the target in the sample set. Based on the index position of the sample, the storage position of the parameter value of the target field of the target sample is determined from the storage position of the parameter value of the target field of the plurality of samples, and based on the reading method of the target sample. It includes reading the target parameter value from the storage position of the parameter value of the target field of the target sample.

In a possible implementation, the configuration file contains instructional information indicating whether to allow reading of multiple samples, and may read the target parameter value of the target field of the target sample based on the storage location. To acquire the predetermined read mode of the target sample, and to read the target parameter value from the storage position of the parameter value of the target field of the target sample based on the predetermined read mode and the instruction information. include.

In a possible embodiment, reading the target parameter value from the storage position of the parameter value in the target field of the target sample based on the predetermined read mode and the instruction information means that the predetermined reading mode is used as the instruction information. If they do not match, the reading method corresponding to the instruction information is used to read the parameter value of the target field of the target sample.

According to one aspect of the present application, a storage device is provided. The device is
A first decision component configured to determine the descriptive information corresponding to the sample set based on the label of the sample in the sample set.
A second decision component configured to determine the field information corresponding to the sample set based on the field and storage location of the sample in the sample set.
It comprises a third decision component configured to determine the configuration file of the sample set based on the descriptive information and the field information.

In a possible implementation, the descriptive information includes descriptive information for the label of the sample.
Here, the first determination component further
To obtain a label list based on the labels of the samples in the sample set, the label list includes categories of labels for the samples.
It is configured to add a description to the label in the label list, obtain the description information of the label of the sample, and execute.

In a possible implementation, the sample set includes samples of at least one data type.

In a possible implementation, the field information further includes sample reading method information.
Here, the device is
A fourth decision component configured to determine the reading method for each data type based on the data types of the samples in the sample set.
It further comprises a first acquisition component configured to obtain reading method information for the sample based on the data type of the sample and the corresponding reading method.

In a possible implementation, the first decision component further comprises.
Determine the index range information to separate the categories of samples in the sample set,
It is configured to generate descriptive information corresponding to the sample set based on the descriptive information of the sample in the sample set and the index range information of the sample category.

In a possible implementation, the field information comprises a field list of multiple fields in the sample.
Here, the second determination component further
It is configured to perform a collective process on the fields of a plurality of samples in the sample set to obtain the field list.

In a possible implementation, the storage location includes the storage path of the sample set and the index position of the sample within the sample set.

In a possible implementation, the field information includes instructional information indicating whether to allow reading of multiple samples.

According to one aspect of the present application, a reading device is provided. The device is
A first acquisition component configured to acquire the storage location and target field of the target sample in the sample set, based on the configuration file corresponding to the sample set.
It comprises a first read component configured to read the target parameter value of the target field of the target sample based on the storage location.
The configuration file is determined based on the description information of the sample set and the field information of the sample.

In a possible embodiment, the device is
Further including acquiring the reading method of the target sample based on the configuration file.

In a possible embodiment, the first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Look up the target sample based on the storage path of the sample set and the index position of the target sample in the sample set.
Based on the target sample reading method, the target sample is read, and the parameter values of one or more fields of the target sample are obtained.
It is configured to determine the target parameter value from the parameter values of one or more fields of the target sample.

In a possible embodiment, the first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Based on the storage path of the sample set and the target field, the storage position of the parameter value of the target field of a plurality of samples in the sample set is acquired.
Based on the index position of the target sample in the sample set, the storage position of the parameter value of the target field of the target sample is determined from the storage position of the parameter value of the target field of the plurality of samples.
Based on the target sample reading method, the target parameter value is read from the storage position of the parameter value in the target field of the target sample.

In a possible embodiment, the configuration file contains instructional information indicating whether to allow reading of multiple samples.
The first read component further
The predetermined read mode of the target sample is acquired, and the predetermined read mode is acquired.
Based on the predetermined reading mode and the instruction information, the target parameter value is read from the storage position of the parameter value in the target field of the target sample.

In a possible embodiment, the first read component further
When the predetermined reading mode does not match the instruction information, the reading method corresponding to the instruction information is used to read the parameter value of the target field of the target sample.

According to one aspect of the present application, an electronic device is provided. The electronic device is
With the processor
With memory configured to store instructions that can be executed by the processor,
The processor is configured to perform the storage and / or read method.

According to one aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the storage and / or reading method is realized.

According to one aspect of the present application, a computer program is provided. The computer program includes a computer-readable code, and when the computer-readable code is executed in the electronic device, the processor in the electronic device executes the storage and / or reading method.

It should be understood that the general description above and the details described below are for illustration and illustration purposes only and are not intended to limit the present application.

Other features and embodiments of the present application will become apparent with reference to the detailed description of exemplary embodiments based on the drawings below.

It is a flowchart which shows the storage method by the Example of this application. It is a schematic diagram which shows the list in the Example of this application. It is a flowchart which shows the reading method by the Example of this application. It is a schematic diagram which shows the application of the reading method by the Example of this application. It is a block diagram which shows the storage device by an Example of this application. It is a block diagram which shows the reading apparatus by the Example of this application. It is a block diagram which shows the electronic device by an Example of this application. It is a block diagram which shows the electronic device by an Example of this application.

The drawings attached herein are incorporated into the specification to form a portion of the specification, show examples conforming to the present application, and are used together with the specification to interpret the technical solutions of the present application.

Hereinafter, various exemplary embodiments, features and embodiments of the present application will be described in detail with reference to the drawings. The same reference numerals in the drawings indicate elements having the same or similar functions. The drawings show various aspects of the embodiments, but are not necessarily in proportion to the drawings unless otherwise specified.

The term "exemplary" as used herein means "as an example, as an example, or for illustration purposes". Here, any embodiment described as "exemplary" should not necessarily be construed as more suitable or advantageous than other embodiments.

As used herein, the term "and / or" is used to describe the relationship of a related object and indicates that there are three types of relationships. For example, A and / or B represent three cases: that only A exists, that A and B exist at the same time, and that only B exists. Also, as used herein, the term "at least one" refers to any one of the plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C means containing any one or more elements selected from the set consisting of A, B, and C.

In addition, in order to better explain the embodiment of the present application, many specific details have been described in the following specific embodiments. Those skilled in the art should understand that the embodiments of the present application are similarly feasible, regardless of these specific details. In order to clarify the gist of the embodiments of the present application, the methods, means, elements and circuits familiar to those skilled in the art will not be described in detail in some examples.

FIG. 1 is a flowchart showing a storage method according to an embodiment of the present application. As shown in FIG. 1, the method includes:

In step S11, the descriptive information corresponding to the sample set is determined based on the label of the sample in the sample set.

In step S12, the field information corresponding to the sample set is determined based on the field and the storage position of the sample in the sample set.

In step S13, the configuration file of the sample set is determined based on the description information and the field information.

According to the reading method of the embodiment of the present application, the description information of the sample set and the field information of the sample can be determined, the configuration file can be determined based on the description information and the field information, and the configuration file can be generated. It supports storing samples of multiple data types in one sample set and can be adapted to store more complex types of sample sets. According to the configuration file, information on each type of sample data in the sample set can be obtained, which contributes to reading the sample data.

In a possible implementation, the execution subject of the storage method may be a terminal device, a server, or another processing device. Here, the terminal device includes a user device (User Equipment: UE), a mobile device, a user terminal, a terminal, a cellular telephone, a cordless telephone, a personal digital assistant (PDA), a handheld device, a computing device, and an in-vehicle device. , Wearable devices, etc. may be used. In a possible embodiment, the method may be implemented by calling a computer-readable instruction stored in memory by a processor, or may be executed by a server. For example, the data in the server can be stored by the storage method.

In a possible implementation, the sample set may include a plurality of samples. The data type of the sample may include images, videos, numbers or text data and the like. The storage format of the sample may include CSV (Comma-Separated Values: values separated by commas), HDF5 (Hierarchical Data Format 5 hierarchical data format version 5), etc., and examples of the present application include sample data types and samples. Does not limit the storage format. The sample set can store each sample in the form of a list. That is, each sample can occupy a row of storage in the list. The storage area of the row may contain one or more fields of the sample. Alternatively, each sample can be stored in a dictionary, sequence, or other format. The embodiments of the present application do not limit the storage method. When storing a sample set, a configuration file corresponding to the sample set can be generated based on information such as a description of the sample set input by the user, or based on the sample information of the stored sample set. , The configuration file corresponding to the sample set can be automatically generated.

FIG. 2 is a schematic diagram showing a list in the embodiments of the present application. As shown in FIG. 2, the list can have sample indexes as row coordinates and fields as column coordinates. For example, one sample (eg, sample 1) is the data of a person. The field of the sample may include the person's height (eg, field 1), weight (eg, field 2), educational background (eg, field 3), and the like. In addition, the sample may have data corresponding to each field. The embodiments of the present application do not limit the fields.

In a possible implementation, the sample set may include a csv file. The csv file can store information such as numerical data, category data, character data, and data path in a table format. The sample set may include audio / audio image files such as audio, video, and images. For example, an AU file, an MP4 file, a JPEG file, or the like may be included. The sample set may further include HDF5 files and can record the file formats of files such as audio, video and images. The sample set may further include a txt file. The txt file can store character type data and the like. The sample set may contain at least one of the above types of files. The samples in the sample set are the data contained in the file. In the example, the sample set may contain multiple types of subsets. For example, one subset is a csv file, the file contains a plurality of numerical data samples, and the other subset is a subset consisting of image samples and the like. In the example, each subset can store a sample in the form of a list. For example, the csv file contains a list of multiple numeric data samples. A subset of image samples contains a list of multiple image samples. You can also put together a list of multiple subsets to get a list that stores all the samples in the sample set. The embodiments of the present application do not limit the format of the list in the sample set.

In a possible implementation, the sample set may contain samples of multiple data types. Each type of sample may have a particular storage and reading scheme. When storing multiple types of samples in a sample set, the sample set can be described by the field information corresponding to the sample set, and multiple types of samples can be stored or read. In the example, the sample set may include a plurality of videos, and the field information may include a description of the video storage or reading method.

In a possible implementation, the sample in the sample set may be a training sample used when training a neural network. The sample set may have descriptive information that describes the label. The descriptive information may be descriptive information in the metadata meta format. The embodiments of the present application do not limit the type of descriptive information. In the example, each sample may have a label. For example, it may have labels to represent information such as sample categories, attributes or probabilities. For example, the sample may include a facial image sample. When performing identity verification by a neural network, the neural network compares each face image sample with a reference image (for example, a face image of one target person) and determines the degree of similarity between the face image sample and the reference image. If the similarity is greater than the similarity threshold (eg, 0.5), it is determined that the face image sample has a high similarity to the reference image and the identity verification has passed, otherwise the identity verification is not passed. When training the neural network, each face image sample is labeled and parameters are provided for the loss function of the neural network. For example, when one face image sample is a face image of a target person, it is labeled as 1, indicating that the face in the face image sample is the face of the target person. When one face image sample is not the face image of the target person, it is labeled as 0 to indicate that the face in the face image sample is not the face of the target person.

In a possible embodiment, the descriptive information includes descriptive information for a label of the sample, wherein step S11 is to obtain a label list based on the label of the sample in the sample set, said label. The list includes that the category of the label of the sample is included, and that a description is added to the label in the label list to obtain the description information of the label of the sample.

In a possible embodiment, the descriptive information of the label of the sample may be the information descriptive of the label. You can get a label list by putting together the labels for each sample. In an example, the sample may include a face image sample, where each face image sample is labeled 0 or 1, and in the sample labeled 0, the face in the face image sample is not the face of the target person. And the sample labeled 1 can represent that the face in the face image sample is the face of the target person. Labels can be grouped together, for example, labels 0 and 1 can be grouped together in a label list. That is, the label list contains two columns, one column is 0 and the other column is 1. The embodiments of the present application do not limit the type of label and the type of label list. Of course, in other embodiments, label names, that is, label descriptions can be combined to generate a list of label names (label_names) for each sample included in the sample set.

In a possible implementation, a description can be added to each label in the label list. For example, add the description "The face in the face image sample is not the face of the target person" to the row labeled 0, and "The face in the face image sample is the face of the target person" in the row labeled 1. "There is" can be added. After adding the description to each label, the description information (label_imap) of the label of the sample can be obtained. The above labels and descriptions are examples only, and the embodiments of the present application do not limit the types and descriptions of labels.

In a possible implementation, in the process of storing a sample in the sample set, a description is added to the label of the sample in a unified manner. For example, add the description "The face in the face image sample is not the face of the target person" to the row labeled 0, and "The face in the face image sample is the face of the target person" in the row labeled 1. "There is" is added. These are not replaceable. Descriptions can be added to multiple types of data in a consistent manner.

In a possible implementation, the descriptive information on the sample label may be configured as dictionary data. That is, the key of the dictionary is the label, the value of the dictionary is the description for the label, and the embodiment of the present application does not limit the data type of the description information of the label of the sample.

In such a method, the description information of the label of the sample is obtained for the label of the sample, the meaning represented by the label of each sample is unified, and it contributes to the storage and reading of the sample.

In a possible implementation, the descriptive information further includes index range information for separating the categories of each sample in the sample set. The process of determining the descriptive information by the above method determines the index range information for dividing the category of each sample in the sample set, and the descriptive information of the sample in the sample set and the index range information of the sample category. Based on this, it further includes generating descriptive information corresponding to the sample set. For example, the categories can be separated based on the label of the sample. For example, the sample set can be divided into a training set, a verification set, and a test set. The training set contains 5000 samples, where the label of 3000 samples is 1 and the label of 2000 samples is 0. Alternatively, the labels can be divided directly based on the number and index of the samples in the sample set without dividing the labels. For example, the samples in the sample set are training samples used when training a neural network. There may be, and the training sample can be divided into multiple categories. For example, 40% of the samples can be the training set, 30% of the samples can be the test set, and 30% of the samples can be the verification set. Alternatively, the index range of each category can be determined separately based on the index of each sample. For example, the samples store each sample in the form of a list. 10,000 samples are stored in the list, the 1st-4000th sample in the list is the training set, the 4001-7000th sample in the list is the test set, and the 7001-10000th sample is the validation set. Can be. Here, the description information may include index range information (for example, 1st-4000th, 4001st-7000th, 7001st-10000th, etc.) that divides the sample category. The embodiments of the present application do not limit the division method and the index range.

In addition to the sample subsets of the three sample categories of the training set, the test set, and the verification set, the sample set may further include other types of data subsets such as image subsets, video subsets, and the like. The descriptive information of the sample set is not only the descriptive information of the sample subsets of the three sample categories of the above training set, test set, and verification set, but also the descriptive information of other types of data subsets such as image subsets, video subsets, and the like. Descriptive information may be further included.

When reading a sample in such a manner, it contributes to looking up the target sample in a specific index range and improves reading efficiency.

The descriptive information includes, but is not limited to, the above descriptive information, and may include other types of descriptive information such as sample outline information, attribute information, occupied space information, and the like. The embodiments of the present application do not limit the information contained in the descriptive information.

In a possible implementation, in step S12, field information (fields) of the sample can be determined based on the field and storage location of the sample, eg, each sample file included in the sample set (eg, CSV file, etc.). Based on the sample fields and storage locations in the HDF5 file), the field information for each sample file can be determined, thereby obtaining all the field information for the sample set. Alternatively, the field information of each type of sample is determined based on the field and storage position of each type of sample (eg, picture, video, text, etc.) in the sample set, thereby providing all the field information of the sample set. Obtainable. Of course, it is also possible to obtain the field information of the entire sample set based on the fields and storage positions of all the samples in the sample set. Each sample in the sample set may contain one or more fields. For example, one sample is data of a person's material, and the fields of the sample may include the person's height, weight, educational background, nationality, and the like. Another sample is a facial image of a person, and the fields of the sample may include the person's identity information and the like. The embodiments of the present application do not limit the fields.

In a possible implementation, the field information includes a field list (name) of a plurality of fields of the sample, where step S12 performs a collective process on the fields of each sample in the sample set. Includes getting a field list. For example, you can put all the fields together in a field list. For example, fields such as height, weight, educational background, and nationality of a person can be entered in the field list, and fields such as identity information of a facial image can also be entered in the field list. The field list can represent the fields of a plurality of samples in the sample set. For example, a sample set contains multiple sample files or multiple types of samples. Each sample file or each type of sample may correspond to one field list. Each field list contains the fields contained in the corresponding sample file or multiple samples of the corresponding type. Alternatively, one field list may include fields for all samples in the sample set. Each field corresponds to a row of data in the list of samples. In some areas of the data in the column, the data may be blank. For example, the list of samples may include a "nationality" field, and in a sample of a person's data, the field may contain data such as "China", "US", and the like. In the face image sample, there is no data in the field. That is, in the list, the data is blank in the area corresponding to the row in which the face image sample is located and the column in which the "nationality" field is located.

In a possible implementation, the field information may include recording the storage location of the sample. The storage position includes the storage path of the sample set and the index position of the sample in the sample set. For example, the storage path for the sample set is that the sample set is stored in the "sample set" folder on the C drive of the computer.

In a possible implementation, the index position of the sample in the sample set is the specific position of the sample. In the example, the data of each sample may be stored in the sample set in the form of an object. That is, it occupies one storage area in the sample set. The sample set can be stored in the form of a list for each sample, in which case the sample data can occupy a row in the sample set. The index position can represent a line number. For example, the first sample is located in the first row of the list in the sample set, the second sample is located in the second row of the list in the sample set ... The field information includes the storage path of the sample set and each. The index position of the sample may be included. The specific position of each sample can be determined based on the field information. For example, the second sample is located on the second line of the list of sample sets in the "Sample Sets" folder on the computer's C drive. Also, for example, some samples may have only some fields, or multiple samples belong to the same type or the same sample file. The index position may be a matrix range of samples in a sample set or a matrix range of a plurality of samples. The embodiments of the present application do not limit the storage position.

In a possible implementation, the sample set includes samples of at least one data type. For example, samples of types such as CSV, HDF5, txt, and other images and images are included. The field information further includes reading method information for each sample, wherein the method determines a reading method for reading a sample based on the data type of the sample in the sample set. Further including obtaining sample reading method information based on the data type and the corresponding reading method.

In a possible implementation, the reading method information may be information describing a reading method of the sample. The reading method of the sample can be determined based on the data type of the sample. For example, if the data type of one sample is an image, the sample can be read by an image reading function. If the data type of one sample is video, the sample can be read by a video reading function. The reading method of each sample can be determined individually. You can also combine the data types of each sample. For example, a sample set or a sample file contained in a sample set contains a plurality of samples, and the plurality of samples have three data types, which are images, videos, and general data, respectively. It can be determined that the image reading function reads the image type sample, the video reading function reads the video type sample, and the data reading function reads the common data type sample. Therefore, the reading method information of the sample set or the sample file may include the above data type and the corresponding reading method. The embodiments of the present application do not limit the data type and reading method of the sample.

In a possible implementation, the reading method information (proc) can be obtained based on the data type of the sample and the corresponding reading method. In the example, the reading method information can be configured as dictionary type data. For example, the key of the dictionary is the data type, and the value of the dictionary is the reading method corresponding to the data type. For example, the key of the dictionary is the image and the value of the dictionary is the image reading function. The embodiments of the present application do not limit the data type of the reading method information.

In a possible implementation, the field information may not include reading method information. When reading a sample, the reading method can be determined based on information such as the extension of the sample file. For example, if the extension of one sample file is txt, the sample can be read by a text opening method. If the extension of one sample file is jpg, the sample can be read by the picture opening method. The embodiments of the present application do not limit the extension and opening method of the sample.

In this way, it is possible to determine how to read multiple types of samples, which contributes to the storage of multiple types of data, adapts to the storage of more complex types of data sets, and at the same time, when reading, the configuration file. It contributes to reading the sample by acquiring an appropriate reading method based on.

In a possible embodiment, the field information further includes instructional information (fulllonly) instructing whether to allow reading of multiple samples. The instruction information can indicate whether or not the data of a plurality of samples can be read at once. For example, when the instruction information indicates that the data of all the samples in the sample set or the sample file can be read at once, when reading the data, all the samples in the sample set or the sample file can be read. .. For example, an HDF5 file supports reading samples in bulk throughout the file. The instruction information can indicate that the sample data in the sample set or sample file cannot be read in a batch, and when reading the data, all the samples in the sample set or sample file can be read in a batch. It supports reading at any time as needed. For example, read the data of only one sample at a time, or read the data of some samples.

In a possible implementation, in step S13, the configuration file (index.yml) of the sample set can be determined based on the descriptive information and the field information. In the example, the contents of the descriptive information and the field information can be entered in the configuration file, or the descriptive information and the field information can be packaged to obtain the configuration file. The embodiment of the present application does not limit the acquisition method of the configuration file.

In a possible implementation, the configuration file may be a file that records multiple types of storage schemes. That is, the configuration file contains information such as fields, labels, storage locations and reading methods for each sample. Each sample of each data type may be stored in a sample set and information such as fields, labels, storage locations and reading methods can be stored in the configuration file for reading. The configuration file provides the basis for storing and reading samples of multiple data types.

FIG. 3 is a flowchart showing a reading method according to the embodiment of the present application. As shown in FIG. 3, the method includes:

In step S14, the storage position and the target field of the target sample in the sample set are acquired based on the configuration file corresponding to the sample set.

In step S15, the target parameter value of the target field of the target sample is read based on the storage position.

Here, the configuration file is determined based on the description information of the sample set and the field information of the sample.

According to the reading method of the embodiment of the present application, the storage position and the target field of the target sample can be determined based on the configuration file, and the target parameter of the target sample can be read. Based on the configuration file, you can read the samples of each category from the sample set, allowing different types of samples to be read by the configuration file, without the need to place a program interface for each category of samples. Contributes to the storage and reading of sample data and improves reading efficiency. According to the configuration file, it supports storage and reading of sample files of multiple data types (eg, CSV, HDF5, samples of other file (image, audio) types), and within a more complex type of sample set. Can be adapted to read samples in.

In a possible implementation, the configuration file may be a file for describing the samples in the sample set, and the configuration file can be used to read the samples in the sample set. The configuration file may include descriptive information and field information corresponding to the sample set. When reading a sample, the sample can be read based on the description information and the field information. The sample set may contain samples of multiple data types. For example, samples of types such as CSV, HDF5, and other files (image audio) may be included. The descriptive information may be descriptive information for the sample label, and may include a list of labels for each sample in the sample set, descriptive information for the labels, and index range information for separating the categories of each sample in the sample set. good. The field information may be information for describing a sample. The field information can record the field list, data type, reading method, storage path, index position, etc. of a plurality of types of samples. A sample such as a target sample can be read based on the descriptive information and the field information. Any sample of each data type can be read by the above method. Therefore, it is possible to support multiple types of data reading.

In a possible implementation, the sample or sample field data in the sample set can be read based on the configuration file. In step S14, the sample set can store each sample in the form of a list. From the field information of the configuration file, the stored information of the target sample to be read and the target field to be read can be obtained. In step S15, the target parameter value of the target field of the target sample is read.

In a possible implementation, when reading a target sample, the reading method of the target sample can be determined based on the name or extension of the sample file to which the target sample belongs. For example, if the extension of one sample file is txt, the sample can be read by a text opening method. When the extension of one sample file is jpg, the sample can be read by the picture opening method. Alternatively, the field information in the configuration file may include a sample, a sample file, or a reading method for one data type. The method further comprises acquiring a reading method for the target sample based on the configuration file. That is, the sample reading method information can be obtained from the field information of the configuration file. For example, the reading method of the target sample can be obtained directly, or the corresponding reading method can be obtained based on the data type to which the target sample belongs (eg, image, text, etc.), followed by the reading. The method can read the target sample.

In a possible implementation, in step S14, the target sample can be looked up based on the storage position of the target sample, and the target sample and the target field of the target sample can be read by the target sample reading method. The target field may be any one or a plurality of fields of the target sample, or may be all fields of the target sample. Step S14 acquires the storage path of the sample set and the index position of the target sample in the sample set based on the storage position of the target sample, and the storage path of the sample set and the index of the target sample in the sample set. Looking up the target sample based on the position, reading the target sample based on the reading method of the target sample, and obtaining the parameter value of one or more fields of the target sample, and the target sample. The target parameter value may be determined from the parameter values of one or more fields of the above.

In a possible implementation, the storage location of the target sample can be determined. For example, the exact position of the target sample can be determined based on the storage path of the sample set recorded in the field information and the index position of the target sample in the sample list, and the exact position can be used to look up the target sample. .. For example, the target sample is sample 2, and the sample can be looked up from the second line of the list in the sample set in the "sample set" folder of the computer's C drive based on the storage location of the sample.

In a possible implementation, the target sample can be read by the reading method of the target sample recorded in the reading method information. For example, the target sample is text data, and the text data reading function can read the target sample and obtain the parameter values of one or more fields of the target sample. Further, the target parameter value of the target field can be determined from the parameter values of one or more fields. For example, it is possible to read the parameter values of the fields such as height, weight, educational background, and nationality of sample 2. Here, the target field is height. From the above parameter values, the parameter value of the height field can be determined.

In a possible implementation, in step S14, one field (target field) can be acquired from the field information of the configuration file, and the parameter values of the target fields of a plurality of samples can be acquired from the list. Furthermore, the parameter value of the target field of the target sample can be acquired. Step S14 acquires the storage path of the sample set and the index position of the target sample in the sample set based on the storage position of the target sample, and the sample is based on the storage path of the sample set and the target field. Based on the storage position of the parameter value of the target field of multiple samples in the set and the index position of the target sample in the sample set, the target is stored from the storage position of the parameter value of the target field of the multiple samples. Determining the storage position of the parameter value of the target field of the sample, and reading the target parameter value from the storage position of the parameter value of the target field of the target sample based on the reading method of the target sample. It may be included.

In a possible implementation, the storage position of the target sample can be determined based on the field information. For example, the exact position of the target sample can be determined based on the storage path of the sample set recorded in the field information and the index position of the target sample in the sample list. For example, the storage path of the sample set is the sample set in the "sample set" folder on the C drive of the computer, and the index position of the target sample is the second line of the list in the sample set.

In a possible implementation, the target field can be selected from the field information and the storage position of the parameter value of the target field can be acquired based on the storage path of the sample set and the target field. For example, the storage position of the parameter value of the height field can be obtained from the sample set in the "sample set" folder of the C drive of the computer. Also, the storage position of the target field of the target sample is determined based on the index position of the target sample in the sample set. For example, the storage position of the height field of the sample 2 is determined from the storage positions of a plurality of parameter values in the height field. That is, the storage position of the second line in the storage position of the plurality of parameter values in the height field is determined. Further, the target sample reading method can be obtained from the configuration file and the target parameter value can be read. For example, the data reading function can read the target parameter value at the storage position.

In a possible implementation, the target sample can also be read based on the label. For example, in the process of training a neural network, samples of each label type can be read according to a predetermined percentage. For example, a predetermined number of samples can be read. Here, 50% is a sample having a label of 1, and 50% is a sample having a label of 0. Alternatively, the sample can be read based on the index range information. For example, in the case of training, the sample in the training set can be read, in the case of the test, the sample in the test set can be read, and in the case of verification, the sample in the verification set can be read. The embodiments of the present application do not limit the reading method.

In a possible implementation, the configuration file contains instructional information indicating whether to allow reading of multiple samples, and may read the target parameter value of the target field of the target sample based on the storage location. To acquire the predetermined read mode of the target sample, and to read the target parameter value from the storage position of the parameter value of the target field of the target sample based on the predetermined read mode and the instruction information. include. In the example, when reading samples, the user can set a predetermined read mode, each sample can be read only when necessary, and all samples can be read at once. For example, for a sample such as an image, video, text, etc., the user can set it to be able to read at any time as needed, and set to read one or more samples respectively. Multiple samples can be read in sequence, or all samples can be read at once.

In a possible embodiment, reading the target parameter value from the storage position of the parameter value in the target field of the target sample based on the predetermined read mode and the instruction information means that the predetermined reading mode is used as the instruction information. If they do not match, the reading method corresponding to the instruction information is used to read the parameter value of the target field of the target sample. Samples in some data formats only support reading all samples at once. For example, for HDF5 format samples, it only supports reading all samples at once. In this case, if the instruction information of the sample file is instruction information that allows reading of a plurality of samples (for example, instruction information that supports reading all samples at once), the predetermined reading mode is necessary. If it can be read at any time, the predetermined reading mode does not match the instruction information, and the HDF5 format sample cannot be read in the predetermined reading mode. Therefore, it is possible to read all HDF5 format samples at once based on the instruction information of the sample of the type, and further, look up the target parameter value of the target field of the target sample from all the read samples. can do. Similarly, if samples in some data formats do not support reading all samples at once, the instruction information will be referred to as instruction information that does not allow reading of multiple samples (eg, all samples at once). Does not support reading). If the predetermined reading mode reads all the samples at once, the predetermined reading mode does not match the instruction information and the samples cannot be read in the predetermined reading mode. Therefore, sample data can be read at any time as needed, based on the instructional information of the type of sample.

According to the reading method of the embodiment of the present application, the description information of the label of the sample is obtained with respect to the label of the sample, the meaning represented by the label of each sample is matched, and it contributes to the storage and reading of the sample. In addition, it is possible to determine the reading method of a plurality of types of samples, which contributes to the storage of a plurality of types of data and adapts to the storage of a more complicated type of data set. In addition, the configuration file can be determined based on the description information and field information, and the samples of each category can be read from the sample set based on the configuration file, and for different types of samples, the sample description and fields can be determined by the configuration file. It allows information to be retrieved and read, does not require a program interface for each category of sample, contributes to the storage and reading of sample data, and can support multiple types of data storage. , Applies to the storage of more complex types of sample sets.

FIG. 4 is a schematic diagram showing the application of the reading method according to the embodiment of the present application. As shown in FIG. 4, a plurality of samples may be included in the sample set. The data type of the sample may include images, audio, video and / or general data (eg, numerical data, categorical data, character data) and the like. The sample set can store each sample in the form of a list. That is, each sample can occupy a row of storage in the list of samples. Each sample may contain one or more fields. For example, one sample is the data of a person. The field of the sample may include the person's height, weight, educational background, nationality, and the like. The sample may have data corresponding to each field.

In a possible implementation, the samples in the sample set may contain multiple data types. Each type of sample may have a particular storage and reading scheme, and each sample may have a corresponding label. Descriptive information may be included in the configuration file (index.yml). For example, descriptive information in the metadata meta format may be included. The description information can record the label of each sample and the description for the label.

In the example, the description information (label_imap) of the label of the sample may be dictionary type data, the key of the dictionary is the label, and the value of the dictionary is the description for the label. For example, the dictionary key is 0, the dictionary value is that the object in the sample is not the target object, the dictionary key is 1, and the dictionary value is that the object in the sample is the target object. be.

In the example, the descriptive information further includes index range information for separating the categories of each sample in the sample set (eg, sample subset subsets). For example, a sample in a sample set is a training sample used when training a neural network, and the training sample can be classified into a plurality of categories. For example, 40% of the samples can be the training set, 30% of the samples can be the test set, and 30% of the samples can be the verification set. The index range information determines the index range of the training set sample, the test set sample, and the validation set sample in the list of samples.

In a possible implementation, the configuration file may contain field information. The field information includes a field list (name) of a plurality of fields of the sample and a storage position of the sample. It is used to read sample data by field and storage location.

In the example, the field information may further include sample reading method information. The sample reading method information may be dictionary type data, the key of the dictionary is the data type of the sample, and the value of the dictionary is the reading method corresponding to the data type. For example, the dictionary key is an image, the dictionary value is an image reading function, the dictionary key is a video, and the dictionary value is a video reading function.

In the example, the fields of each sample can be grouped together. For example, one sample is the data of a person. The field of the sample may include the person's height, weight, educational background and nationality. Another sample is the person's face image sample. The field of the face image sample may include the person's identity information (eg, first and last name, number, etc.). You can get a field list by putting together all the above fields. For example, the field list includes fields such as height, weight, educational background, nationality and identity information.

In the example, the feed information may further include the storage location of the sample. The storage position of each sample can be determined. The storage position may include the storage path of the sample set and the index position of the sample in the sample list. For example, if the storage path of the sample set is the "sample set" folder on the C drive, and the index position of the target sample to be looked up is the second line in the sample list, the storage position of the sample is C. The second line of the list in the sample set in the drive's "sample set" folder.

In the example, the field information further includes instructional information (fulllonly) instructing whether to allow reading of all samples. The instruction information can indicate whether or not the data of all the samples can be read at once. For example, if the instruction information indicates that the data of all the samples can be read at once, when reading the data, all of the samples in the sample can be read. For example, you can read the entire sample list. If the instruction information indicates that the data of all the samples cannot be read at once, when reading the data, all the samples cannot be read at once, for example, only one at a time. The data of the sample can be read, or the data of a part of the sample can be read.

In a possible implementation, the sample or sample data in the sample set can be read based on the configuration file. For example, from the field information in the configuration file, read the index of the target sample to be read in the sample list, obtain the reading method of the target sample from the description file of the configuration file, and based on the index of the target sample in the list. The storage position of the target sample can be looked up, and one or more parameter values of the target sample or the target sample can be read by the target sample reading method. Further, for example, one field is acquired from the field information, and the storage position of the data of the field of a plurality of samples (the sample in which the field is blank may be ignored) is acquired from the list, and the storage position in the sample list is acquired. Based on the index of the target sample, the storage position of the data in the field of the target sample can be looked up, and the data can be read from the storage position of the field of the target sample by the reading method of the target sample.

In a possible implementation, the storage and reading methods are applicable to the storage of multiple types of samples. For example, in the field of computer education, by using the storage method, a plurality of types of samples can be stored by a matching storage method, which contributes to memory and reading, and contributes to operation by students. The storage method is not limited to the programming language environment, and different types of samples can be stored in a matched manner by using a configuration file.

Hereinafter, exemplary application of the embodiments of the present application in one actual application scene will be described.

In machine learning, the storage method of a data set is extremely important, especially when it is necessary to process a plurality of types of data sets. Different data sets need to be grouped together by the same protocol. There are no general-purpose protocols and rules in the storage method of current data sets. The storage method of a data set is generally a method that adapts to the structure of the data set itself and cannot be adapted to each other. Conventional data storage methods such as CSV, HDF5, and file storage (image audio) have strong functions and support limited data types, and cannot meet the storage demand for various data in a data set.

In machine learning, data sets are becoming more and more widespread. We need to find a data set storage protocol for storing data sets where most of the data is different and the metadata description is complex. This makes the entire data framework clearer and makes the description of each type of data in the data set more complete.

Accordingly, the embodiments of the present application provide a data set storage protocol. The purpose is to organize the data sets related to machine learning in the same way as possible, and to include the metadata of the data sets and the data file information of each kind. At the same time, the protocol can be ported and stored in different bottom layer data sets and adapted to multiple linguistic environments.

In a possible embodiment, the embodiments of the present application include:

1. The protocol provides a data set storage solution means by means of a configuration file. The protocol includes metadata and data set field information.

2. Metadata includes information such as data description, subset range, and data dictionary.

3. The data set field contains information of each kind of data file included in the data set.

In a possible embodiment, the embodiments of the present application include:

Each data set contains one configuration file (index.yml), which contains metadata information (meta) and data set field information (fields) of each class of data in the data set.

The metadata (meta) may include a plurality of pieces of information for describing the data set. In general, the following fields may be included (but not limited to).

1. List of label names of each sample (label_names). That is, a list of all label names.

2. Descriptive information (label_imap) of the label of the sample. In other words, it is a dictionary that has a one-to-one correspondence with label names and numeric labels.

3. Index range information (subsets) for each sample category. That is, in the case of a machine learning model, it is necessary to divide the data set into a training set, a test set, and a verification set. This includes the index range corresponding to the data. It can also store other types of data subsets.

The data set fields (fields) contain the following information.

1. Field list (name). That is, it is a field spot of the data in the data set.

2. Index position (file). That is, the file path where the data is located and the corresponding matrix range.

3. Reading method information (proc). That is, the data reading method (text, image, etc.) and data type.

4. Instruction information (fulllonly). That is, it is instructional information instructing whether or not to support reading all the data at once.

The program can extract the required information from the corresponding file based on the metadata and data set field information of the configuration file.

In a possible implementation, the storage data files of each class in the embodiments of the present application include:

1. csv file: Data stored in a table. In general, it may include information such as numerical data, categorical data, character data and paths of other data.

2. HDF5 file: Integrates all image data, audio data, etc. in the data set and provides support for fulllonly.

3. txt file: A file that stores character data.

4. Other image / video files: Generally, different data files (au files, MP4 files, JPEG files, etc.) are generated depending on the data type (audio, video, image, etc.).

In contrast to the drawback that there is no general purpose data set storage protocol in the prior art, the embodiments of the present application make it possible to store most machine learning data sets in a more consistent manner.

In contrast to the drawbacks of the prior art that it is generally limited to storage in a sub-aggregate language environment, the embodiments of the present application can be stored in different bottom layers and are not limited to the language environment.

In short, the embodiments of the present application provide a method of collecting data of different storage methods to form a data set, and are not limited to a specific programming language and environment.

Examples of the above methods referred to in the present application can be combined with each other as long as they do not deviate from the principle or logic, and the number of papers is limited. Should be understood.

The embodiments of the present application further provide storage and reading devices, electronic devices, computer-readable storage media, and programs. All of the above are for realizing any one of the storage and reading methods provided in the present application. Corresponding technical solutions and description may refer to description relating to the method, where detailed description is omitted.

In the above method of the embodiment, the description order of each step does not limit the execution process as a strict execution order, and the specific execution order of each step is determined by its function and possible intrinsic logic. Any trader should understand.

FIG. 5 is a block diagram showing a storage device according to an embodiment of the present application. As shown in FIG. 5, the apparatus is
A first decision component 11 configured to determine descriptive information corresponding to the sample set based on the label of the sample in the sample set.
A second decision component 12 configured to determine the field information corresponding to the sample set based on the field and storage position of the sample in the sample set.
It includes a third determination component 13 configured to determine a configuration file for the sample set based on the descriptive information and the field information.

In a possible implementation, the descriptive information includes descriptive information for the label of the sample.
Here, the first determination component further
To obtain a label list based on the labels of the samples in the sample set, the label list includes categories of labels for the samples.
It is configured to add a description to the label in the label list, obtain the description information of the label of the sample, and execute.

In a possible implementation, the sample set includes samples of at least one data type.

In a possible implementation, the field information further includes sample reading method information.
Here, the device is
A fourth decision component configured to determine the reading method for each data type based on the data types of the samples in the sample set.
It further comprises a first acquisition component configured to obtain reading method information for the sample based on the data type of the sample and the corresponding reading method.

In a possible implementation, the first decision component further comprises.
Determine the index range information to separate the categories of samples in the sample set,
It is configured to generate descriptive information corresponding to the sample set based on the descriptive information of the sample in the sample set and the index range information of the sample category.

In a possible implementation, the field information comprises a field list of multiple fields in the sample.
Here, the second determination component further
It is configured to perform a collective process on the fields of a plurality of samples in the sample set to obtain the field list.

In a possible implementation, the storage location includes the storage path of the sample set and the index position of the sample within the sample set.

In a possible implementation, the field information includes instructional information indicating whether to allow reading of multiple samples.

FIG. 6 is a block diagram showing a reading device according to an embodiment of the present application. As shown in FIG. 6, the apparatus is
A first acquisition component 14 configured to acquire the storage location and target field of the target sample in the sample set based on the configuration file corresponding to the sample set.
It comprises a first read component 15 configured to read the target parameter value of the target field of the target sample based on the storage location.
The configuration file is determined based on the description information of the sample set and the field information of the sample.

In a possible embodiment, the device is
Further including acquiring the reading method of the target sample based on the configuration file.

In a possible embodiment, the first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Look up the target sample based on the storage path of the sample set and the index position of the target sample in the sample set.
Based on the target sample reading method, the target sample is read, and the parameter values of one or more fields of the target sample are obtained.
It is configured to determine the target parameter value from the parameter values of one or more fields of the target sample.

In a possible embodiment, the first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Based on the storage path of the sample set and the target field, the storage position of the parameter value of the target field of a plurality of samples in the sample set is acquired.
Based on the index position of the target sample in the sample set, the storage position of the parameter value of the target field of the target sample is determined from the storage position of the parameter value of the target field of the plurality of samples.
Based on the target sample reading method, the target parameter value is read from the storage position of the parameter value in the target field of the target sample.

In a possible embodiment, the configuration file contains instructional information indicating whether to allow reading of multiple samples.
The first read component further
The predetermined read mode of the target sample is acquired, and the predetermined read mode is acquired.
Based on the predetermined reading mode and the instruction information, the target parameter value is read from the storage position of the parameter value in the target field of the target sample.

In a possible embodiment, the first read component further
When the predetermined reading mode does not match the instruction information, the reading method corresponding to the instruction information is used to read the parameter value of the target field of the target sample.

In some embodiments, the functions and modules in the apparatus provided in the embodiments of the present application are used to perform the methods described in the embodiments of the above methods, the specific embodiment of which is the embodiments of the above methods. Please refer to the explanation of. For the sake of brevity, detailed description is omitted here.

The embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the above method is realized. The computer-readable storage medium may be a volatile computer-readable storage medium or a non-volatile computer-readable storage medium.

The embodiments of the present application further provide electronic devices. The electronic device comprises a processor and a memory configured to store instructions that can be executed by the processor, the processor being configured to perform the method.

The electronic device may be provided as a terminal, a server or other form of device.

FIG. 7 is a block diagram showing an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a terminal such as a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a personal digital assistant.

Referring to FIG. 7, the electronic device 800 includes a processing unit 802, a memory 804, a power supply unit 806, a multimedia unit 808, an audio unit 810, an input / output (I / O) interface 812, a sensor unit 814, and the like. One or more of the communication units 816 may be provided.

The processing unit 802 generally controls the overall operation of the electronic device 800. For example, it controls operations related to display, call call, data communication, camera operation and recording operation. The processing unit 802 may include one or more processors 820 for executing commands. Thereby, all or part of the steps of the above method are performed. The processing unit 802 may include one or more modules for interaction with other units. For example, the processing unit 802 includes a multimedia component, which contributes to the interaction between the multimedia unit 808 and the processing unit 802.

The memory 804 is configured to support operations in the electronic device 800 by storing various types of data. Examples of these data include instructions, contact data, phonebook data, messages, images, videos, etc. of any application or method that can be manipulated on the electronic device 800. The memory 804 is realized by any type of volatile or non-volatile storage device, or a combination thereof. For example, static random access memory (Static Random Access Memory: SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable ReadOnly Memory: EEPROM), erasable programmable read-only memory (ErasableMemory) ), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The power supply unit 806 provides power to various units of the electronic device 800. The power supply unit 806 may include a power supply management system, one or more power supplies, and other units involved in power generation, management, and distribution for the electronic device 800.

The multimedia unit 808 includes a screen for providing an output interface between the electronic device 800 and the user. In some embodiments, the screen comprises a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, it is realized as a touch panel and receives an input signal from the user. The touch panel comprises one or more touch sensors that sense touches, slides and gestures on the panel. The touch sensor can not only detect the boundary of the touch or slide motion, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia unit 808 comprises a front camera and / or a rear camera. If the electronic device 800 is in an operating mode such as a shooting mode or a video mode, the front camera and / or the rear camera can receive multimedia data from the outside. Each front and rear camera may have a fixed optical lens system or focal and optical zoom capabilities.

The audio unit 810 is configured to output and / or input an audio signal. For example, the audio unit 810 includes a microphone (MIC). If the electronic device 800 is in an operating mode such as a call mode, a recording mode and a voice recognition mode, the microphone is configured to receive an audio signal from the outside. The received audio signal can be further stored in the memory 804 or transmitted via the communication unit 816. In some embodiments, the audio unit 810 further comprises a speaker configured to output an audio signal.

The I / O interface 812 provides an interface between the processing unit 802 and the peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, or the like. These buttons include, but are not limited to, a home button, a volume button, a start button and a lock button.

The sensor unit 814 comprises one or more sensors and is configured to perform various state assessments for the electronic device 800. For example, the sensor unit 814 can detect the on / off state of the electronic device 800 and the relative positioning of the unit. For example, the unit is a display and a keypad of an electronic device 800. The sensor unit 814 detects a change in the position of one unit in the electronic device 800 or the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature fluctuation of the electronic device 800. You can also do it. The sensor unit 814 may include a proximity sensor and is configured to detect the presence of surrounding objects in the absence of any physical contact. The sensor unit 814 may include an optical sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge-coupled Device) image sensor, and is applied to imaging. It is configured as follows. In some embodiments, the sensor unit 814 may include an accelerometer, gyro sensor, magnetic sensor, pressure sensor or temperature sensor.

The communication unit 816 is configured to contribute to wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 is a communication standard such as WiFi, 2G (2-Generation wireless telephone technology: 2nd generation mobile communication technology) or 3G (3-Generation wireless telephone technology: 3rd generation mobile communication technology), or a combination thereof. You can access the wireless network based on. In an exemplary embodiment, the communication unit 816 receives broadcast signals or broadcast-related information from an external broadcast channel management system via the broadcast channel. In an exemplary embodiment, the communication unit 816 further comprises a Near Field Communication (NFC) component to facilitate short-range communication. For example, NFC modules include Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth: Bluetooth, et al. It is realized based on the technology of.

In an exemplary embodiment, the electronic device 800 includes one or more Applied Specific Integrated Circuits (ASICs), a Digital Signal Process (DSP), and a Digital Signal Process. Implemented by a Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic element to perform the above method. It may be configured as follows.

In an exemplary embodiment, further provided is a non-volatile computer readable storage medium, such as memory 804, which stores computer program instructions. The computer program instruction is executed by the processor 820 of the electronic device 800 to complete the method.

The embodiments of the present application further provide computer program products. The computer program product includes a computer-readable code, and when the computer-readable code is executed in the device, the processor in the device executes an instruction for realizing the method provided in any one of the above embodiments. ..

The computer program product can be realized by hardware, software or a combination thereof. In one selectable embodiment, the computer program product is embodied as a computer storage medium, and in another selectable embodiment, the computer program product is, for example, a software development kit (SDK) or the like. It is embodied as a software product such as.

FIG. 8 is a block diagram showing another electronic device 1900 according to the embodiment of the present application. For example, the electronic device 1900 may be provided as a server. Referring to FIG. 8, the electronic device 1900 includes a processing unit 1922. It further comprises one or more processors and a memory resource represented by memory 1932. The memory lease is for storing instructions executed by the processing unit 1922, such as an application program. The application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. The processing unit 1922 is configured to execute the instruction and execute the above method.

The electronic device 1900 includes a power supply unit 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and input / output (I / O). O) Interface 1958 may be further provided. The electronic device 1900 can run an operating system stored in memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

In an exemplary embodiment, further provided is a non-volatile computer readable storage medium such as memory 1932 containing computer program instructions. The computer program instruction is executed by the processing unit 1922 of the electronic device 1900 to complete the method.

The embodiments of the present application may be systems, methods and / or computer program products. The computer program product may include a computer-readable storage medium, in which computer-readable program instructions for realizing each aspect of the embodiments of the present application are stored in the processor.

The computer-readable storage medium may be a tangible device capable of holding or storing instructions used in the instruction execution device. The computer-readable storage medium may be, for example, an electric storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above, but is not limited thereto. More specific examples (non-exhaustive lists) of computer-readable storage media are portable computer disks, hard disks, random access memory (RAM), read-only memory (Read Only Memory: ROM), and erasable. Programmable Read-Only Memory (Electrical Program Read Only Memory: EPROM or Flash), Static Random Access Memory (SRAM), Portable Compact Disk Read-Only Memory (Compact Computer Read-Only Memory: Multi-purpose) Includes a disk (Digital Visual Disk: DVD), a memory stick, a flexible disk, a mechanical coding device such as a punch card in which instructions are stored or a protrusion structure in a groove, and any suitable combination described above. The computer-readable storage medium used herein is a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, an optical pulse passing through an optical fiber cable), or through an electric wire. It should not be construed as a temporary signal itself, such as a transmitted electrical signal.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or networks such as the Internet, local area networks, wide area networks and / or wireless networks. It can be downloaded to an external computer or an external storage device via. The network may include copper cables for transmission, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. The network interface card or network interface in each computing / processing device receives computer-readable program instructions from the network, transfers the computer-readable program instructions, and stores them in a computer-readable storage medium in each computing / processing device.

The computer-readable program instructions for performing the operations of the embodiments of the present application are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or It may be source code or target code written in one or more programming languages. The programming language includes object-oriented programming languages such as Smalltalk, C ++, and conventional procedural programming languages such as "C" programming languages or similar programming languages. Computer-readable program instructions may be executed completely on the user computer, partially on the user computer, as a separate software package, or partially on the user computer. It may be executed partially on the remote computer, or it may be executed completely on the remote computer or the server. In the case of a remote computer, the remote computer connects to or connects to the user's computer through any type of network, including a local area network (LAN) or a wide area network (Wide Area Network: WAN). , Can connect to an external computer (eg, connect through the internet using an internet service provider). In some embodiments, the state information of computer-readable program instructions is used to customize electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs). The electronic circuit can realize each aspect of the present application by executing a computer-readable program instruction.

Here, each aspect of the present application will be described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments of the present application. Each block of the flowchart and / or the block diagram and each combination of the blocks in the flowchart and / or the block diagram can be realized by a computer-readable program instruction.

These computer-readable program instructions can be provided to the processor of a general purpose computer, dedicated computer or other programmable data processing device, thereby creating a device, and when these instructions are executed by the processor of the computer or other programmable data processing device, the flowchart. And / or created a device that realizes the functions / operations specified in one or more blocks in the block diagram. These computer-readable program instructions may be stored in a computer-readable storage medium. According to these instructions, computers, programmable data processing devices and / or other devices operate in a particular manner. Accordingly, a computer-readable storage medium in which instructions are stored comprises a product comprising instructions of each aspect that realizes a function / operation defined in one or more blocks in a flow chart and / or a block diagram.

Computer-readable program instructions may be loaded into a computer, other programmable data processing device, or other device. This causes a computer, other programmable data processing device, or other device to perform a series of steps of operation to create a process performed on the computer. Therefore, instructions executed by a computer, other programmable data processing device, or other device realize the functions / operations specified in one or more blocks in the flowchart and / or block diagram.

The flowcharts and block diagrams in the drawings illustrate the feasible architectures, functions and operations of the systems, methods and computer program products according to the plurality of embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent part of a module, program segment or instruction. Some of the modules, program segments or instructions include executable instructions to implement one or more predetermined logical functions. During implementation as some replacement, the functions shown in the blocks can occur in a different order than shown in the drawing. For example, two consecutive blocks can actually be executed essentially in parallel, and in some cases in opposite order, which is determined by the functions involved. Each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, is realized by a dedicated hardware-based system for performing a predetermined function or operation, or a dedicated hardware and a computer instruction. It can be realized by the combination of.

Although each embodiment of the present invention has been described above, the above description is exemplary, not exhaustive, and is not limited to the disclosed examples. Many modifications and changes are readily conceivable to those skilled in the art, provided that they do not deviate from the scope and intent of each of the embodiments described. The choice of terminology used herein best interprets the principles, practical applications, or technological improvements in the market of each embodiment, or each embodiment disclosed herein by one of ordinary skill in the art. The purpose is to be able to understand.

Claims (31)

It ’s a memory method,
Determining the descriptive information corresponding to the sample set based on the label of the sample in the sample set.
Determining the field information corresponding to the sample set based on the field and storage position of the sample in the sample set.
A storage method comprising determining a configuration file of the sample set based on the descriptive information and the field information. The descriptive information includes descriptive information for the label of the sample.
Here, it is not possible to determine the descriptive information corresponding to the sample set based on the label of the sample in the sample set.
To obtain a label list based on the labels of the samples in the sample set, the label list includes categories of labels for the samples.
The method according to claim 1, wherein a description is added to the label in the label list to obtain description information of the label of the sample, and the present invention includes. The method according to claim 1 or 2, wherein the sample set includes samples of at least one data type. The field information further includes sample reading method information.
Here, the method is
Determining the reading method for each data type based on the data types of the samples in the sample set,
The method of claim 3, further comprising obtaining reading method information for the sample based on the data type of the sample and the corresponding reading method. Determining the descriptive information corresponding to the sample set based on the label of the sample in the sample set is
Determining the index range information to separate the categories of samples in the sample set,
One of claims 1-4, further comprising generating descriptive information corresponding to the sample set based on the descriptive information of the sample in the sample set and the index range information of the sample category. The method described in paragraph 1. The field information includes a field list of multiple fields in the sample.
Here, determining the field information corresponding to the sample set is based on the field and storage position of the sample in the sample set.
The method according to any one of claims 1-5, which comprises performing a grouping process on a field of a plurality of samples in the sample set to obtain the field list. The method according to any one of claims 1-6, wherein the storage position includes a storage path of the sample set and an index position of a sample in the sample set. The method according to any one of claims 1-7, wherein the field information includes instructional information indicating whether or not reading of a plurality of samples is permitted. It ’s a reading method.
Obtaining the storage location and target field of the target sample in the sample set based on the configuration file corresponding to the sample set,
Including reading the target parameter value of the target field of the target sample based on the storage location.
The reading method, wherein the configuration file is determined based on the description information of the sample set and the field information of the sample. The method is
The method of claim 9, further comprising acquiring a reading method for the target sample based on the configuration file. Reading the target parameter value of the target field of the target sample based on the storage position is
To obtain the storage path of the sample set and the index position of the target sample in the sample set based on the storage position of the target sample.
To look up the target sample based on the storage path of the sample set and the index position of the target sample in the sample set.
Based on the target sample reading method, the target sample is read to obtain the parameter values of one or more fields of the target sample.
10. The method of claim 10, comprising determining the target parameter value from the parameter values of one or more fields of the target sample. Reading the target parameter value of the target field of the target sample based on the storage position is
To obtain the storage path of the sample set and the index position of the target sample in the sample set based on the storage position of the target sample.
To acquire the storage position of the parameter value of the target field of a plurality of samples in the sample set based on the storage path of the sample set and the target field.
To determine the storage position of the parameter value of the target field of the target sample from the storage position of the parameter value of the target field of the plurality of samples based on the index position of the target sample in the sample set.
The method according to claim 10, wherein the target parameter value is read from a storage position of a parameter value in a target field of the target sample based on the reading method of the target sample. The configuration file contains instructional information indicating whether to allow reading of multiple samples.
Reading the target parameter value of the target field of the target sample based on the storage position is
Acquiring the predetermined read mode of the target sample and
One of claims 9-12, comprising reading the target parameter value from a storage position of the parameter value in the target field of the target sample based on the predetermined read mode and the instruction information. The method described in paragraph 1. Reading the target parameter value from the storage position of the parameter value in the target field of the target sample based on the predetermined read mode and the instruction information is not possible.
13. The thirteenth aspect of the present invention, wherein when the predetermined reading mode does not match the instruction information, the reading method corresponding to the instruction information is used to read the parameter value of the target field of the target sample. Method. It ’s a storage device,
A first decision component configured to determine the descriptive information corresponding to the sample set based on the label of the sample in the sample set.
A second decision component configured to determine the field information corresponding to the sample set based on the field and storage location of the sample in the sample set.
A storage device comprising a third determination component configured to determine a configuration file for the sample set based on the descriptive information and the field information. The descriptive information includes descriptive information for the label of the sample.
Here, the first determination component further
To obtain a label list based on the labels of the samples in the sample set, the label list includes categories of labels for the samples.
The apparatus according to claim 15, wherein a description is added to a label in a label list to obtain description information of the label of the sample, and the device is configured to execute. The apparatus according to claim 15 or 16, wherein the sample set includes samples of at least one data type. The field information further includes sample reading method information.
Here, the device is
A fourth decision component configured to determine the reading method for each data type based on the data types of the samples in the sample set.
17. The apparatus of claim 17, further comprising a first acquisition component configured to obtain reading method information for the sample based on the data type of the sample and the corresponding reading method. The first decision component further
Determine the index range information to separate the categories of samples in the sample set,
One of claims 15-18, characterized in that it is configured to generate descriptive information corresponding to the sample set based on the descriptive information of the sample in the sample set and the index range information of the sample category. The device according to paragraph 1. The field information includes a field list of multiple fields in the sample.
Here, the second determination component further
The apparatus according to any one of claims 15-19, wherein the fields of a plurality of samples in the sample set are collectively processed to obtain the field list. The apparatus according to any one of claims 15-20, wherein the storage position includes a storage path of the sample set and an index position of a sample in the sample set. The apparatus according to any one of claims 15-21, wherein the field information includes instruction information indicating whether or not reading of a plurality of samples is permitted. It ’s a reading device,
A first acquisition component configured to acquire the storage location and target field of the target sample in the sample set, based on the configuration file corresponding to the sample set.
It comprises a first read component configured to read the target parameter value of the target field of the target sample based on the storage location.
The configuration file is a reading device determined based on the description information of the sample set and the field information of the sample. The device is
23. The apparatus of claim 23, further comprising acquiring a method of reading a target sample based on the configuration file. The first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Look up the target sample based on the storage path of the sample set and the index position of the target sample in the sample set.
Based on the target sample reading method, the target sample is read, and the parameter values of one or more fields of the target sample are obtained.
24. The apparatus of claim 24, wherein the target parameter value is determined from the parameter values of one or more fields of the target sample. The first read component further
Based on the storage position of the target sample, the storage path of the sample set and the index position of the target sample in the sample set are acquired.
Based on the storage path of the sample set and the target field, the storage position of the parameter value of the target field of a plurality of samples in the sample set is acquired.
Based on the index position of the target sample in the sample set, the storage position of the parameter value of the target field of the target sample is determined from the storage position of the parameter value of the target field of the plurality of samples.
24. The apparatus according to claim 24, wherein the target parameter value is read from a storage position of a parameter value in a target field of the target sample based on the reading method of the target sample. The configuration file contains instructional information indicating whether to allow reading of multiple samples.
The first read component further
The predetermined read mode of the target sample is acquired, and the predetermined read mode is acquired.
One of claims 23-26, which is configured to read the target parameter value from the storage position of the parameter value in the target field of the target sample based on the predetermined reading mode and the instruction information. The device according to paragraph 1. The first read component further
27. The device described. It ’s an electronic device,
With the processor
With memory configured to store instructions that can be executed by the processor,
The processor is configured to read an instruction stored in the memory and execute the method according to any one of claims 1 to 6. Electronics. A computer-readable storage medium that stores computer program instructions and realizes the method according to any one of claims 1 to 14 when the computer program instructions are executed by a processor. Medium. A computer program comprising a computer-readable code, wherein when the computer-readable code is executed in the electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 14. Computer program.

Images