CN115576948A

CN115576948A - Data storage method and device, electronic equipment and storage medium

Info

Publication number: CN115576948A
Application number: CN202211309918.5A
Authority: CN
Inventors: 狄晓帆
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2022-10-25
Filing date: 2022-10-25
Publication date: 2023-01-06

Abstract

The invention discloses a data storage method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a data table to be stored; determining the data volume of the data table to be stored and the service type matched with the data table to be stored; wherein the service type comprises online and/or batch; determining a storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored; according to the technical scheme, the data size of the data table to be stored is quantized, and the service type of the data table to be stored is matched. The occurrence of redundant situations during data export is reduced, unnecessary export and import pressure is further reduced, and the resource occupancy rate of equipment is reduced. And a reasonable storage mode is matched for the data table to be stored, so that the complexity of a storage framework is reduced.

Description

Data storage method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a data storage method and apparatus, an electronic device, and a storage medium.

Background

The information processing method includes online processing and batch processing. The online transaction is also called online transaction processing, is suitable for single event transaction, can quickly feed back a processing result, and has strong real-time interactivity. Online typically uses traditional relational databases, such as Oracle, DB2, etc., which use a line type storage method, data is stored in a logical storage unit based on line data, and data in a line exists in a continuous storage form in a storage medium. The batch analysis processing is also called on-line analysis processing, is suitable for a large batch of work tasks, has relatively long processing time and poor real-time performance, but has high efficiency and saves resource consumption. Batches typically employ analytical columnar storage databases such as GBase, hbase, clickhause, and the like. Such databases employ a columnar storage manner, data is stored in a column-based logical storage unit, and data in a column exists in a storage medium in a continuous storage form.

In order to satisfy both online and batch business requirements, a system typically deploys a line-based stored online database for online transactions and a line-based stored batch database for batch processing. By the arrangement, 2 sets of databases need to be considered in the processes of design, development, operation and maintenance and the like, the technical threshold and the cost are high, and the data are frequently exported and imported from the databases, so that the resource consumption of a central processing unit, a memory and the like of the databases is high. The prior art provides a TiDB database, which performs line storage and column storage on all data tables respectively, and uses a data synchronization mechanism between two pieces of data, and runs online on line-stored data and batch on column-stored data. However, the data synchronization pressure is large in the mode, data redundancy exists, and storage resources are wasted.

Disclosure of Invention

The invention provides a data storage method, a data storage device, an electronic device and a storage medium, which are used for realizing mixed storage of a row type and a column type, reducing the structural complexity of a database and the import and export pressure of data, and reducing the data synchronization pressure and data redundancy.

In a first aspect, an embodiment of the present invention provides a data storage method, where the method includes:

acquiring a data table to be stored;

determining the data volume of the data table to be stored and the service type matched with the data table to be stored;

wherein the service type comprises online and/or batch;

and determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored.

In a second aspect, an embodiment of the present invention further provides a data storage apparatus, including:

the data table acquisition module is used for acquiring a data table to be stored;

the attribute determining module is used for determining the data volume of the data table to be stored and the service type matched with the data table to be stored;

wherein the service types comprise online and/or batch;

the storage mode determining module is used for determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored;

wherein the storage mode comprises row-wise storage, column-wise storage or row-wise column-wise hybrid storage.

In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the data storage method according to any one of the embodiments of the present invention when executing the program.

In a fourth aspect, embodiments of the present invention also provide a storage medium storing computer-executable instructions, which when executed by a computer processor, are used to perform the data storage method according to any one of the embodiments of the present invention.

According to the technical scheme of the embodiment of the invention, the data volume of the data table to be stored is determined by a data volume calculation method through acquiring the data table to be stored, and the service type is matched for the data table to be stored according to the actual operation requirement of a user. And determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored. According to the embodiment of the application, the storage mode of the data table to be stored is determined by quantifying the data volume of the data table to be stored and matching the service type of the data table to be stored. And a reasonable matching storage mode is carried out on the data table to be stored, so that the pressure of data export and import is reduced, and the complexity of a storage framework is reduced.

It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart of a data storage method according to an embodiment of the present invention;

fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a data storage method according to an embodiment of the present invention, where this embodiment is applicable to a case where a data storage manner is determined, and the method may be executed by a data storage device, where the data storage device may be implemented in a form of hardware and/or software, and the data storage device may be configured in any device with storage capability and arithmetic capability.

As shown in fig. 1, the method includes:

s110, acquiring a data table to be stored;

the data table to be stored may be a data table to be stored in a database.

Illustratively, if the current data table to be stored is in a relational database, it is necessary to find out which database the data table to be stored should be in, retrieve and acquire the data table to be stored, and if the current data table is stored in a text form, it is necessary to write a data table to be stored generating program, and convert each piece of data in the text into a form specified by the data table to be stored.

S120, determining the data volume of the data table to be stored and the service type matched with the data table to be stored; wherein the service type comprises online and/or batch;

wherein the data amount may be all data pieces of the data table stored in the storage medium. Online may be the process of inputting data into the system by the terminal generating the data, and transferring the data directly to the device needing the data by the system processing. A batch may be an online analytical processing operation that processes a large batch of work tasks.

Optionally, determining the data amount of the data table to be stored may include: determining the number of fields of the data table to be stored, the field data length corresponding to each field and the number of data; and determining the data quantity of the data table to be stored according to the field quantity, the field data length corresponding to each field and the data number.

The number of fields may be the number of all attribute columns in the storage data table. The field data length may be the length in bytes when the computer allocates storage space for this field. The number of data strips may be all the number of rows in the stored data table.

Specifically, the number of all attribute columns in the data table to be stored is calculated as the number of fields, the data length of each field is determined according to the data type shown by each field, and each record in the data table to be stored is counted, that is, all rows in the data table to be stored are determined to determine the number of data. Alternatively, weights may be assigned to the number of fields, the field data length, and the number of data pieces, a field number evaluation value matching the number of fields may be determined according to the number of fields, an evaluation value of the field data length may be determined according to the field data length corresponding to each field, and a number evaluation value matching the number of data pieces may be determined according to the number of data pieces. And taking the sum of the products of each evaluation value and the weight thereof as the data quantity evaluation value of the data table to be stored.

Illustratively, the data size of the data table to be stored is determined according to the method, and the service type of the data table to be stored is determined to belong to online according to the customer requirements that the data table to be stored needs to support, for example, the user needs to quickly obtain a feedback result to perform real-time interactive operation, and the online service type can realize the requirements of the user. For example, if a user needs to perform online analysis and processing on a large batch of work tasks according to the data table to be stored and real-time interaction is not needed, and the batch service type can meet the user's requirement, it is determined that the service type of the data table to be stored belongs to batch.

S130, determining a storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored; wherein the storage mode comprises row-type storage, column-type storage or row-column-type hybrid storage.

The storage mode may be a mode of data in the storage space, and the storage mode is distinguished according to a physical address distribution condition of the storage space where the data is located. The line type storage is a storage form which is established on a file system of an operating system and is stored according to a logic storage unit based on line data, and the database in the line type storage form comprises an Oracle, DB2, mySQL and SQL SERVER database. Columnar storage is relative to row storage, where data is stored in column-based logical storage units and each column of storage units is stored in a sequential fashion, and the database in the form of columnar storage includes a distributed database such as Hbase. The row-column hybrid storage may be a storage mode for performing both row-column storage and column-column storage on the data table to be stored, and the storage mode enables the data table to be stored to be distributed in the row-column storage database and the column-column storage database at the same time, so that the storage mode has certain data redundancy. Meanwhile, when data modification, data addition, data deletion and the like are required, data synchronization of the line storage table and the column storage table is required.

Specifically, the data volume of the data table to be stored is determined through the steps, the service type of the data table to be stored is determined according to the customer requirement, and whether the data table to be stored is stored in a column form or a row form is determined.

For example, when data is written in the form of a row, since the row storage is stored in one row at a time and the column storage stores each field value of one row in turn per each field, the time cost consumed for the column storage is higher than that consumed for the row storage. When the data modification operation is carried out, the row storage modifies the data at the specified position once, and the address access complexity of the column storage is multiple times of that of the row storage. When data is read, redundant columns are easy to appear when a row storage operation storage data table reads several columns of data, and column storage is one section or all of a read data set, so that the redundancy problem does not exist. Inline storage is suitable because it is suitable for operations that read one or several lines. While batches are suitable for processing a column of data in many rows, for columnar storage.

According to the technical scheme of the embodiment of the invention, the data volume of the data table to be stored is determined by a data volume calculation method through acquiring the data table to be stored, and the service type is matched for the data table to be stored according to the actual operation requirement of a user. And determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored. According to the embodiment of the application, the storage mode of the data table to be stored is determined by quantifying the data volume of the data table to be stored and matching the service type of the data table to be stored. And the reasonable matching storage mode is carried out on the data table to be stored, so that the pressure of data export and import is reduced, and the complexity of a storage framework is reduced.

Example two

Fig. 2 is a flowchart of a data storage method according to a second embodiment of the present invention, where the embodiment of the present invention further embodies a method for selecting a storage mode of a to-be-stored data table on the basis of the above embodiment.

As shown in fig. 2, the method includes:

s210, acquiring a data table to be stored;

specifically, the data table to be stored is retrieved and derived according to the current storage address of the data table to be stored, or the data table to be stored is generated according to the data structure conversion of the current data to be stored.

S220, determining the data volume of the data table to be stored and the service type matched with the data table to be stored; wherein the service type comprises online and/or batch;

specifically, the data volume of the data table to be stored is determined according to the data volume determination method of the data table to be stored. And determining the service type matched with the data table to be stored according to the requirement type of the data table to be stored, which needs to respond to the client. The determination method of the data volume and the service type has been described in the above embodiments, and details of this embodiment are not described herein again.

And S230, judging whether the data volume of the data table to be stored is smaller than a preset first threshold value, if so, executing S240, and otherwise, executing S250.

The preset first threshold refers to a preset value which is the same as the data quantity type of the data table to be stored, and may be set according to historical data or set in response to input data of a user.

Specifically, the data amount of the data table to be stored is calculated according to the data amount calculation method of the data table to be stored. And comparing the data quantity of the data table to be stored with a preset first threshold value.

S240, determining that the storage mode of the data table to be stored is line storage;

specifically, if the data amount of the data table to be stored is smaller than a preset first threshold, it is determined that the storage mode of the data table to be stored is row storage.

And S250, judging whether the data volume of the data table to be stored is larger than or equal to a preset second threshold, if so, executing S260, and otherwise, executing S270.

In this embodiment, the first threshold is the same as the second threshold, and the determination of the data size is performed. It should be noted that the first threshold and the second threshold may be different. When the first threshold and the second threshold are different, when the data amount of the to-be-stored data table is greater than or equal to the first threshold and smaller than the second threshold, whether line-type storage is used or the storage mode of the to-be-stored data table is determined flexibly according to the processing performance, the memory size and the like of the current system.

The preset second threshold is a value which is different from the preset first threshold and is consistent with the data type of the data size of the data table to be stored, and can be set according to a historical data distribution rule or set in response to input data of a user. The preset second threshold may be equal to or different from the preset first threshold.

Specifically, the magnitude relation between the data amount of the data table to be stored and a preset second threshold value is compared. S260, determining a storage mode of the data table to be stored according to the service type of the data table to be stored;

as an alternative but non-limiting implementation manner, the storage manner of the data table to be stored is determined according to the service type of the data table to be stored, which may include, but is not limited to, the following five cases A1-A5:

a1, if the service type of the data table to be stored is determined to be online, determining that the storage mode of the data table to be stored is line storage;

illustratively, when a user needs to quickly obtain a feedback result to perform real-time interactive operation and the online service type can meet the requirement of the user, it is determined that the service type of the data table to be stored belongs to online. Since inline is suitable for reading one or several rows, it is suitable for line-type storage.

And A2, if the service type of the data table to be stored is determined to be batch, determining that the storage mode of the data table to be stored is columnar storage.

Illustratively, when a user needs to process a large batch of work tasks to perform online analysis processing operation, and the batch service type can meet the requirement of the user, the service type of the data table to be stored is determined to be batch. A batch job includes one column of data for a plurality of rows, while a batch is suitable for processing one column of data for a plurality of rows and is suitable for columnar storage.

Step A3, if the service types of the data table to be stored are determined to be online and batch, and the difference value between the online operation number and the batch operation number of the data table to be stored is greater than or equal to a preset third threshold value, determining that the storage mode of the data table to be stored is line storage;

the preset third threshold may be preset data consistent with the difference type between the online operation number and the batch operation number of the data table to be stored, may be set according to historical data distribution, and may also be set in response to a user.

Specifically, the number of online operations of the data table to be stored and the number of batch operations of the data table to be stored may be determined, and when a difference between the number of online operations of the data table to be stored and the number of batch operations of the data table to be stored is greater than or equal to a preset third threshold, that is, when a user performs one-row or multi-row reading operation and is greater than or equal to a user performs multi-row column data reading operation, it is determined that the storage mode of the data table to be stored is line storage.

Step A4, if the service types of the data table to be stored are determined to be online and batch, and the difference value between the batch operation number and the online operation number of the data table to be stored is greater than or equal to a preset fourth threshold value, determining that the storage mode of the data table to be stored is columnar storage; wherein the third threshold is the same as or different from the fourth threshold.

The preset fourth threshold may be preset data consistent with the difference type between the online operation number and the batch operation number of the data table to be stored, may be set according to historical data distribution, or may be set in response to a user, where the preset fourth threshold may be equal to or different from the preset third threshold.

Specifically, when the difference between the batch operation number and the online operation number of the to-be-stored data table is greater than or equal to a preset fourth threshold, that is, when a user performs a column data reading operation on a plurality of rows or is greater than or equal to a reading operation on one or more rows, it is determined that the storage mode of the to-be-stored data table is line storage.

And step A5, if the service types of the data table to be stored are determined to be online and batch, and the absolute value of the difference value between the online operation number and the batch operation number of the data table to be stored is smaller than or equal to a fifth threshold, determining that the storage mode of the data table to be stored is row-type and column-type mixed storage.

Wherein the fifth threshold may be data in accordance with an absolute value type of a difference between the number of online operations and the number of batch operations of the data table to be stored.

For example, when the absolute value of the difference between the number of online operations of the data table to be stored and the number of batch operations of the data table to be stored is less than or equal to the fifth threshold, it indicates that the number of online operations of the data table to be stored is similar to the number of batch operations of the data table to be stored, that is, when the difference between the number of reading operations performed by a user on one or more lines and the number of reading operations performed by a user on multiple lines is small, it is determined that the storage manner of the data table to be stored is row-type column-type hybrid storage.

And S270, ending.

According to the technical scheme of the embodiment of the invention, the data volume of the data table to be stored is determined by a data volume calculation method, and the service type is matched for the data table to be stored according to the actual operation requirement of a user. And determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored. According to the embodiment of the application, the storage mode matching operation of the data table to be stored is detailed according to the data volume and the service type of the data table to be stored, the reasonable storage mode is matched for the data table to be stored, the data redundancy when a plurality of rows of data are imported into a column of data is reduced, the unnecessary pressure of exporting and importing is reduced, and the resource occupancy rate of equipment is reduced.

EXAMPLE III

Fig. 3 is a schematic structural diagram of a data storage device according to a third embodiment of the present invention. As shown in fig. 3, the apparatus includes:

a data table obtaining module 310, configured to obtain a data table to be stored;

the attribute determining module 320 is configured to determine a data amount of the data table to be stored and a service type matched with the data table to be stored;

wherein the service type comprises online and/or batch;

the storage mode determining module 330 is configured to determine a storage mode of the to-be-stored data table according to the data volume and the service type of the to-be-stored data table;

wherein the storage mode comprises row-type storage, column-type storage or row-column-type hybrid storage.

In this embodiment, the attribute determining module 320 includes:

the data table attribute determining unit is used for determining the number of fields of the data table to be stored, the field data length corresponding to each field and the number of data;

and the data quantity determining unit is used for determining the data quantity of the data table to be stored according to the field quantity, the field data length corresponding to each field and the data number.

In this embodiment, the apparatus further includes:

and the first determining module is used for determining that the storage mode of the data table to be stored is a line type storage mode if the data quantity of the data table to be stored is determined to be smaller than a preset first threshold value.

In this embodiment of the application, the storage mode determining module 330 includes:

the second determining unit is used for determining the storage mode of the data table to be stored according to the service type of the data table to be stored if the data amount of the data table to be stored is determined to be larger than or equal to a preset second threshold;

wherein the first threshold is the same as or different from the second threshold.

In an embodiment of the present application, the second determining unit is specifically configured to:

if the service type of the data table to be stored is determined to be online, determining that the storage mode of the data table to be stored is line storage;

and if the service type of the data table to be stored is determined to be batch, determining that the storage mode of the data table to be stored is columnar storage.

If the service types of the data table to be stored are determined to be online and batch, and the difference value between the online operation number and the batch operation number of the data table to be stored is greater than or equal to a preset third threshold value, determining that the storage mode of the data table to be stored is line storage;

if the service types of the data table to be stored are determined to be online and batch, and the difference value between the batch operation number and the online operation number of the data table to be stored is greater than or equal to a preset fourth threshold value, determining that the storage mode of the data table to be stored is columnar storage;

wherein the third threshold is the same as or different from the fourth threshold.

And if the service types of the data table to be stored are determined to be online and batch, and the absolute value of the difference value between the online operation number and the batch operation number of the data table to be stored is smaller than or equal to a fifth threshold, determining that the storage mode of the data table to be stored is the row-type and column-type mixed storage.

The data storage device provided by the embodiment of the invention can execute the data storage method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

FIG. 4 illustrates a block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to the bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (cpu), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a data storage method.

In some embodiments, the data storage method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data storage method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data storage method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the Internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of storing data, comprising:

acquiring a data table to be stored;

wherein the service types comprise online and/or batch;

determining a storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored;

2. The method of claim 1, wherein determining the amount of data in the table of data to be stored comprises:

determining the number of fields of the data table to be stored, the field data length corresponding to each field and the number of data;

and determining the data quantity of the data table to be stored according to the field quantity, the field data length corresponding to each field and the data number.

3. The method of claim 1, after determining the amount of data of the data table to be stored, further comprising:

and if the data quantity of the data table to be stored is smaller than a preset first threshold value, determining that the storage mode of the data table to be stored is line storage.

4. The method of claim 3, wherein determining the storage mode of the data table to be stored according to the data volume and the service type of the data table to be stored comprises:

if the data volume of the data table to be stored is determined to be larger than or equal to a preset second threshold, determining a storage mode of the data table to be stored according to the service type of the data table to be stored;

5. The method according to claim 4, wherein determining the storage mode of the data table to be stored according to the service type of the data table to be stored comprises:

and if the service type of the data table to be stored is determined to be batch, determining that the storage mode of the data table to be stored is column storage.

6. The method according to claim 4, wherein determining the storage mode of the data table to be stored according to the service type of the data table to be stored comprises:

7. The method of claim 6, wherein determining the storage mode of the data table to be stored according to the service type of the data table to be stored comprises:

8. A data storage device, comprising:

wherein the service type comprises online and/or batch;

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements a data storage method according to any one of claims 1-7 when executing the program.

10. A storage medium storing computer-executable instructions for performing a data storage method as claimed in any one of claims 1 to 7 when executed by a computer processor.