CN111737404A - Data processing method, device, equipment and computer storage medium - Google Patents
Data processing method, device, equipment and computer storage medium Download PDFInfo
- Publication number
- CN111737404A CN111737404A CN202010593115.1A CN202010593115A CN111737404A CN 111737404 A CN111737404 A CN 111737404A CN 202010593115 A CN202010593115 A CN 202010593115A CN 111737404 A CN111737404 A CN 111737404A
- Authority
- CN
- China
- Prior art keywords
- field information
- type
- data
- target
- digital
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 43
- 230000008569 process Effects 0.000 claims abstract description 22
- 239000012634 fragment Substances 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 14
- 230000015654 memory Effects 0.000 claims description 29
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000013467 fragmentation Methods 0.000 claims description 3
- 238000006062 fragmentation reaction Methods 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims 1
- 238000013500 data storage Methods 0.000 abstract description 3
- 238000007667 floating Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 9
- 230000009466 transformation Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001808 coupling effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/151—Transformation
Abstract
The application discloses a data processing method, a data processing device, data processing equipment and a computer storage medium. The data processing method comprises the following steps: acquiring at least one field information in data generated in the system operation process; determining a target number type of each field information according to the content type of each field information in at least one field information; and storing each field information according to the target number type of each field information. According to the method and the device, before data storage, the digital types of the field information stored on different fragments are ensured to be consistent, and program exception caused by inconsistent digital types is avoided.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and computer storage medium.
Background
With the development of internet technology, almost every enterprise will adopt a certain network system for the service operation requirement, and store the record data and so on. The data generated during the operation of the network system is more and more, and even increases at an exponential level, reaching the degree of mass data. Part of the data generated by the system in the operation process is carried in the format of the strongly typed language. If the data in different memory fragments belong to the same content type, for example, the data belong to the same commodity price, the data is loaded in the strongly typed language format, and if the data in the same content type stored in different memory fragments are inconsistent in data type, the system operation may be abnormal.
Disclosure of Invention
The present disclosure provides an encoding method, apparatus, device, and storage medium.
According to an aspect of the present disclosure, there is provided a data processing method including:
acquiring at least one field information in data generated in the system operation process;
determining a target number type of each field information according to the content type of each field information in at least one field information;
and storing each field information according to the target number type of each field information.
According to another aspect of the present disclosure, there is provided a data processing apparatus including:
the field information acquisition module is used for acquiring at least one field information in data generated in the system operation process;
the target number type determining module is used for determining the target number type of each field information according to the content type of each field information in at least one field information;
and the storage module is used for storing each field information according to the target digital type of each field information.
According to another aspect of the present disclosure, there is provided an electronic device including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to cause the at least one processor to perform a method provided by any one of the embodiments of the present application.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method provided by any one of the embodiments of the present application.
According to the technology of the application, the data types of the field information can be unified before the field information in the data is stored, so that the field information of the same content type obtained at different time periods can be stored in the same data type, and program abnormity of a system caused by inconsistency of the stored digital types is avoided.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a schematic diagram of a data processing method according to an embodiment of the application;
FIG. 2 is a schematic diagram of a data processing method according to an embodiment of the application;
FIG. 3 is a schematic diagram of a data processing method according to another embodiment of the present application;
FIG. 4 is a schematic diagram of a data processing method according to a specific example of the present application;
FIG. 5 is a schematic diagram of a data processing apparatus according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a data processing apparatus according to another embodiment of the present application;
FIG. 7 is a schematic diagram of a data processing apparatus according to yet another embodiment of the present application;
fig. 8 is a block diagram of an electronic device for implementing the data processing method according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
Fig. 1 illustrates a data processing method according to an embodiment of the present application, and as shown in fig. 1, the data processing method includes:
step 101: acquiring at least one field information in data generated in the system operation process;
step 102: determining a target number type of each field information according to the content type of each field information in at least one field information;
step 103: and storing each field information according to the target number type of each field information.
In the embodiment of the present application, the data generated in the system operation process may be service data in the system operation process. One or more fields of information are included in the traffic data. For example, the advertisement system generates a large amount of data during operation, and the data may include field information of advertisement price, field information of stay time of network user on advertisement page, etc.
In this embodiment, the field information included in the data generated in the system operation process may be field information of the same content type, for example, the data generated in the system operation process includes field information of a plurality of advertisement prices. The data generated during the system operation process may include field information, or may also include field information of different types of content types, for example, the data generated during the system operation process includes field information of advertisement price, field information of advertisement ID (Identity), field information of advertisement name, and the like.
In this embodiment, the content type of each field information in the at least one field information may be, for example, the content type to which the field information belongs, such as advertisement price, advertisement ID, advertisement name, network user's stay time on the advertisement page, advertisement material address, and the like.
The target digital type is determined according to the content type, and the target digital type corresponding to the specific content type may be determined according to a preset correspondence between the content type and the digital type. For example, the price of an advertisement may be an integer, for example, the price of an advertisement is 100 dollars. Or may be a single precision floating point decimal (float), for example, with an advertisement price of 100.1 dollars. It may also be a double precision floating point decimal (double), for example, with an advertisement price of 100.12 dollars.
As an example, according to the content type being "advertisement price", the target number type may be determined to be a double-precision floating point decimal number according to a preset correspondence.
In this embodiment, the storage content type may be one of a long integer, a short integer, a byte type, an integer, a single-precision floating point decimal, a double-precision floating point decimal, and the like, according to the target number type.
In the embodiment of the application, before the field information is stored, the corresponding target digital type is determined according to the content type of the field information, and then the field information is stored according to the target digital type, so that the digital types stored in the field information can be consistent, and the situation that the digital types of the field information of the same content type stored in different fragments are inconsistent is avoided.
In one embodiment, determining the target number type of each of the field information comprises:
determining a target digital type of each field information based on a preset content type and a corresponding relation between digital types;
the digital type corresponding to the content type is the digital type with the highest precision in at least one digital type corresponding to the content type.
In this embodiment, a special storage space may be preset to configure the corresponding relationship between various content types that may appear in the data generated by the system operation and various possible digital types. For example, there are 100 content types in data generated by a certain system, and the 100 content types are respectively associated with a digital type in advance, and the association is stored in a corresponding storage space. When data generated by new system operation is obtained each time, field information in the data is obtained, a content type corresponding to the field information is determined, and then a target digital type is searched according to the content type.
In this embodiment, the number types that may occur in the field information of some content types are specific, for example, for the field information of which the content type is an advertisement name, the number types are generally byte types, and no data types, such as short integer, long integer, single-precision floating point decimal, double-precision floating point decimal, etc., may occur. For another example, for field information whose content type is date, the number type is generally long integer or short integer, and cannot be floating point decimal type. In this embodiment, the digital type with the highest precision of the digital type is selected from the digital types possibly corresponding to the content types, and the target digital type corresponding to the content type can ensure that the data precision is not lost in the data storage process.
For example, for field information whose content type is advertisement price, it may be an integer data type, for example, advertisement price may be 1-element. Data types that are floating point decimal are also possible, for example, the advertisement price may be 1.11 dollars. The advertisement price data obtained by the advertisement system at different time periods may appear in integer and floating point decimal types. In this case, the field information of which the content type is the advertisement price is set to the target number type as the floating point decimal, so that the precision of the advertisement price field information which is originally of the floating point decimal type in the original data can be ensured not to be lost. Specifically, for advertisement prices, if integer data, such as 1-tuple, is present, the integer data can be converted to a floating point decimal, such as 1.00-tuple.
As an example of the present embodiment, the preset correspondence between content types and numeric types may only include correspondence between content types and set numeric types, where multiple numeric types may exist. For example, for the field information whose content type is the advertisement name, the number type is generally only possible to be bytes, and therefore, the target number type corresponding to the field information of the advertisement name does not need to be set.
In one embodiment, as shown in fig. 2, storing the field information according to the target number type includes:
step 201: and if the number type of the field information is consistent with the target number type, directly storing the field information.
In this embodiment, the number type of the field information in the data may be exactly the same as the target number type, and in this case, the field information may be directly stored without changing the number type of the field information.
In one embodiment, as shown in fig. 3, storing the field information according to the target number type includes:
step 301: and if the number type of the field information is not consistent with the target number type, converting the number type of the field information into the target number type, and storing the field information after the conversion into the target number type.
For example, the target number type corresponding to the field information for setting the content type as the page dwell time is a floating point decimal. When the field information of the page stay time extracted from the data is just the floating point decimal number, directly storing the field information of the page stay time with the number type of the floating point decimal number. When the field information of the page dwell time extracted from the data is integer, converting the integer field information into floating point decimal numbers, specifically, converting by increasing decimal numbers. And then storing the field information of the converted floating point decimal type page dwell time.
In this embodiment, the number type of the field information in the data is not identical to the target number type, and in this case, the field information is changed so that the number type of the field information is converted into the target number type, and then stored. Therefore, the field information of the same content type stored in different partitions is consistent in digital type, and errors caused by inconsistency of the digital type in the subsequent system operation process are avoided.
In one embodiment, storing the field information according to the target digit type further comprises:
determining the generation time of data corresponding to the field information;
determining a corresponding memory fragment according to the generation time of the data;
and storing the field information into the corresponding memory fragment.
In this embodiment, the data is generally stored in sequence according to the generated time, for example, the data of 0 th to 10 th minutes is stored in the first memory slice, the data of 11 th to 20 th minutes is stored in the second memory slice, the data of 21 st to 30 th minutes is stored in the third memory slice, and so on. The data stored by each memory slice may contain a plurality of field information, which may be field information of different or same content types. Since the digital types of the field information in the data are unified when the data are stored, the digital types of the field information of the same content type stored in different storage slices are also consistent.
In one embodiment, acquiring field information in data generated during operation of the system includes:
generating an operation log according to the data generated in the system operation process;
reporting the running log to an asynchronous storage unit;
and acquiring the data from the asynchronous storage unit, and extracting at least one field information in the data.
In this embodiment, the asynchronous storage unit may form a buffer pool, which can implement isolation between data storage and data retrieval, and reduce data coupling effect.
In an example, the data processing method provided by the embodiment of the application can be applied to an advertisement system. When the advertisement system runs, a large amount of service data can be generated, the data system collects the service data and loads the service data in a JSON (JavaScript object notation) format, invalid data is cleaned and removed through a Spark program, and then the data is sliced and stored in a data warehouse in a queue format. One problem with using JSON to carry data is that all numbers in JSON are numeric types, e.g., 11.0 is a numeric type, and the numeric types representing numbers in the Java system are Byte, Short, Int, Long, Float, Double. Meanwhile, Java is a strongly typed language, for example, field information JSON A { "date": 2020-03-24"," value ":1.12}, and field information JSON B {" date ": 2020-03-24", "value":1} are normal business data, and the content types are prices. When the data of one fragment is JSON A, the Spark program infers that the JSON data is of Double data type and stores the JSON data in a data warehouse, and when the data of the other fragment is JSON B, the Spark program infers that the JSON data is of Long data type and stores the JSON data in the data warehouse, and when the data in the data warehouse is read, if the data types of the two fragments are inconsistent, program exception can be caused. By the data processing method provided by the embodiment of the application, the field information of the same content type can be converted into a uniform digital type for storage, so that the consistency of data types stored in different fragments is ensured.
Another embodiment of the present application further provides a data processing apparatus, which has main components as shown in fig. 4, and includes:
a field information obtaining module 401, configured to obtain at least one field information in data generated in a system operation process;
a target number type determining module 402, configured to determine a target number type of each field information according to a content type of each field information in at least one field information;
a storage module 403, configured to store each piece of field information according to the target number type of each piece of field information.
In one embodiment, the data processing apparatus further includes a field information obtaining module 401, a target number type determining module 402, and a storage module 403 shown in fig. 4, and on the basis of this, the target number type determining module is further configured to:
determining a target digital type of each field information based on a preset content type and a corresponding relation between digital types;
the digital type corresponding to the content type is the digital type with the highest precision in at least one digital type corresponding to the content type.
In one embodiment, the data processing apparatus further includes a field information obtaining module 401, a target number type determining module 402, and a storage module 403 shown in fig. 4, and on the basis of this, as shown in fig. 5, the storage module 403 further includes:
a first storage unit 501, configured to directly store the field information if the number type of the field information is consistent with a target number type;
a second storage unit 502, configured to convert the number type of the field information into a target number type if the number type of the field information is not consistent with the target number type, and store the field information after being converted into the target number type.
In one embodiment, the data processing apparatus further includes a field information obtaining module 401, a target number type determining module 402, and a storage module 403 shown in fig. 4, and on the basis of this, as shown in fig. 6, the storage module 403 further includes:
a time unit 601, configured to determine a generation time of data corresponding to the field information;
the fragmentation unit 602 is configured to determine a corresponding storage fragment according to the generation time of the data;
a segment storage unit 603, configured to store the field information in the corresponding storage segment.
In one embodiment, the data processing apparatus further includes a field information obtaining module 401, a target number type determining module 402, and a storage module 403 shown in fig. 4, and on the basis of this, as shown in fig. 7, the field information obtaining module 401 further includes:
a log unit 701, configured to generate an operation log according to the data generated in the system operation process;
a log storage unit 702, configured to report the operation log to an asynchronous storage unit;
an extracting unit 703 is configured to obtain the data from the asynchronous storage unit, and extract at least one field information in the data.
Fig. 8 is a block diagram of an electronic device according to an encoding method of an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 8, the electronic apparatus includes: one or more processors 801, memory 802, and interfaces for connecting the various components, including a high speed interface and a low speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Fig. 8 illustrates an example of a processor 801.
The memory 802 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the encoding method provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the encoding method provided herein.
The memory 802, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the encoding method in the embodiment of the present application (for example, the field information acquisition module 401, the target digit type determination module 402, and the storage module 403 shown in fig. 4). The processor 801 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 802, that is, implements the encoding method in the above-described method embodiments.
The memory 802 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the video encoding electronic device, and the like. Further, the memory 802 may include high speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 802 optionally includes memory located remotely from the processor 801, which may be connected to video encoding electronics over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the encoding method may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803, and the output device 804 may be connected by a bus or other means, and are exemplified by a bus in fig. 8.
The input device 803 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the video encoded electronic device, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick, or other input device. The output devices 804 may include a display device, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, whether the transformation unit is divided or not can be judged according to at least one live intermediate loss value in the transformation unit with the second depth and then according to the intermediate loss value and the loss value of the target unit with the first depth, so that whether the next-depth TU division is needed or not can be determined in advance before calculating the loss values of all the transformation units with the second depth or calculating the intermediate loss values according to the loss values of all the transformation units with the second depth in the process of video coding for TU-level transformation of a plurality of coding units, redundant calculation is reduced, the calculation efficiency is improved, and meanwhile the quality of food coding can be guaranteed.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (12)
1. A data processing method, comprising:
acquiring at least one field information in data generated in the system operation process;
determining a target number type of each field information according to the content type of each field information in at least one field information;
and storing each field information according to the target number type of each field information.
2. The method of claim 1, wherein determining a target digit type for each of the field information comprises:
determining a target digital type of each field information based on a preset content type and a corresponding relation between digital types;
the digital type corresponding to the content type is the digital type with the highest precision in at least one digital type corresponding to the content type.
3. The method of claim 1, wherein storing the field information according to the target digit type comprises:
if the digital type of the field information is consistent with the target digital type, directly storing the field information;
and if the number type of the field information is not consistent with the target number type, converting the number type of the field information into the target number type, and storing the field information after the conversion into the target number type.
4. The method of claim 1, wherein the field information is stored according to the target digit type, further comprising:
determining the generation time of data corresponding to the field information;
determining a corresponding memory fragment according to the generation time of the data;
and storing the field information into the corresponding memory fragment.
5. The method of claim 1, wherein obtaining field information in data generated during system operation comprises:
generating an operation log according to the data generated in the system operation process;
reporting the running log to an asynchronous storage unit;
and acquiring the data from the asynchronous storage unit, and extracting at least one field information in the data.
6. A data processing apparatus, comprising:
the field information acquisition module is used for acquiring at least one field information in data generated in the system operation process;
the target number type determining module is used for determining the target number type of each field information according to the content type of each field information in at least one field information;
and the storage module is used for storing each field information according to the target digital type of each field information.
7. The apparatus of claim 6, wherein the target digital type determination module is further configured to:
determining a target digital type of each field information based on a preset content type and a corresponding relation between digital types;
the digital type corresponding to the content type is the digital type with the highest precision in at least one digital type corresponding to the content type.
8. The apparatus of claim 6, wherein the storage module further comprises:
the first storage unit is used for directly storing the field information if the digital type of the field information is consistent with the target digital type;
and the second storage unit is used for converting the digital type of the field information into the target digital type and storing the field information after the conversion into the target digital type if the digital type of the field information is inconsistent with the target digital type.
9. The apparatus of claim 6, wherein the storage module further comprises:
the time unit is used for determining the generation time of the data corresponding to the field information;
the fragmentation unit is used for determining a corresponding storage fragmentation according to the generation time of the data;
and the fragment storage unit is used for storing the field information into the corresponding storage fragment.
10. The apparatus of claim 6, wherein the field information obtaining module further comprises:
the log unit is used for generating an operation log according to the data generated in the system operation process;
the log storage unit is used for reporting the running log to the asynchronous storage unit;
and the extraction unit is used for acquiring the data from the asynchronous storage unit and extracting at least one field information in the data.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010593115.1A CN111737404A (en) | 2020-06-26 | 2020-06-26 | Data processing method, device, equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010593115.1A CN111737404A (en) | 2020-06-26 | 2020-06-26 | Data processing method, device, equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111737404A true CN111737404A (en) | 2020-10-02 |
Family
ID=72651188
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010593115.1A Pending CN111737404A (en) | 2020-06-26 | 2020-06-26 | Data processing method, device, equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111737404A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114528444A (en) * | 2022-02-25 | 2022-05-24 | 北京百度网讯科技有限公司 | Graph data processing method and device, electronic equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103677654A (en) * | 2012-09-24 | 2014-03-26 | 联想(北京)有限公司 | Method and electronic equipment for storing data |
| CN106156227A (en) * | 2015-04-26 | 2016-11-23 | 阿里巴巴集团控股有限公司 | A kind of data transmission method and device |
| CN107608714A (en) * | 2017-09-30 | 2018-01-19 | 广州酷狗计算机科技有限公司 | Byte-aligned method, apparatus and computer-readable recording medium |
| CN107943542A (en) * | 2017-11-28 | 2018-04-20 | 山东浪潮通软信息科技有限公司 | A kind of configuration information management method, device, computer-readable recording medium and storage control |
| CN108062272A (en) * | 2017-11-06 | 2018-05-22 | 福建省天奕网络科技有限公司 | A kind of quick-fried table detection method of database and terminal |
| CN108874387A (en) * | 2018-06-27 | 2018-11-23 | 深圳市金蝶天燕中间件股份有限公司 | A kind of Data Serialization method and its relevant device |
| CN108959430A (en) * | 2018-06-11 | 2018-12-07 | 北京金山安全软件有限公司 | Advertisement promotion data acquisition method, device and equipment |
| CN109241167A (en) * | 2018-09-18 | 2019-01-18 | 四川爱联科技有限公司 | List data introduction method based on B/S framework |
| CN109558404A (en) * | 2018-10-19 | 2019-04-02 | 中国平安人寿保险股份有限公司 | Date storage method, device, computer equipment and storage medium |
| CN110888862A (en) * | 2019-11-26 | 2020-03-17 | 深圳市随手金服信息科技有限公司 | Data storage method, data query method, data storage device, data query device, server and storage medium |
-
2020
- 2020-06-26 CN CN202010593115.1A patent/CN111737404A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103677654A (en) * | 2012-09-24 | 2014-03-26 | 联想(北京)有限公司 | Method and electronic equipment for storing data |
| CN106156227A (en) * | 2015-04-26 | 2016-11-23 | 阿里巴巴集团控股有限公司 | A kind of data transmission method and device |
| CN107608714A (en) * | 2017-09-30 | 2018-01-19 | 广州酷狗计算机科技有限公司 | Byte-aligned method, apparatus and computer-readable recording medium |
| CN108062272A (en) * | 2017-11-06 | 2018-05-22 | 福建省天奕网络科技有限公司 | A kind of quick-fried table detection method of database and terminal |
| CN107943542A (en) * | 2017-11-28 | 2018-04-20 | 山东浪潮通软信息科技有限公司 | A kind of configuration information management method, device, computer-readable recording medium and storage control |
| CN108959430A (en) * | 2018-06-11 | 2018-12-07 | 北京金山安全软件有限公司 | Advertisement promotion data acquisition method, device and equipment |
| CN108874387A (en) * | 2018-06-27 | 2018-11-23 | 深圳市金蝶天燕中间件股份有限公司 | A kind of Data Serialization method and its relevant device |
| CN109241167A (en) * | 2018-09-18 | 2019-01-18 | 四川爱联科技有限公司 | List data introduction method based on B/S framework |
| CN109558404A (en) * | 2018-10-19 | 2019-04-02 | 中国平安人寿保险股份有限公司 | Date storage method, device, computer equipment and storage medium |
| CN110888862A (en) * | 2019-11-26 | 2020-03-17 | 深圳市随手金服信息科技有限公司 | Data storage method, data query method, data storage device, data query device, server and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114528444A (en) * | 2022-02-25 | 2022-05-24 | 北京百度网讯科技有限公司 | Graph data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111523001B (en) | Method, device, equipment and storage medium for storing data | |
| CN112015775A (en) | Label data processing method, device, equipment and storage medium | |
| CN110888926B (en) | Method and device for structuring medical text | |
| CN111737965A (en) | Document comparison method and device, electronic equipment and readable storage medium | |
| CN111259107A (en) | Storage method and device of determinant text and electronic equipment | |
| CN111241396B (en) | Information pushing method and device, electronic equipment and storage medium | |
| CN111966925A (en) | Building interest point weight judging method and device, electronic equipment and storage medium | |
| CN111737404A (en) | Data processing method, device, equipment and computer storage medium | |
| CN108804451B (en) | Method and device for generating primary key, electronic equipment and readable storage medium | |
| CN111737534A (en) | File processing method, device and equipment | |
| US10318650B2 (en) | Identifying corrupted text segments | |
| CN111177479A (en) | Method and device for acquiring feature vectors of nodes in relational network graph | |
| CN111596897B (en) | Code multiplexing processing method and device and electronic equipment | |
| CN112598139B (en) | Category encoding method, category encoding device, category encoding apparatus, category encoding device, category encoding storage medium, and category encoding program product | |
| CN112115113B (en) | Data storage system, method, device, equipment and storage medium | |
| CN112560460B (en) | Method and device for extracting structured information, electronic equipment and readable storage medium | |
| CN111694686A (en) | Abnormal service processing method and device, electronic equipment and storage medium | |
| CN112052347A (en) | Image storage method and device and electronic equipment | |
| CN112559497A (en) | Data processing method, information transmission method and device and electronic equipment | |
| CN111400431A (en) | Event argument extraction method and device and electronic equipment | |
| CN111966576A (en) | Abnormal code positioning method and device, electronic equipment and storage medium | |
| CN112269789B (en) | Method and device for storing data, and method and device for reading data | |
| CN111815500B (en) | Image processing method, device, equipment and storage medium | |
| CN113127512B (en) | Multi-data stream data splicing triggering method and device, electronic equipment and medium | |
| CN111339241B (en) | Problem duplicate checking method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |