CN113392252B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN113392252B CN113392252B CN202110609634.7A CN202110609634A CN113392252B CN 113392252 B CN113392252 B CN 113392252B CN 202110609634 A CN202110609634 A CN 202110609634A CN 113392252 B CN113392252 B CN 113392252B
- Authority
- CN
- China
- Prior art keywords
- task
- processed
- query
- data object
- editing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5846—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2308—Concurrency control
- G06F16/2315—Optimistic concurrency control
- G06F16/2322—Optimistic concurrency control using timestamps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Library & Information Science (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data processing method and a data processing device, which are used for sequentially determining tasks to be processed, sequentially determining task types corresponding to the tasks to be processed according to a receiving sequence and allocating corresponding timestamps to the tasks to be processed. And sequentially processing each task to be processed according to the corresponding timestamp and the task type. When the task type of the task to be processed is the query task, each task to be processed is firstly split to obtain a plurality of subtasks, and then each subtask is respectively processed according to the corresponding timestamp. The embodiment of the invention processes the tasks based on the time stamps and the task types, and ensures the orderliness and the data consistency when the tasks are processed frequently. Meanwhile, the complex query scene is dealt with by splitting the query task into a plurality of subtasks for processing.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
The existing task processing systems are all currently used for querying or editing a specific scene or a single type of data. For example, a task processing system implemented based on a conventional relational database is a full-text task processing technology implemented based on task processing engines such as Lucene or elastic search, for reading and writing and retrieving structured data. The task processing scenes which can be covered by the task processing systems are relatively single, and some complex task processing requirements cannot be met.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method and apparatus, which aim to deal with task processing in a complex scene, and simultaneously ensure ordering and data consistency when processing frequent tasks.
In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
determining tasks to be processed in sequence;
respectively determining the task type corresponding to each task to be processed;
distributing corresponding time stamps to the tasks to be processed in sequence according to the determined sequence;
sequentially processing each task to be processed according to the corresponding timestamp and the task type;
wherein the sequentially processing each task to be processed according to the corresponding timestamp and task type comprises:
in response to the fact that the task type corresponding to the task to be processed is a query task, splitting the task to be processed to obtain a plurality of subtasks;
and respectively processing each subtask according to the corresponding timestamp, wherein the processing mode of each subtask is multi-thread parallel processing.
Further, the respectively determining the task types corresponding to the to-be-processed tasks includes:
determining a database language corresponding to each task to be processed;
responding to the database language to data control statements or data definition statements, and determining the task to be processed as an editing task;
and responding to the database language as a data query statement, and determining the task to be processed as a query task.
Further, the sequentially processing the tasks to be processed according to the corresponding timestamps and task types further includes:
responding to the fact that the corresponding task type is an editing task, and recording the task to be processed in a task log according to the corresponding time stamp sequence;
and sequentially processing each editing task in the task log.
Further, the sequentially processing each editing task in the task log includes:
determining a to-be-processed editing task recorded first in a task log as a target editing task;
determining at least one data object corresponding to the target editing task;
distributing corresponding threads for each data object;
and executing the target editing task in parallel through each thread so as to edit the corresponding data objects respectively.
Further, each of the data objects has a corresponding time state;
the sequentially processing each editing task in the task log further comprises:
and in response to the completion of the editing of the corresponding data object, updating the time state of the data object according to the timestamp of the target editing task.
Further, the method further comprises:
inquiring a timestamp corresponding to the editing task recorded last time in the task log at a preset frequency;
responding to the fact that the time length of the timestamp from the current time is larger than a time length threshold value, and generating a blank editing task, wherein the blank editing task does not change any data object;
and distributing a corresponding time stamp for the blank editing task and recording the time stamp to the task log.
Further, the respectively processing the subtasks according to the corresponding timestamps includes:
determining a data object corresponding to each subtask;
respectively processing each subtask according to the timestamp corresponding to the query task to query a data object to obtain a corresponding query result;
and determining a target query result according to each query result.
Further, the respectively processing the subtasks according to the timestamp corresponding to the query task to query the data object to obtain the corresponding query result includes:
determining the time state of the data object corresponding to each subtask;
responding to the time state that the time stamp of the query task corresponding to the subtask is not larger than the time state of the corresponding data object, and querying the corresponding data object to obtain a corresponding query result;
and responding to the time state that the timestamp of the query task corresponding to the subtask is larger than the time state of the corresponding data object, and returning error reporting information.
In a second aspect, the present invention provides a computer-readable storage medium for storing computer program instructions, which when executed by a processor implement the method according to any one of the first aspect.
In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory is used to store one or more computer program instructions, where the one or more computer program instructions are executed by the processor to implement the method according to any one of the first aspect.
The embodiment of the invention determines the tasks to be processed in sequence, determines the task type corresponding to each task to be processed in sequence according to the receiving sequence, and allocates the corresponding time stamp to each task to be processed. And sequentially processing each task to be processed according to the corresponding timestamp and the task type. When the task type of the task to be processed is the query task, the task to be processed is firstly split to obtain a plurality of subtasks, and then the subtasks are respectively processed according to the corresponding timestamps. The embodiment of the invention processes the tasks based on the time stamps and the task types, and ensures the orderliness and the data consistency when the tasks are processed frequently. Meanwhile, the complex query scene is dealt with by splitting the query task into a plurality of subtasks for processing.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an edit task execution process according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an editing task execution process according to an embodiment of the present invention;
FIG. 4 is a flow diagram of a query task execution process according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a query task execution process according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an electronic device according to an embodiment of the invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, flows, components and circuits have not been described in detail so as not to obscure the present invention.
Furthermore, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.
Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including but not limited to".
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.
The data processing method of the embodiment of the invention can be realized by the server and is used for editing or inquiring the data object stored in the memory in the server or the data object stored in the database connected with the server. The server may be a single server or a server cluster composed of a plurality of servers. The server comprises a processor for executing the data processing method of the embodiment of the invention. Alternatively, the processor included in the server may be a CPU (central Processing Unit) or a GPU (Graphics Processing Unit).
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention. As shown in fig. 1, the data processing method according to the embodiment of the present invention includes the following steps:
and step S100, determining tasks to be processed in sequence.
Specifically, the task to be processed may be generated by the server, or may be uploaded to the server by a terminal device communicatively connected to the server. And the server sequentially determines the tasks to be processed according to the sequence generated or received by the tasks to be processed. Optionally, the task to be processed includes task content for editing or querying a data object stored in a memory in the server, or a task connecting the data object stored in the database with the server. Editing the data object may include adding a new data object, deleting an existing data object or content in the data object, inserting new content into the existing data object, modifying content in the existing data object, and the like.
Optionally, a to-be-processed task queue may be maintained in the server, and when the to-be-processed tasks are generated or received, the to-be-processed tasks are added to the to-be-processed task queue according to the sequence, so as to sequentially process the to-be-processed tasks in the to-be-processed task queue according to the sequence.
And S200, respectively determining the task types corresponding to the tasks to be processed.
Specifically, the server respectively has corresponding task types in a plurality of tasks to be processed determined in sequence, and the task types are used for representing the processing of the data objects required for executing the tasks to be processed. Optionally, the task types corresponding to the tasks to be processed in the embodiment of the present invention include an editing task and a query task, where the editing task is used to perform processing such as adding, deleting, modifying, and updating on the data object, and the query task is used to query corresponding content in the data object.
In the embodiment of the invention, the server can determine the task type according to the language corresponding to the task to be processed. The task to be processed may be generated and transmitted in a format of a domain-specific Language (DSL) or a database Language (SQL). The language corresponding to the task to be processed in the embodiment of the invention is taken as the database language for explanation. In the database language, the statements applied by different task types are different, so that the task type can be determined by determining the database language corresponding to the task to be processed. And determining the task to be processed as an editing task when the corresponding database language is a data control statement or a data definition statement, and determining the task to be processed as a query task when the corresponding database language is a data query statement. The Data Definition statement is DDL (Data Definition Language) and is used to create various objects in the database, such as tables, views, indexes, synonyms, clusters, and the like. The Data Manipulation statement is DML (Data management Language) and is used for manipulating Data contained in a Data object in a database, for example, updating, deleting or adding the Data object or content in the Data object. The Data Query statement is DQL (Data Query Language) and is used for querying each object in the database.
Optionally, the to-be-processed task determined by the server includes task content, and is recorded through a corresponding database language. For example, when the database language corresponding to the task to be processed is a data manipulation statement, the task to be processed is an editing task, and the task content may include a data object identifier, an editing mode, and editing content. When the database language corresponding to the task to be processed is the data control statement, the task to be processed is an editing task, and the task content may include a data object identifier and an editing mode. When the database language corresponding to the task to be processed is the data query statement, the task to be processed is the query task, and the task content may include the data object identifier and the query condition.
And S300, sequentially distributing corresponding time stamps to the tasks to be processed according to the determined sequence.
Specifically, when the server determines each task to be processed, the server sequentially allocates a corresponding timestamp to each task to be processed according to the determined time. For example, when the to-be-processed task is a task uploaded by another device in communication connection with the server, the timestamp allocated to the to-be-processed task corresponds to the time when the to-be-processed task is received by the server. When the task to be processed is a task generated by the server, the timestamp allocated to the task to be processed corresponds to the time when the task to be processed is generated by the server.
And S400, sequentially processing each task to be processed according to the corresponding timestamp and the corresponding task type.
Specifically, the server sequentially processes the determined multiple tasks to be processed according to the corresponding timestamps and task types, wherein the processing sequence of each task to be processed is the same as the determined sequence. The server processes the tasks to be processed with different task types through different processing modes because the tasks to be processed correspond to different task types. For example, in order to improve processing efficiency in a task process and deal with task processing in a complex scene, a server splits an inquiry task to obtain a plurality of subtasks when a task to be processed currently needs to be the inquiry task, and then processes each subtask according to a corresponding timestamp.
Further, when the tasks to be processed are editing tasks, the server records the tasks to be processed in the task log according to the corresponding time stamp sequence, and the task processing sequence is guaranteed in a mode of sequentially processing all editing tasks in the task log.
Fig. 2 is a flowchart of an editing task execution process according to an embodiment of the present invention. As shown in fig. 2, when the task type corresponding to the task to be processed is an editing task, the process of processing the corresponding task to be processed by the server includes the following steps:
and S410, determining the editing task to be processed recorded in the task log as a target editing task.
Specifically, the server determines a target editing task according to the sequence of editing tasks to be processed stored in the task log so as to process the target editing task. Optionally, the server determines the editing task to be processed, which is recorded first in the task log, as the target editing task, so as to allocate at least one thread to process the target editing task. The target editing task is processed in parallel by each thread, the processing processes are not interfered mutually, and the target editing task is determined again after the current target editing task is processed and completed by one thread. That is to say, after the target editing task of the current thread is processed, the next editing task to be processed is obtained from the task log and used as the target editing task of the current thread, so that the task processing is performed again until all the tasks to be processed in the task log are processed.
The following description will be given by taking as an example that the tasks to be processed sequentially determined by the server are "edit task 1", "edit task 2", and "edit task 3". The server allocates timestamps to the editing tasks, wherein the timestamps are respectively 'timestamp 1', 'timestamp 2' and 'timestamp 3', and the timestamp 3 > the timestamp 2 > the timestamp 1. For example, "timestamp 1" may be 10 00, "timestamp 2" may be 10. After the task type of each task to be processed is determined, the task type is recorded in a task log from first to last in sequence according to the corresponding time stamp sequence.
Therefore, "edit task 1: timestamp 1"," edit task 2: timestamp 2 "and" edit task 3: timestamp 3". When processing the editing task, the server firstly acquires the editing task 1 recorded firstly from the task log according to the corresponding timestamp as a target editing task, and allocates the thread 1, the thread 2 and the thread 3 to process the target editing task. After the current target editing task of the thread 1 is processed, even if neither the thread 2 nor the thread 3 finishes the editing task 1 currently serving as the target editing task, the next recorded editing task 2 is determined from the task log to be processed as the target editing task of the thread 1, and the last recorded editing task 3 is determined to be processed as the target editing task after the processing is finished.
Optionally, each editing task in the task log may also include a task status. That is, each thread in the server determines the task state of each editing task in the task log when processing the editing task, determines the editing task recorded first as the target editing task from among editing tasks whose states are not processed, and updates the task state of the current target editing task to processed after the editing tasks are processed by all threads. Further, the next target editing task is determined again among the unprocessed editing tasks.
And step S420, determining at least one data object corresponding to the target editing task.
Specifically, when processing the target editing task, the server determines a plurality of data objects corresponding to the target editing task first, so as to edit each data object by processing the target editing task. Since the target editing task corresponds to a data manipulation sentence or a data definition sentence, it is used for performing different processes. Therefore, for different target editing tasks, the server determines the corresponding data objects in different ways.
In an optional implementation manner of the embodiment of the present invention, the database language corresponding to the target editing task is a data manipulation statement, that is, the task content corresponding to the database language includes an editing mode and an editing content for representing the data to be edited. In the embodiment of the invention, the server can correspond to a plurality of data objects for storing data in different formats. When the database language corresponding to the target editing task is a data manipulation sentence, the server may identify editing contents included in the contents, determine a format of each editing content when the editing contents are multiple, and determine a corresponding data object according to the format of each editing content.
The embodiment of the present invention is described by taking an example in which the embodiment of the present invention is applied to management of a plurality of cameras in a target area. And a data table corresponding to a plurality of cameras in the target area is stored in a memory in the server or a database in communication connection with the server and is used for storing the characteristics of the pictures acquired by the cameras. The method comprises a text characteristic data table for storing information text characteristics in the picture, a vector characteristic data table for storing information vector characteristics in the picture and a picture characteristic data table for storing picture acquisition information. Further, the text features may include, for example, color, size, and style descriptions of the content of the picture as a whole or in part. The vector characteristic is a picture characteristic vector obtained by converting a picture through a specific model or algorithm. The picture collecting information may include information of a camera device collecting the picture, time, position, number, and the like of collecting the image.
After a camera in the target area acquires a picture, extracting information in the picture to determine text features and vector features. And generating and uploading an editing task for storing picture information to a server, wherein the corresponding database language is a data manipulation language, and the editing task comprises data storage as an editing mode and corresponding acquisition information, text features and vector features as editing contents. Therefore, when the current task is a target editing task to be processed, the server respectively determines a picture feature data table for storing the acquired information, a text feature data table for storing the text feature and a vector feature data table for storing the vector feature as data objects according to formats of the editing content acquisition information, the text feature and the vector feature.
In another optional implementation manner of the embodiment of the present invention, the database language corresponding to the target editing task is a data definition statement, which is used to create various objects in the database, including data object identification and an editing manner. Therefore, the data object corresponding to the target editing task corresponding to the data definition statement is the data object represented by the data object identifier in the task content.
Still take the application of the embodiment of the present invention to the management of multiple cameras in a target area as an example for explanation. And a data table corresponding to the plurality of cameras in the target area is stored in a memory in the server or a database in communication connection with the server and is used for storing the picture characteristics acquired by each camera. The method comprises a text feature data table used for storing information text features in the picture and a vector feature data table used for storing information vector features in the picture. When the server receives or generates a picture characteristic data table which is used for storing picture acquisition information and needs to be created, a database language is determined as an editing task of a data definition statement, wherein the editing task comprises a data object identifier which is used for representing the picture characteristic data table to be created. Therefore, when the current task is a target editing task to be processed, the server directly determines that the data object corresponding to the data object identifier in the task of the target editing task is the picture characteristic data table.
Step S430, allocating corresponding threads to the data objects respectively.
Specifically, after the data objects corresponding to the target editing task are determined, a plurality of threads are determined according to the number of the data objects corresponding to the target editing task, so that corresponding threads are allocated to the data objects. And each thread independently edits the corresponding data object, so that the task processing efficiency is improved while the interference of other threads is avoided.
Step S440, the target editing task is executed in parallel by each thread, so as to edit the corresponding data objects respectively.
Specifically, after a plurality of data objects and a plurality of threads corresponding to the target editing task are determined, the target editing task is executed in parallel through the threads, so that the corresponding data objects are respectively edited according to the task content of the target editing task.
Furthermore, each data object corresponding to the server includes a corresponding time status, which is used to represent the time when the data object was edited last time. And after the corresponding thread finishes editing the data object, updating the time represented by the target editing task timestamp into the time state of the corresponding data object.
The embodiment of the present invention is described by taking an example in which the embodiment of the present invention is applied to management of a plurality of cameras in a target area. And a data table corresponding to a plurality of cameras in the target area is stored in a memory in the server or a database in communication connection with the server and is used for storing the characteristics of the pictures acquired by the cameras. The method comprises a text characteristic data table used for storing information text characteristics in the picture, a vector characteristic data table used for storing information vector characteristics in the picture and a picture characteristic data table used for storing picture acquisition information.
When the timestamp corresponding to the target editing task is 10. And determining that the data objects corresponding to the target edit are a picture characteristic data table, a text characteristic data table and a vector characteristic data table respectively. The server can process the target editing task through three threads respectively, namely, the collected information in the target editing task is stored in the picture characteristic data table through the thread 1, the thread 2 and the thread 3 respectively, the text characteristic in the target editing task is stored in the text characteristic data table, and the vector characteristic in the target editing task is stored in the vector characteristic data table. Further, the picture feature data table, the text feature data table, and the vector feature data table have corresponding time states, respectively, and when one thread process is completed, the corresponding timestamp 10.
Fig. 3 is a schematic diagram of an editing task execution process according to an embodiment of the present invention. As shown in fig. 3, when the server determines that the task to be processed is an editing task, the editing task is recorded in the task log 30. And then sequentially acquiring editing tasks to be processed from the task log as target editing tasks 31. Further, the server identifies a plurality of data objects 33 corresponding to the target editing task, and assigns a corresponding thread 32 to each data object 33. The server processes each thread 32 in parallel to edit the corresponding data object 33 for each processing target edit task, and updates the time status of the corresponding data object to the time stamp of the target edit task after the processing is completed.
And under the condition that the task to be processed is the query task, the server directly determines the current task to be processed as the query task, and obtains a plurality of corresponding subtasks by dividing the query task so as to respectively determine the data objects corresponding to the subtasks. And further, processing each subtask respectively according to the time stamp corresponding to the query task to query the data object to obtain a corresponding query result, and determining a finally obtained target query result according to each query result.
Fig. 4 is a flowchart of a query task execution process according to an embodiment of the present invention. As shown in fig. 4, when the task to be processed is the query task, the process of executing the task according to the embodiment of the present invention further includes the following steps:
and step S450, determining the data object corresponding to each subtask.
Specifically, when the server receives the task to be processed, the task type of the task to be processed is judged according to the database language corresponding to the task to be processed. When the database language of the task to be processed is a data query statement, determining that the task type of the task to be processed is a query task. Optionally, the query task further includes task content, i.e. data object identification and query condition. The data object identification is used for representing the data object for storing the data to be checked, and the query condition is used for limiting the data to be checked.
Furthermore, after determining the query task, the server analyzes the query task and splits the query task to obtain a plurality of subtasks, and copies the corresponding timestamp information to the split subtasks. Alternatively, the server may split the query task into a plurality of subtasks according to the query condition. For example, when the query condition includes a text feature, a vector feature, and a picture, the query task is split into a subtask 1 including the query condition as the text feature, a subtask 2 including the query condition as the vector feature, and a subtask 3 including the query condition as the picture feature.
Or, the data object corresponding to the data object identifier in the query task further includes a plurality of child data objects. The server can also split the query task according to the query condition and the sub data object type included by the data object corresponding to the data object identification. The embodiment of the present invention is described by taking an example in which the embodiment of the present invention is applied to management of a plurality of cameras in a target area. And a data table corresponding to the plurality of cameras in the target area is stored in a memory in the server or a database in communication connection with the server and is used for storing the picture characteristics acquired by each camera. The method comprises a text feature data table used for storing information text features in pictures, a vector feature data table used for storing information vector features in the pictures and a picture feature data table used for storing picture acquisition information, wherein when query conditions in a query task comprise the pictures, a server extracts the text features and the vector features corresponding to the information in the pictures. Further, the query task is divided into a subtask 1 including a query condition of text features, a subtask 2 including a query condition of vector features, and a subtask 3 including a query condition of picture features.
Furthermore, after determining the subtasks corresponding to the query task, the server can match the corresponding data objects directly according to the query conditions in each subtask. For example, when the query condition is a text feature, the corresponding data object is determined to be a text feature data table for storing the text feature of the information in the picture. And when the query condition is the vector feature, determining that the corresponding data object is a vector feature data table for storing the information vector feature in the picture.
And step S460, processing each subtask according to the timestamp corresponding to the query task, so as to query the data object to obtain a corresponding query result.
Specifically, each subtask corresponding to the query task inherits the timestamp of the query task, and the server processes each subtask according to the timestamp corresponding to the query task so as to obtain a query result in the data object corresponding to each subtask. Furthermore, the processing mode of each subtask is multi-thread parallel processing, so that the processing of each subtask can be independently completed, the interference of other subtasks is avoided, and the task processing efficiency is improved. Optionally, a thread may also include multiple subtasks.
In the embodiment of the invention, the process of executing each subtask by the server is to determine the inter-state of the data object corresponding to the subtask, and carry out data query according to the time state and the time stamp corresponding to the query task. And when the timestamp of the query task corresponding to the subtask is not greater than the time state of the corresponding data object, executing the subtask to query the corresponding data object to obtain a corresponding query result. And when the timestamp of the inquiry task corresponding to the subtask is larger than the time state of the corresponding data object, generating and returning error reporting information. Optionally, when the timestamp corresponding to the subtask is greater than the time status of the data object, the time status update of the data object may be further waited until the timestamp corresponding to the subtask is not greater than the time status of the corresponding data object, and the subtask is executed to determine the query result of the corresponding data object. The query mode can ensure the real-time performance of the data in the data object and avoid the omission of query results.
And step S470, determining a target query result according to each query result.
Specifically, after each subtask is processed, a target query result is determined according to a query result obtained by searching in the corresponding data object. In the embodiment of the present invention, the server may preset a determination rule of the target query result, for example, take an intersection of the query results when processing the subtasks in parallel. Or determining the query result corresponding to the last executed subtask as the target query result when executing each subtask according to the execution sequence.
In an optional implementation manner of the embodiment of the present invention, the server may process each subtask corresponding to the query task in parallel. The data objects corresponding to the server include a text feature data table for storing text features of information in the picture, a vector feature data table for storing vector features of information in the picture, and a picture feature data table for storing picture acquisition information. Each text feature in the text feature data table has a corresponding picture, and each vector feature in the vector feature data table has a corresponding picture. The server executes the subtask 1 to screen a plurality of text features included in the text feature data table to obtain a plurality of target text features, and determines pictures corresponding to the target text features as first candidate pictures. And executing the subtask 2 to screen a plurality of target vector characteristics from the plurality of vector characteristics included in the vector characteristic data table, and determining the picture corresponding to each target vector characteristic as a second candidate picture. And executing a subtask 3 to filter a plurality of third candidate pictures from the plurality of pictures included in the picture feature data table. The server may determine an intersection of the plurality of first candidate pictures, the second candidate picture, and the third candidate picture as a target query result of the query task.
In another optional implementation manner of the embodiment of the present invention, the server may also perform partial parallel processing, and some have a sequential processing order. The subtasks corresponding to the query task determined by the server are respectively a subtask 1 including a query condition as a text feature, a subtask 2 including a query condition as a vector feature, and a subtask 3 including a query condition as a picture feature, and the data objects corresponding to the server include a text feature data table for storing information text features in a picture, a vector feature data table for storing information vector features in a picture, and a picture feature data table for storing picture acquisition information. Each text feature in the text feature data table has a corresponding picture, and each vector feature in the vector feature data table has a corresponding picture. The server executes the subtask 1 to screen a plurality of text features included in the text feature data table to obtain a plurality of target text features, and determines pictures corresponding to the target text features as first candidate pictures. Meanwhile, the subtask 2 is executed to obtain a plurality of target vector features by screening among the plurality of vector features included in the vector feature data table, and a picture corresponding to each target vector feature is determined to be a second candidate picture. The server screens the first candidate pictures and the second candidate pictures in the picture characteristic data table, and then executes the subtask 3 to screen a plurality of target pictures in the plurality of first candidate pictures and the second candidate pictures to serve as target query results of the query task.
Fig. 5 is a schematic diagram of a query task execution process according to an embodiment of the present invention. As shown in fig. 5, when determining that the task type of the task to be processed is the query task, the embodiment of the present invention directly determines that the current query task is the query task 50. Further, the server divides the query task into a plurality of subtasks 51, each of which inherits the timestamp of the query task. The server executes each subtask respectively to query the corresponding data object. Specifically, the process of the server executing the subtask includes first comparing the time stamp corresponding to the subtask with the time status 52 of the corresponding data object, and querying the corresponding data object 53 when the time stamp is smaller than the time status of the corresponding data object, that is, the time represented by the time stamp is before the time represented by the time status of the corresponding data object, to obtain the corresponding query result. When the timestamp is not less than the corresponding data object time state, i.e., the time represented by the timestamp is after the time represented by the corresponding data object time state, an error message 54 is returned. Optionally, the server may also wait for the next update of the time status of the data object, so that when the timestamp corresponding to the subtask is smaller than the time status of the corresponding data object, the corresponding data object is queried. And the server determines a target query result of the query task according to the query result corresponding to each subtask.
Furthermore, in order to ensure that each data object stored in the server or the database connected with the server is in a latest update state, the query result cannot be obtained within a long time after the query task is received. The server may generate a blank editing task according to a preset task generation rule to update the time status of each data object. Specifically, the server may query, at a preset frequency, a timestamp corresponding to the editing task recorded most recently in the task log, and generate a blank editing task when a time length of the timestamp from the current time is greater than a time length threshold, that is, a time length of the last determined editing task is too long, where the blank editing task does not modify any data object. Further, after the server generates the blank editing task, the blank editing task is also used as a task to be processed, a corresponding timestamp is allocated to the blank editing task, and the corresponding timestamp is recorded in a task log. When the server processes the blank editing task as a target editing task, the time state corresponding to each data is updated only through the timestamp corresponding to the blank editing task without changing each data object corresponding to the server.
The data processing method of the embodiment of the invention improves the data searching efficiency by splitting the query task into a plurality of subtasks for processing, and can cope with complex scenes needing to search various types of data. Meanwhile, the task is processed based on the timestamp and the task type, when the task type is the editing task, the task is recorded according to the corresponding timestamp through the task log, and the orderliness and the data consistency during frequent task processing are guaranteed. When the task type is the query task, whether the task is executed is judged by comparing the timestamp with the time state of the data object, so that the timeliness of the query result is ensured.
Fig. 6 is a schematic diagram of an electronic device of an embodiment of the invention. As shown in fig. 6, the electronic device shown in fig. 6 is a general address query device, which includes a general computer hardware structure, which includes at least a processor 60 and a memory 61. The processor 60 and the memory 61 are connected by a bus 62. The memory 61 is adapted to store instructions or programs executable by the processor 60. Processor 60 may be a stand-alone microprocessor or a collection of one or more microprocessors. Thus, processor 60 implements the processing of data and the control of other devices by executing instructions stored by memory 61 to thereby perform the method flows of embodiments of the present invention as described above. The bus 62 connects the above components together, and also connects the above components to a display controller 63 and a display device and an input/output (I/O) device 64. Input/output (I/O) devices 64 may be a mouse, keyboard, modem, network interface, touch input device, motion sensing input device, printer, and other devices known in the art. Typically, the input/output devices 64 are connected to the system through an input/output (I/O) controller 56.
As will be appreciated by one of skill in the art, embodiments of the present application may be provided as a method, apparatus (device) or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may employ a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations of methods, apparatus (devices) and computer program products according to embodiments of the application. It will be understood that each flow in the flow diagrams can be implemented by computer program instructions.
These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows.
These computer program instructions may also be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows.
Another embodiment of the invention is directed to a non-transitory storage medium storing a computer-readable program for causing a computer to perform some or all of the above-described method embodiments.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be accomplished by specifying the relevant hardware through a program, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A method of data processing, the method comprising:
determining tasks to be processed in sequence;
respectively determining the task type corresponding to each task to be processed;
distributing corresponding time stamps to the tasks to be processed in sequence according to the determined sequence;
sequentially processing each task to be processed according to the corresponding timestamp and the task type;
wherein the sequentially processing the tasks to be processed according to the corresponding timestamps and task types comprises:
in response to that the task type corresponding to the task to be processed is a query task, splitting the task to be processed to obtain a plurality of subtasks, wherein the timestamp of each subtask is the same as the timestamp corresponding to the query task;
processing each subtask according to the corresponding timestamp, wherein the processing mode of each subtask is multithread parallel processing;
wherein the respectively processing the subtasks according to the corresponding timestamps includes:
determining a data object corresponding to each subtask;
respectively processing each subtask according to the timestamp corresponding to the query task to query a data object to obtain a corresponding query result;
determining a target query result according to each query result;
wherein, respectively processing each subtask according to the timestamp corresponding to the query task, so as to query the data object to obtain the corresponding query result comprises:
and in response to the timestamp corresponding to the subtask being greater than the time status of the data object, waiting for the time status of the data object to be updated, and executing the subtask until the corresponding timestamp is not greater than the time status of the corresponding data object, so as to determine a query result of the corresponding data object.
2. The method according to claim 1, wherein the determining the task type corresponding to each task to be processed respectively comprises:
determining a database language corresponding to each task to be processed;
responding to the database language to be a data control statement or a data definition statement, and determining the task to be processed as an editing task;
and responding to the database language as a data query statement, and determining the task to be processed as a query task.
3. The method of claim 1, wherein the sequentially processing each of the tasks to be processed according to the corresponding timestamp and task type further comprises:
responding to the fact that the corresponding task type is an editing task, and recording the task to be processed in a task log according to the corresponding time stamp sequence;
and processing each editing task in the task log in sequence.
4. The method of claim 3, wherein the sequentially processing each of the editing tasks in the task log comprises:
determining a to-be-processed editing task recorded first in a task log as a target editing task;
determining at least one data object corresponding to the target editing task;
distributing corresponding threads for each data object;
and executing the target editing task in parallel through each thread so as to edit the corresponding data objects respectively.
5. The method of claim 4, wherein each of the data objects has a corresponding temporal state;
the sequentially processing each editing task in the task log further comprises:
and in response to the completion of the editing of the corresponding data object, updating the time state of the data object according to the timestamp of the target editing task.
6. The method of claim 2, further comprising:
inquiring a timestamp corresponding to a latest recorded editing task in a task log at a preset frequency;
responding to the fact that the time length of the timestamp from the current time is larger than a time length threshold value, and generating a blank editing task, wherein the blank editing task does not change any data object;
and distributing a corresponding time stamp for the blank editing task and recording the time stamp to the task log.
7. The method according to claim 1, wherein the processing the subtasks respectively according to the timestamp corresponding to the query task to query the data object to obtain the corresponding query result comprises:
determining the time state of the data object corresponding to each subtask;
responding to the time state that the time stamp of the query task corresponding to the subtask is not larger than the time state of the corresponding data object, and querying the corresponding data object to obtain a corresponding query result;
and returning error reporting information in response to the fact that the timestamp of the query task corresponding to the subtask is larger than the time state of the corresponding data object.
8. A computer readable storage medium storing computer program instructions which, when executed by a processor, implement the method of any one of claims 1-7.
9. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110609634.7A CN113392252B (en) | 2021-06-01 | 2021-06-01 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110609634.7A CN113392252B (en) | 2021-06-01 | 2021-06-01 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113392252A CN113392252A (en) | 2021-09-14 |
| CN113392252B true CN113392252B (en) | 2023-01-17 |
Family
ID=77619809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110609634.7A Active CN113392252B (en) | 2021-06-01 | 2021-06-01 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392252B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103384206A (en) * | 2012-05-02 | 2013-11-06 | 中国科学院计算机网络信息中心 | Concurrent processing method and system for mass data |
| CN109558237A (en) * | 2017-09-27 | 2019-04-02 | 北京国双科技有限公司 | A kind of task status management method and device |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106919691A (en) * | 2017-03-06 | 2017-07-04 | 广东神马搜索科技有限公司 | Method, device and the searching system retrieved based on web page library |
| CN107315629A (en) * | 2017-06-14 | 2017-11-03 | 北京小米移动软件有限公司 | Task processing method, device and storage medium |
| CN109308214A (en) * | 2017-07-27 | 2019-02-05 | 北京京东尚科信息技术有限公司 | Data task processing method and system |
| CN111488492B (en) * | 2020-04-08 | 2023-11-17 | 北京百度网讯科技有限公司 | Method and device for searching graph database |
| CN112860730A (en) * | 2021-03-29 | 2021-05-28 | 中信银行股份有限公司 | SQL statement processing method and device, electronic equipment and readable storage medium |
-
2021
- 2021-06-01 CN CN202110609634.7A patent/CN113392252B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103384206A (en) * | 2012-05-02 | 2013-11-06 | 中国科学院计算机网络信息中心 | Concurrent processing method and system for mass data |
| CN109558237A (en) * | 2017-09-27 | 2019-04-02 | 北京国双科技有限公司 | A kind of task status management method and device |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113392252A (en) | 2021-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339071B (en) | Method and device for processing multi-source heterogeneous data | |
| US10789231B2 (en) | Spatial indexing for distributed storage using local indexes | |
| US7987160B2 (en) | Status tool to expose metadata read and write queues | |
| CN110908641B (en) | Visualization-based stream computing platform, method, device and storage medium | |
| CN113767388A (en) | Techniques for workflow analysis and design task optimization | |
| CN107870949B (en) | Data analysis job dependency relationship generation method and system | |
| CN106874281B (en) | Method and device for realizing database read-write separation | |
| CN110196884B (en) | Data writing method based on distributed database, storage medium and electronic equipment | |
| CN111723161A (en) | Data processing method, device and equipment | |
| CN112835714A (en) | Container arrangement method, system and medium for CPU heterogeneous cluster in cloud edge environment | |
| JP5713652B2 (en) | Data search apparatus, method, and program | |
| CN113886111B (en) | Workflow-based data analysis model calculation engine system and operation method | |
| CN108959458B (en) | Data generation and use method, system, medium and computer device | |
| CN112800091B (en) | Flow batch integrated calculation control system and method | |
| CN113392252B (en) | Data processing method and device | |
| CN113761052A (en) | Database synchronization method and device | |
| CN110737794A (en) | Image query method, system, server and storage medium | |
| CN114610719B (en) | Cross-cluster data processing method and device, electronic equipment and storage medium | |
| US11157506B2 (en) | Multiform persistence abstraction | |
| CN109902067B (en) | File processing method and device, storage medium and computer equipment | |
| JP6506773B2 (en) | INFORMATION PROCESSING APPARATUS, METHOD, AND PROGRAM | |
| CN110019448B (en) | Data interaction method and device | |
| CN117390040B (en) | Service request processing method, device and storage medium based on real-time wide table | |
| KR101638048B1 (en) | Sql query processing method using mapreduce | |
| CN115455010B (en) | Data processing method based on milvus database, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |