KR102635613B1 - Method for embedding non-structured data and apparatus for performing the method - Google Patents

Abstract

The present invention relates to an embedding method for unstructured data and an apparatus for performing such method. The method of embedding unstructured data in a data processing system may include a step in which the embedding artificial intelligence model group of the data processing system changes unstructured data into an embedding value and a step in which the database of the data processing system stores the embedding value.

Description

Embedding method for unstructured data and apparatus for performing the method {Method for embedding non-structured data and apparatus for performing the method}

The present invention relates to an embedding method for unstructured data and an apparatus for performing such method. More specifically, it relates to an embedding method for unstructured data to reduce data size and output fast results by storing the characteristics of unstructured data, and a device for performing this method.

Due to the rapid non-face-to-face environment and mobile-first strategy, the explosive increase and creation of structured and unstructured data every year is demanding new decisions and services utilizing big data in all fields.

As such, the rapid increase and consumption of data is expected to accelerate further in the future, and finding future growth engines by collecting, refining and analyzing various patterns contained in not only structured data but also unstructured data will become a new business model for companies. It is becoming.

Existing prior art includes domestic application number 10-2014-0036626.

The purpose of the present invention is to solve all of the above-mentioned problems.

In addition, the purpose of the present invention is to save file characteristics through embedding in unstructured data to reduce data size and produce quick results.

In addition, the present invention aims to derive results more quickly by accelerating the process of embedding unstructured data using CONVERT and SEARCH syntax and performing unstructured search through embedding through GPU. .

A representative configuration of the present invention to achieve the above object is as follows.

According to an embodiment of the present invention, a method of embedding unstructured data in a data processing system includes the steps of an embedding artificial intelligence model group of a data processing system changing the unstructured data into an embedding value, and a database of the data processing system converting the embedding value into an embedding value. It may include the step of storing.

Meanwhile, the embedding artificial intelligence model group includes at least one different artificial intelligence model that generates a necessary embedding value, and the embedding value may be determined based on a feature map of the at least one different artificial intelligence model.

Additionally, the embedding value is generated by a convert statement based on an extended SQL engine, and the embedding value can be searched by a search statement based on the extended SQL engine.

According to another embodiment of the present invention, a data processing system that performs embedding of unstructured data may be implemented so that an embedding artificial intelligence model group changes the unstructured data into an embedding value and a database stores the embedding value.

Meanwhile, the embedding artificial intelligence model group includes at least one different artificial intelligence model that generates a necessary embedding value, and the embedding value may be determined based on a feature map of the at least one different artificial intelligence model.

Additionally, the embedding value is generated by a convert statement based on an extended SQL engine, and the embedding value can be searched by a search statement based on the extended SQL engine.

According to the present invention, the data size can be reduced and quick results can be obtained by storing file characteristics through embedding in unstructured data.

In addition, according to the present invention, the process of embedding unstructured data using CONVERT and SEARCH syntax and the process of unstructured search through embedding are accelerated through GPU, so that results can be derived more quickly.

Figure 1 is a conceptual diagram showing an existing data processing system.
Figure 2 is a conceptual diagram showing a data processing system for processing structured data and unstructured data on one platform according to an embodiment of the present invention.
Figure 3 is a conceptual diagram showing a data processing system for processing structured data and unstructured data on one platform according to an embodiment of the present invention.
Figure 4 is a conceptual diagram showing the operation of a data processing system according to an embodiment of the present invention.
Figure 5 is a conceptual diagram showing the operation of a data processing system according to an embodiment of the present invention.
Figure 6 is a conceptual diagram showing a data processing method based on a data processing system according to an embodiment of the present invention.
Figure 7 is a conceptual diagram showing a method of analyzing queries and processing data in a data processing system according to an embodiment of the present invention.
Figure 8 is a conceptual diagram showing a resource distribution algorithm according to an embodiment of the present invention.
Figure 9 is a conceptual diagram showing a data processing system that processes input multi-query according to an embodiment of the present invention.
Figure 10 is a conceptual diagram showing the operation of a multi-query scheduler according to an embodiment of the present invention.
Figure 11 is a conceptual diagram showing a method of processing an input multi-query according to an embodiment of the present invention.
Figure 12 is a conceptual diagram showing a server that provides processing services for structured data and unstructured data according to an embodiment of the present invention.
Figure 13 is a conceptual diagram showing a query processing method of a shared workspace hub according to an embodiment of the present invention.
Figure 14 is a conceptual diagram showing a resource distribution method according to an embodiment of the present invention.
Figure 15 is a conceptual diagram showing an unstructured embedding method for unstructured data according to an embodiment of the present invention.
Figure 16 is a conceptual diagram showing an unstructured embedding method for unstructured data according to an embodiment of the present invention.
Figure 17 is a conceptual diagram showing the results of using the convert syntax and search syntax according to an embodiment of the present invention.
Figure 18 is a conceptual diagram showing a method for binarizing an embedding value according to an embodiment of the present invention.
Figure 19 is an example of binarization according to an embodiment of the present invention.
Figure 20 discloses a method for storing an embedding value or a binary value according to an embodiment of the present invention.
Figure 21 is a conceptual diagram showing a workspace backup method according to an embodiment of the present invention.
Figure 22 is a flowchart showing a workspace backup method according to an embodiment of the present invention.
Figure 23 is a conceptual diagram showing a server backup method according to an embodiment of the present invention.
Figure 24 is a flowchart showing a server backup method according to an embodiment of the present invention.
Figure 25 is a conceptual diagram showing a workspace migration method according to an embodiment of the present invention.
Figure 26 is a flowchart showing a workspace migration method according to an embodiment of the present invention.
Figure 27 is a conceptual diagram showing a server migration method according to an embodiment of the present invention.
Figure 28 is a flowchart showing a server migration method according to an embodiment of the present invention.
Figure 29 is a conceptual diagram showing an unstructured model caching method according to an embodiment of the present invention.
Figure 30 is a conceptual diagram showing a first caching algorithm according to an embodiment of the present invention.
Figure 31 is a conceptual diagram showing an unstructured model caching method according to an embodiment of the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein may be implemented with changes from one embodiment to another without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description described below is not intended to be limited, and the scope of the present invention should be taken to encompass the scope claimed by the claims and all equivalents thereof. Like reference numbers in the drawings indicate identical or similar elements throughout various aspects.

Hereinafter, several preferred embodiments of the present invention will be described in detail with reference to the attached drawings in order to enable those skilled in the art to easily practice the present invention.

Figure 1 is a conceptual diagram showing an existing data processing system.

In Figure 1, a data processing system that processes existing structured data and unstructured data is disclosed.

Referring to FIG. 1, a data processing method for structured data 100 and unstructured data 120 in an existing data processing system is disclosed.

Structured data 100 is data that is stored in tables according to schema and can be connected between tables through relationships. Structured data 100 can be displayed in rows and columns with an appropriately defined schema for the information it holds. Each column represents a different property, while each row contains data associated with a single instance of the property. Rows and columns can form a table that can be easily referenced, different tables can be linked, and a relational database 140 can be formed when several tables are sequentially linked.

Unstructured data 120 is the opposite of structured data 100, and is data whose meaning is difficult to easily understand because there are no set rules, and may include data such as voice, image, and video.

The existing data processing system could only query structured data (100) based on SQL (structured query language), and a NoSQL database without a specific schema was used to process unstructured data (120).

In addition, the existing data processing system was capable of real-time querying of structured data (100), but real-time querying of unstructured data (120) was not possible. In existing database processing systems, unstructured data 120 is processed through batch processing instead of real time processing. Because of this, real-time search for images, videos, and voices was impossible in existing data processing systems. More specifically, in existing data processing systems, it is difficult to analyze large amounts of unstructured data 120 in real time. Therefore, processing was performed based on the Lambda architecture (150), which combines a data table that can be acquired in real time and a batch table that has been calculated in advance at a fixed time, and structured data (100) and unstructured data (120) are separated. It was processed based on DMBS (database management system).

Additionally, existing data processing systems used various pipelines, various frameworks, and various languages for batch processing of unstructured data 120. Therefore, processing of data based on a single governance was impossible, and maintenance after development was difficult.

Additionally, in order to learn about unstructured data 120 in the existing data processing system, artificial intelligence learning within the database was not possible. The existing data processing system performed learning on structured data (100) based on an AI engine implemented in the database, but learning on unstructured data (120) was not processed based on SQL within the database, so unstructured data within the database AI engine modeling based on was impossible.

In addition, when performing modeling for an AI engine, the existing data processing system creates a sample table 160 through sampling from the parameter table of the operating system to perform modeling, and a modeling platform that performs modeling and actual operation are used to perform modeling. The operating platforms are different. In this case, the problem of inaccurate modeling results occurs due to differences between the modeling platform and the operating platform.

In existing data processing systems, it takes a lot of time to perform AI modeling using sample data.

In existing data processing systems, the process of extracting sample data from a parameter table is performed. Because parameter data can exist in various forms other than tables, it takes time to transform and extract the data, and a considerable amount of time is also required to preprocess the data for modeling.

In addition, in the AI modeling process of existing data processing systems, sample data includes both structured and unstructured data, and in order to perform structured/unstructured AI modeling, Lambda architecture must be applied to existing data processing systems. If you develop through Lambda architecture, you will use various platforms and languages, but you will waste a lot of time integrating them due to differences in characteristics and interoperability issues between platforms.

In addition, while extracting data from the parameter table and doing AI modeling on the Lambda architecture, new data is accumulated in the parameter table/data in real time. Then, when applying the AI model created in the existing data processing system, the prediction result (model's There is a problem that the result value is not accurate. In that case, it takes a lot of time to do modeling again, going through processes 1 and 2.

When the data processing system according to the embodiment of the present invention is used, parameter data is managed in one form (table), and the process of extracting sample data is possible through a simple query statement and does not require a lambda architecture. AI modeling for structured and unstructured data also has the advantage of being able to be easily processed without integration problems using one platform and one language.

Therefore, the data processing platform according to an embodiment of the present invention can process structured data 100 and unstructured data 120 based on one language based on one platform.

In addition, the data processing platform according to an embodiment of the present invention not only enables more accurate modeling by having an operating platform and a modeling platform on one platform, but also enables structured data 100 and unstructured data 120 without separate batch processing. It can provide AI modeling functions based on .

Hereinafter, the functions of the data processing platform according to a more specific embodiment of the present invention are disclosed.

Figure 2 is a conceptual diagram showing a data processing system for processing structured data and unstructured data on one platform according to an embodiment of the present invention.

In Figure 2, a data processing system for processing structured data and unstructured data on one platform is disclosed.

Referring to FIG. 2, the data processing system is capable of processing unstructured data 220 and structured data 210 on one platform. In the present invention, a data processing syntax for processing unstructured data 220 together with structured data 210 on one platform is newly defined, and an extended SQL (extended SQL) that can use the newly defined data processing syntax is provided. 240) can be defined.

General queries for structured data 210 may be processed based on existing SQL such as PostgreSQL, and queries for unstructured data may be processed based on extended SQL 240 newly defined in the present invention.

An extended SQL engine 250 may be defined to process the newly defined data processing syntax on the extended SQL 240. The extended SQL engine 250 may be an engine that enables processing of newly defined data processing syntax.

Unlike existing data processing systems, nested queries (230) are possible based on the extended SQL engine (250). Nested query 230 is a mixed query for structured data 210 and unstructured data 220, enabling sequential or complex processing of structured data 210 and unstructured data 220 stored in the database. can do.

That is, unlike the existing structured data 210 and unstructured data 220 that are processed based on separate DMBS (database management system), in the present invention, the structured data 210 and unstructured data 220 are processed on one platform. It is processed based on the extended SQL engine 250, and data processing for structured data 210 and unstructured data 220 is performed simultaneously on one database 260 based on nested query 230. It can be done. Based on this, AI modeling for structured data 210 and unstructured data 220 is also performed on the AI engine 270 of the data processing system.

The AI engine may be provided in advance with various AI engines such as classification models, regression models, recommendation models, and voice recognition models, or can be used without restrictions, such as models directly created by the user or AI engines provided as open source.

The data processing system of the present invention can process unstructured data 220 within one platform without separate batch processing, separate language, or separate platform. The data processing system of the present invention is an integrated platform that allows both structured data 210 and unstructured data 220 to be queried using only SQL and enables AI modeling for structured data 210 and unstructured data 220. Therefore, since the modeling platform and the operating platform are the same, the problem of poor modeling accuracy due to different parameters can be reduced.

In addition, the data processing system of the present invention can apply the functions of RDB (relational database), AI, and big data platform in one platform, and can dramatically reduce inefficiencies that occur during AI-based digital transformation. Based on big data processing and distributed parallel processing technology, it enables data processing more than twice as fast as before.

That is, according to an embodiment of the present invention, a method of processing structured data and unstructured data in a database includes the steps of a data processing system receiving a nested query and the data processing system performing processing on the nested query. can do. A nested query may be a query that mixes a first query for unstructured data and a second query for structured data.

The step of processing a nested query includes a step in which the data processing system performs processing on unstructured data based on an extended SQL engine that processes extended structured query language (SQL), and a data processing system that processes PostgreSQL (PostgreSQL). It may include processing structured data based on a general SQL engine that processes extended structured query language.

The data processing system creates data tables for structured data and data tables for unstructured data and processes them in one database, and the data processing system supports artificial intelligence engine modeling based on structured data and unstructured data in one database. You can.

Additionally, according to an embodiment of the present invention, the data processing system may perform individual processing for each of structured data and unstructured data. The data processing system may be implemented to receive unstructured data processing queries and structured data processing queries, and process the unstructured data processing queries and structured data processing queries. An unstructured data processing query may be a query for processing only unstructured data, and a structured data processing query may be a query for processing only structured data.

Unstructured data processing queries can be processed based on extended SQL and extended SQL engines, and structured data processing queries can be processed based on general SQL (PostgreSQL) and general SQL engines.

Figure 3 is a conceptual diagram showing a data processing system for processing structured data and unstructured data on one platform according to an embodiment of the present invention.

In Figure 3, a previously defined general query and an extended query defined based on extended SQL for unstructured data form a nested query, and a method of processing the nested query in a data processing system is disclosed.

Referring to FIG. 3, a nested query for processing unstructured data and structured data may be input as the input query 300.

For example, a nested query may include a first query 310, a second query 320, and a third query 330, and the first query 310 and the third query 330 are extended queries. 350, and the second query 320 may be a general query 360.

The first query 310 may be PRINT IMAGE, the second query 320 may be SELECT, and the third query 330 may be SEARCH IMAGE. The first query 310, the second query 320, and the third query 330 may form an input query in a nested structure.

The input query 300 may be parsed through a parser. Based on the lexer, nested queries are divided into general queries (360) and extended queries (350), and the parser can split the general queries (360) and extended queries (350).

The first query 310, the second query 320, and the third query 330 may be interpreted and processed through cloud analysis and a query tree. The third query 330, the second query 320, and the first query 310 may be processed in this order.

The first query 310 and the third query 330 are extended queries 350 and can be processed based on an extended SQL engine, and the second query 320 is a general query, which is PostgreSQL, a SQL engine for general query processing. It can be processed based on the engine.

The standardized SQL engine and PostgreSQL engine can be connected to one database and process queries. Artificial intelligence learning based on structured and unstructured data is possible based on one database.

Figure 4 is a conceptual diagram showing the operation of a data processing system according to an embodiment of the present invention.

In Figure 4, an extended SQL query function for simultaneously processing structured data and unstructured data on one platform is disclosed.

Referring to FIG. 4, the query function for unstructured data can be performed based on the extended SQL below.

(1) Check storage model (LIST) (410)

Users can use the "LIST" syntax to check pre-built models and user-created models for unstructured data tables for processing unstructured data.

For example, it is possible to check user-generated models created by users through the LIST MODEL function, and it is possible to check pre-created models using the LIST PREBUILT MODEL function.

(2) Add unstructured characteristics (CONVERT USING) (430)

Using the "CONVERT USING" statement, users can use information from unstructured data such as images, videos, and voices to convert it into vector format using a numerical algorithm and add this value to the data set to be used.

Table 2 below is an example of the CONVERT USING statement.

<Table 2>

CONVERT USING [AI model to use]

OPTIONS (

table_name=[table name to be saved]

)

AS

[Dataset to use]

For example, using the CONVERT USING function, an image file that exists in a specific path can be created on the database as a data table using an additional attribute extraction artificial intelligence model.

(3) Unstructured data search (SEARCH) (440)

The SEARCH syntax can be used to search for content, meaning, or similarity in unstructured data.

Table 3 below is an example of the SEARCH syntax.

<Table 3>

SEARCH [custom data table name]

USING [AI model to use]

AS [dataset to use]

For example, the SEARCH statement can be used to search for similar images based on an image quantification artificial intelligence model.

(4) Print results (PRINT) (450)

Users can output image, audio, and video files using the "PRINT" syntax. Additionally, you can use a subquery to immediately output the results obtained through the "PRINT" statement.

Table 4 below is an example of the "PRINT" syntax.

<Table 4>

PRINT IMAGE, AUDIO, VIDEO

AS [data set to output]

For example, you can use the PRINT query statement to output image files/video files/audio files in a data table.

The above query syntax is a newly defined syntax for SQL confirmed in the present invention.

It is possible to search image data, audio data, and video data based on keywords or text based on an unstructured data table created based on the above query syntax. In addition, it is possible to search image data, audio data, and video data based on image data, audio data, and video data.

That is, in the data processing system according to an embodiment of the present invention, real-time search for the above unstructured data is possible in addition to real-time search for existing structured data. In addition, based on the above extended SQL, nested queries, which are a combination of queries on unstructured data and structured data, are also possible, making modeling using both unstructured and structured data possible.

Figure 5 is a conceptual diagram showing the operation of a data processing system according to an embodiment of the present invention.

In Figure 5, the ML (machine learning) function of extended SQL for simultaneously processing structured data and unstructured data on one platform is disclosed.

Referring to FIG. 5, ML functions for unstructured data can be performed based on extended SQL as shown below.

(1) Model learning (BUILD MODEL) (510)

Users can create an artificial intelligence model using the “BUILD MODEL” statement.

Table 5 below is an example of the “BUILD MODEL” syntax.

<Table 5>

BUILD MODEL [custom model name]

USING [Artificial intelligence model to use]

OPTIONS([Option values required when creating an artificial intelligence model])

AS [dataset to use]

For example, a user can use the "BUILD MODEL" syntax to create a movie recommendation model that recommends movies.

(2) Model EVALUATE (520)

Users can perform performance evaluation of artificial intelligence models using the “EVALUATE” statement.

Table 6 below is an example of the "EVALUATE" syntax.

<Table 6>

EVALUATE

USING [name of previously learned model]

OPTIONS ([Option values required when evaluating each model])

AS

[Dataset to use]

For example, the "EVALUATE" syntax can be used to evaluate the classification model that the user created in Learning a Model.

(3) Model retraining (FIT MODEL) (530)

Users can retrain the model using the "FIT MODEL" statement.

Table 7 below is an example of the “FIT MODEL” syntax.

<Table 7>

FIT MODEL [custom model name]

USING [Name of previously learned model | Pre-trained artificial intelligence model name]

OPTIONS ([Option values required when creating an artificial intelligence model])

AS

[Dataset to use]

For example, using "FIT MODEL", a user can retrain a previously created model using a newly added dataset.

(4) UPLOAD MODEL (540)

Users can use the "UPLOAD MODEL" syntax to upload their models as extended SQL so that models created directly/self-created in the Python environment can be operated based on extended SQL.

UPLOAD MODEL [custom model name]

OPTIONS([Options required when uploading model])

FROM [path to model to upload]

(5) Applying the model (PREDICT) (550)

Using the "PREDICT" syntax, users can apply artificial intelligence models to test data sets to perform tasks such as prediction, classification, and recommendations.

Table 9 below is an example of the "PREDICT" syntax.

<Table 9>

PREDICT

USING [name of previously learned model]

OPTIONS ([Option values required for inference for each model])

AS

[Test dataset to use]

For example, using the “PREDICT USING” syntax, it is possible to recommend a list of movies that the user with user ID 31 might like using the existing recommendation model created in the previous model training.

(6) Deleting a model (DELETE MODEL) (560)

Users can delete created or uploaded models using the "DELETE MODEL" statement.

Table 10 below is an example of the “DELETE MODEL” statement.

<Table 10>

DELETE MODEL [model name to delete]

For example, the movie recommendation model that the user created in model training based on the "DELETE MODEL" statement may be deleted from the database.

Based on the above extended SQL, AI modeling based on unstructured data and structured data can be performed on a single platform, a data processing system, without a separate batch process.

In the data processing system, a pre-generated AI model and an AI model created by a user may be located. Through this AI model creation, various AI models such as classification models, regression models, recommendation systems, and voice recognition models can be created.

Figure 6 is a conceptual diagram showing a data processing method based on a data processing system according to an embodiment of the present invention.

In Figure 6, a method of processing data on a separate database based on the data processing system described above is disclosed.

Referring to FIG. 6, as described above with reference to FIGS. 1 to 5, processing of structured data and unstructured data may be performed based on the data processing system's own database. However, users can use their own database and utilize the functions of the extended SQL and extended SQL engine provided by the data processing system based on the API.

The processing of structured and unstructured data based on the data processing system's own database can be expressed in the term internal data processing. The processing of structured and unstructured data based on an external database rather than the data processing system's own database can be expressed in the term external data processing.

In the case of internal data processing, it can be processed based on the process disclosed in FIGS. 1 to 5 described above.

In order to use the data processing system according to an embodiment of the present invention from the outside for external data processing, external data must be stored and converted into the data processing system of the present invention using the provided 'API' or 'data transfer method'. For data that has been stored and converted, the data processing system of the present invention can be utilized using the API. That is, both the internal engine and the PostgreSQL engine can perform data processing by accessing the database according to the embodiment of the present invention rather than an external database.

In the case of external data processing, users can perform learning based on separate unstructured data stored in the user's database based on the functions of extended SQL and extended SQL engine through API.

For example, a specific user may be a security company and operate a user database that stores CCTV footage. Based on the extended SQL of the data processing system of the present invention, users can perform artificial intelligence learning on CCTV images based on data stored in the user database. Structured data and unstructured data can be inserted from an external database into the database of the data processing system of the present invention based on a query statement for unstructured data for processing structured data and unstructured data defined in the present invention. AI modeling for structured data and unstructured data input to the data processing system according to an embodiment of the present invention can be performed based on the AI engine of the data processing system according to an embodiment of the present invention.

That is, the method of processing structured data and unstructured data on a plurality of different databases includes the steps of a data processing system receiving external data from an external database, the data processing system converting the external data, and the data processing system converting the external data. It may include processing the external data.

At this time, the external data includes structured data and unstructured data, the data processing system processes structured data and unstructured data based on nested queries, and the nested query is the first query for unstructured data and the second query for structured data. It may be a mixed query of 2 queries.

A data processing system can process unstructured data based on unstructured data processing queries, and the data processing system can process structured data based on structured data processing queries.

A nested query is a query that combines a first query for unstructured data and a second query for structured data, an unstructured data processing query is a query for processing only the unstructured data, and a structured data processing query is a query for processing only structured data. It could be a query for

The data processing system creates data tables for structured data and data tables for unstructured data and processes them in one database, and the data processing system supports artificial intelligence engine modeling based on structured data and unstructured data in one database. You can.

Figure 7 is a conceptual diagram showing a method of analyzing queries and processing data in a data processing system according to an embodiment of the present invention.

In Figure 7, a method is disclosed that analyzes an input query and processes unstructured data and structured data in one data processing system, but utilizes different processing resources when processing the query.

Referring to FIG. 7, a query for processing unstructured data and/or structured data may be input as an input query. The entered query may be a general query for unstructured data or structured data, or a nested query for mixed processing of unstructured data and structured data.

The input query may be parsed through the parser 700. Nested queries are divided into general queries and extended queries, and the parser unit 700 can split the general queries and extended queries.

A plurality of queries included in the input query may be analyzed and processed through the cloud analyzer 710 and the query tree 720.

In the present invention, for convenience of explanation, a nested query is input as an input query, and the nested query may include a first query 715, a second query 725, and a third query 735. The cloud analyzer 710 may determine which computing resources should be used to process the first query 715, second query 725, and third query 735. More specifically, the cloud analyzer 710 can determine whether the query should be executed on the CPU 770 or the GPU 780 through analysis of a plurality of queries included in the nested query. The cloud analyzer 710 determines the expected execution efficiency and expected execution speed of each of a plurality of queries based on the resource distribution algorithm according to an embodiment of the present invention, and determines the expected execution efficiency and expected execution speed of the plurality of queries based on the expected execution efficiency and expected execution speed. Each can be assigned to the CPU 770 or GPU 780.

It can be assumed that the first query 715 and the third query 735 are extended queries, and the second query 725 is a general query. The first query 715 and the third query 735 can be processed based on the extended SQL engine 760, and the second query 725 is a general query, and the PostgreSQL engine (750) is a SQL engine for general query processing. ) can be processed based on.

At this time, based on the resource distribution algorithm, the first query 715 may be processed based on the GPU 780, the third query 735 may be processed based on the CPU 770, and the second query 725 may be processed based on the GPU 780. . Based on the resource distribution algorithm, the processing speed of nested queries can be improved and computing resources can be utilized more efficiently.

This resource distribution algorithm can be applied and utilized to both the PostgreSQL engine (750), a SQL engine for general queries, and the extended SQL engine (760) for extended queries, and modeling is also performed using GPU, providing fast modeling results. can be derived.

Figure 8 is a conceptual diagram showing a resource distribution algorithm according to an embodiment of the present invention.

In Figure 8, a method for selecting CPU or GPU when processing a query based on a resource distribution algorithm is disclosed.

Referring to FIG. 8, a method for determining whether to use the CPU 860 or the GPU 850 when processing an extended query 800 and a general query 820 is disclosed.

(1) Extended query (800)

1) Among the expansion queries 800, the first type expansion query 803, which must be processed based on a model that essentially requires the use of the GPU 850 on the expansion engine, can be processed based on the GPU 850.

2) Among the extended queries 800, the second type extended query 806, which can be processed without necessarily using the GPU 850 on the expansion engine, is processed by the CPU 860 or It can be processed based on GPU 850.

CPU execution ability can be determined based on query processing speed, resource usage, and query cost when processing queries based on CPU. GPU execution ability can be determined based on query processing speed, resource usage, and query cost when processing queries based on GPU. The query cost to determine CPU execution ability is determined based on data processing volume (row processed) and CPU operation cost, and the query cost to determine GPU execution ability is determined based on data processing volume (row processed) and GPU operation cost. You can. The faster the query processing speed is, the resource usage is relatively small, and the query cost is relatively small, the higher the execution ability can be judged.

(2) General query

General queries can be processed based on CPU or GPU, considering CPU execution ability and GPU execution ability among computing resources.

More specifically, CPU execution ability and GPU execution ability for extended queries and general queries can be determined by considering the overall cost. The smaller the overall cost, the more advantageous the resource may be.

The total cost may be the sum of the startup cost and run cost.

Startup cost may be a cost incurred before the first tuple is fetched. A tuple is a collection of attribute values related to a given list in a database. For example, the startup cost of an index scan node is the cost of reading an index page to access the first tuple of the target table.

The run cost may be the cost of accessing all tuples.

More specifically, CPU run cost and GPU run cost can be determined as shown in the equation below.

<Equation>

GPU run cost = (gpu_tuple_cost + gpu_operator_cost) x N _tuple + seq_page cost x N _page

CPU run cost = (cpu_tuple_cost + cpu_operator_cost) x N _tuple + seq_page cost x N _page

gpu_tuple_cost is the cost of GPU processing table rows during operation.

gpu_operator_cost is the cost for the GPU to process table tuples as operators or functions.

cpu_tuple_cost is the cost for the CPU to process table rows during operation.

cpu_operator_cost is the cost for the CPU to process table tuples as operators or functions.

N_tuple is the number of table tuples.

seq_page cost is the cost of retrieving a page.

N _page is the number of index pages.

Figure 9 is a conceptual diagram showing a data processing system that processes input multi-query according to an embodiment of the present invention.

FIG. 9 discloses a method of scheduling processing of a plurality of queries included in the input multi-query based on a multi-query scheduler when receiving the input multi-query.

Referring to FIG. 9, the input multi-query 900 may be a query including a plurality of queries. For example, the input multi-query 900 may include a first query, a second query, a third query, and a fourth query. The first query may be SELECT, the second query may be BUILD, and the fourth query may be PRINT.

The multi-query scheduler 920 may include a multi-query analyzer 940 and a query queue 960.

The multi-query analysis unit 920 may analyze a plurality of queries included in the input multi-query 900 and determine the processing order of the plurality of queries.

The query queue 960 may schedule a plurality of queries on the queue by considering the processing order of the plurality of queries determined by the multi-query analysis unit 920.

Queries scheduled on the queue can be delivered to the parser unit. Thereafter, as in the above-mentioned procedure, the query may be parsed by the parser unit, divided into a general query and an extended query, and interpreted and processed through the cloud analyzer unit and the query tree unit.

The cloud analysis unit can determine the engine to process the query and the computing resource (CPU or GPU) to process the query and transfer it to the determined engine (PostgreSQL engine or extension engine) for processing.

Figure 10 is a conceptual diagram showing the operation of a multi-query scheduler according to an embodiment of the present invention.

In Figure 10, a method for scheduling a plurality of queries in a multi-query scheduler is disclosed.

Referring to FIG. 10, the multi-query scheduler 1000 analyzes the dependency score 1020 and computing resource allocation data 1040 for an input query that comes in the form of a multi-query, and calculates the dependency score 1020 and computing resource allocation. Based on the data 1040, the processing order of a plurality of queries included in the input multi-query can be determined.

The dependency score 1020 may be determined through cross-correlation between a plurality of queries included in the input multi-query. For example, if the third query is dependent on results processed based on the first query and the second query, the third query may be a query that has dependency on the first query and the second query. When the fourth query is processed independently, the fourth query may be a query that has no dependency on other queries.

The dependency score 1020 may be determined for a query that can be performed after processing another query, and may have a relatively high value as the dependency on other queries increases.

Computing resource allocation data 1040 may be data about whether the query will be processed based on CPU or GPU processing units, and if the query is processed based on CPU or GPU, computing resources to be allocated.

For example, the multi-query scheduler 1000 may determine whether a query will be processed based on which processing unit, CPU or GPU, based on the resource distribution algorithm described above. In addition, the multi-query scheduler 1000 can obtain data on allocated computing resources when a query is processed based on CPU or GPU based on the method for determining CPU execution ability and GPU execution ability described above. .

CPU execution ability can be determined based on query processing speed, resource usage, and query cost when processing queries based on CPU. GPU execution ability can be determined based on query processing speed, resource usage, and query cost when processing queries based on GPU. The query cost to determine CPU execution ability is determined based on data processing volume (row processed) and CPU operation cost, and the query cost to determine GPU execution ability is determined based on data processing volume (row processed) and GPU operation cost. You can.

Additionally, the dependency score 1020 may be determined to be processed based on different priorities depending on the type of query.

Among queries, special statements that create tables or models, such as CREATE, COPY, BUILD, etc., can be recognized and scores assigned so that these types of queries are executed first. Queries for creating tables or models can be defined as priority processing queries, and even within priority processing queries, separate priorities can be given based on scores.

For example, if multiple queries included in the input multi-query are CREATE, SELECT_1, COPY, SELECT_2, PREDICT, and BUILD, COPY, CREATE, and BUILD may be processed preferentially as priority processing queries.

For example, based on the dependency score 1020, processing priority is determined in the order COPY > CREATE > BUILD > PREDICT > SELECT_1 > SELECT_2, and the query queue can reconfigure the queue for performing queries.

More specifically, the multi-query analysis unit may first check the dependency of a plurality of queries included in the input multi-query and then determine the execution order of each of the plurality of queries.

Thereafter, the GPU execution capability and CPU execution capability for each of the plurality of queries may be determined secondarily by considering the total cost described above. Based on GPU execution ability and CPU execution ability, it is determined whether each of the plurality of queries included in the query group is assigned to the CPU or GPU, and the execution order can be determined so that the lower the overall cost, the faster the query is executed.

More specifically, the cloud analyzer of the multi-query analysis unit analyzes the correlation of multiple queries included in the input multi-query, and preferentially executes a specific query among the multiple queries based on the correlation (dependency) of the multiple queries. You can decide to do it.

Thereafter, the total cost (sum of startup cost and run cost) for each of the plurality of queries may be determined. Additionally, the major resource may be determined to be CPU or GPU based on the total cost information of each of the plurality of queries. The lower the overall cost, the higher the priority. Multiple query groups can be queued to be processed with higher priority, but CPU and GPU can be selectively used as major resources depending on execution ability.

Lastly, by additionally considering the positions of queries included in multiple queries, queries located at the front of the input multi-query can be queued to be processed with higher priority.

For example, it may be assumed that queries 1 to 4 exist.

Query 1 has a small overall cost as the first priority, uses GPU as a major resource, and can include the query written at the very beginning in the input multi-query.

Query 2 is ranked 3rd and can use GPU as a major resource with small overall cost.

Query 3 can use CPU as a major resource with a small overall cost of rank 2.

Query 4 is ranked 4th, has a small overall cost, uses CPU as a major resource, and can include queries written last in the input multi-query. In this case, query groups may be stacked in the queue to be processed in the following order: query 1, query 3, query 2, and query 4.

Figure 11 is a conceptual diagram showing a method of processing an input multi-query according to an embodiment of the present invention.

In Figure 11, an algorithm for determining the execution order of query groups included in an input multi-query is disclosed.

Referring to FIG. 11, after the major resources and total cost of a plurality of query groups are determined by the cloud analyzer of the multi-query analysis unit, the execution order of the query group can be determined by considering the dependency, total cost, and location of the query. .

For example, query 1 (1110) includes a total cost of 3 and PREDICT USING (my_model) as a query statement. Query 2 (1120) includes a total cost of 2 and CREATE TABLE (my_model) as a query statement. Query 3 (1130) has a total cost of 5 and may include BUILD MODEL (my_model) as a query statement.

In this case, my_model used in query 1 (1110) and query 2 (1120) is created and available first in query 3 (1130), so query 1 (1110) and query 2 (1120) are related to query 3 (1130). It has dependencies. Therefore, even though the overall cost of query 3 (1130) is high and its location is last, it can be executed first.

Next to query 3 (1130), query 2 (1120) with a total cost of 1 may be executed considering the total cost, and finally, query 1 (1110) with a total cost of 3 may be executed.

If the total cost of Query 1 (1110) and Query 2 (1120) are the same, considering the location of the query statement, after executing Query 3 (1130), query in the order of Query 1 (1110) and Query 2 (1120). can be executed.

Figure 12 is a conceptual diagram showing a server that provides processing services for structured data and unstructured data according to an embodiment of the present invention.

In Figure 12, a server for providing processing services for structured data and unstructured data to user devices based on different architectures is disclosed.

Referring to FIG. 12, the server 1200 may include an instance 1210 and resources. The instance includes a workspace hub (1270, 1280) and a data processing engine (1220), and resources may include a processing resource (1240), storage (1230), and memory (not shown). Workspace hubs 1270 and 1280 may include at least one workspace 1250.

The workspace 1250 can be directly connected only to the storage 1230 corresponding to the workspace 1250, and the data processing engine 1220 can be connected to all storages 1230. An instance can have a structure directly connected to a resource.

Hereinafter, instances and resources constituting the server 1200 will be described in more detail.

The workspace hubs 1270 and 1280 may include a shared workspace hub 1270 implemented based on a shared architecture and a dedicated workspace hub 1280 implemented based on a dedicated architecture. there is.

The shared workspace hub 1270 and the dedicated workspace hub 1280 may include the workspace 1250. The workspace 1250 may be a space for implementing services based on structured data and unstructured data provided by users. For example, one workspace 1250 may be assigned based on one user identifier. The workspace 1250 may include a lab and a database. The lab is a space for generating queries for services implemented by users, and the database is a logical storage that stores organized data. In other words, the lab may be a user interface for accessing the workspace 1250.

The shared workspace hub 1270 may include multiple workspaces 1250 of multiple users. The shared workspace hub 1270 may share the allocated data processing engine 1220 and the allocated processing resources 1240. Dedicated workspace hub 1280 may include a workspace 1250 for one user. A data processing engine 1220 and a processing resource 1240 for exclusive use by one user may be allocated to the dedicated workspace hub 1280.

For convenience of explanation, only the shared workspace hub 1270 and the dedicated workspace hub 1280 are expressed one by one. However, there may be a plurality of shared workspace hubs 1270 and dedicated workspace hubs 1280, and this architectural structure is also shown in this example. It may be included in the scope of invention rights.

More specifically, the shared workspace hub 1270 may use the allocated data processing engine 1220 and the allocated processing resources 1240. The shared workspace hub 1270 may include a plurality of workspaces 1250. A plurality of workspaces 1250 may share the allocated data processing engine 1220 and the allocated processing resources 1240.

The dedicated workspace hub 1280 includes only one workspace 1250, and one workspace 1250 may exclusively use the allocated data processing engine 1220 and the allocated processing resource 1240.

Accordingly, the user selectively selects the shared workspace 1270 or the dedicated workspace 1280 based on the amount of structured data to be processed, the amount of unstructured data, and the services to be implemented based on the structured data and unstructured data. ) can be created.

The data processing engine 1220 is the data processing system described above in FIGS. 2 to 11 and can process queries for unstructured data and structured data based on PostgreSQL and extended SQL. That is, the data processing engine 1220 can perform the operations of the data processing system described above in FIGS. 2 to 11.

The data processing engine 1220 may create a model for the user through a user request based on the workspace 1250, and the model for the user may be created and stored on the storage 1230.

The user may send a query to request an output value based on the model stored in the user's storage 1230 externally through the data processing engine 1220 based on an application programming interface (API) and the user's unique identification information. The data processing engine 1220 may check the user's workspace 1250 based on the user's unique identification information and deliver output values to the user based on the model stored in the storage 1230.

The data processing engine 1220 is implemented to be accessible to all storage 1230 and can access the storage 1230 based on user unique identification information upon user request and utilize data stored in the storage 1230.

Storage 1230 may be a physical storage for files, models, etc., and one workspace 1250 may be connected to one storage 1230 and may be accessible only when the user is authenticated with unique identification information.

Processing resources 1240 may include GPU, CPU, and memory. As described above, CPU and GPU can be selected when processing queries based on a resource distribution algorithm. Additionally, CPU and GPU can be selected based on a multi-query scheduler and optionally used to process multi-queries.

The workspace 1250 of the dedicated workspace hub 1280 can exclusively use the allocated resources. A plurality of workspaces 1250 of the shared workspace hub 1270 share allocated resources. Accordingly, queries requested by each of the plurality of workspaces 1250 can be processed sequentially.

Figure 13 is a conceptual diagram showing a query processing method of a shared workspace hub according to an embodiment of the present invention.

In Figure 13, a method for processing queries by each of a plurality of workspaces included in a shared workspace hub is disclosed.

Referring to FIG. 13, query processing for individual workspaces included in the shared workspace hub may be processed based on the resource distribution algorithm described above.

However, queries generated by multiple workspaces can be processed sequentially based on the queue 1300/stack. In other words, processing of queries from individual workspaces is performed based on a resource distribution algorithm, and processing of queries from multiple workspaces can be processed sequentially, taking query creation time into account.

Figure 14 is a conceptual diagram showing a resource distribution method according to an embodiment of the present invention.

In Figure 14, a method for allocating processing resources considering the characteristics of the workspace is disclosed.

Referring to FIG. 14, a method for allocating processing resources (CPU, GPU, memory) is disclosed taking into account the characteristics of a workspace hub (dedicated workspace hub, shared workspace hub).

The characteristics of the workspace hub for allocating processing resources may include the volume of queries for structured data (1400), the volume of queries for unstructured data (1410), and the data throughput (1420) required to process the queries. .

The more queries a workspace uses for unstructured data (hereinafter referred to as unstructured queries), the more GPU it uses.

The more a workspace uses queries for structured data (hereinafter referred to as structured queries), the more the workspace uses relatively more CPU.

The more data the workspace needs to process when processing queries, the more memory can be allocated.

According to an embodiment of the present invention, resources for the workspace hub are allocated in consideration of the volume of structured queries for structured data used for each workspace hub, the volume of unstructured queries for unstructured data, and the data throughput required to process the queries. Can be allocated adaptively.

According to an embodiment of the present invention, in the case of a workspace (hereinafter referred to as workspace (dedicated)) included in a dedicated workspace hub, processing resources within a set limit are used without sharing. According to an embodiment of the present invention, the workspace (Dedicated) can adaptively adjust the provision limits of CPU, GPU, and memory for the workspace (Dedicated) based on generated structured query data, unstructured query data, and processing resource usage statistical data, which is statistical information on data throughput. For example, Workspace 1 (dedicated) may require a relatively large amount of CPU, and in this case, unused GPU resource allocations within the limit can be replaced and converted to CPU resources. Workspace 2 (Dedicated) may require a relatively large number of GPUs, and in this case, unused CPU resource allocations within the limit can be replaced and converted to GPU resources for use.

In the case of a workspace (hereinafter referred to as workspace (shared)) included in a shared workspace hub, processing resources are shared and used.

Therefore, in order to share processing resources, processing resources may be allocated based on processing resource usage statistics data of a plurality of workspaces (shared) included in the shared workspace hub. There can be fixed and variable methods for allocating processing resources to a shared workspace hub.

The fixed method sets the processing resources allocated to one shared workspace hub according to the number of workspaces (shares) included.

When the fixed method is used, the workspaces allocated to the shared workspace hub may be determined by considering each of the processing resource usage statistics used in each of the plurality of workspaces. For example, workspaces that use relatively more CPU than GPU and workspaces that use GPU relatively more than CPU can be grouped and assigned to a shared workspace hub to achieve overall balanced processing resources. .

The variable method is a method of allocating processing resources allocated to one shared workspace hub by taking into account the processing resources used in multiple shared workspace hubs. In the variable method, the processing resources used in the multiple shared workspace hubs are determined, and different CPU resources, GPU resources, and memory are allocated to each of the multiple shared workspace hubs by considering the processing resource usage statistics of each of the multiple shared workspace hubs. can be assigned.

The fixed method and the variable method can be used when a stable supply of processing resources is required considering the usage of total processing resources, and the variable method can be used when a more efficient supply of processing resources is needed. For example, for certain time zones (business hours), processing resources are provided in a fixed manner, and for certain time zones (non-business hours), processing resources are provided in a variable manner to maximize unused processing resources. It can be used efficiently.

On a larger scale, workspace hubs and workspaces may be allocated on multiple servers based on processing resource usage statistics on a server-by-server basis.

Hereinafter, the unstructured data embedding method, unstructured data binarization method, server/workspace backup method, server/workspace migration method, and caching method disclosed may be performed in a data processing system, and the data processing system may be performed as described above. It may be a concept that includes a server and an external device (external processing server).

Figure 15 is a conceptual diagram showing an unstructured embedding method for unstructured data according to an embodiment of the present invention.

In Figure 15, an unstructured embedding method for performing embedding on unstructured data is disclosed.

Referring to Figure 15, unstructured data (images, video, audio, etc.) has large dimensions, and when stored as a file itself (pixel unit, etc.), the size is large. Therefore, the data processing system of the present invention can process unstructured data by converting it into embeddings. Embedding identifies the characteristics of unstructured data such as images, text, and video and converts them into vector form. Data size can be reduced by storing the characteristics of the file rather than the file itself. This is based on extended SQL in the data processing system. This can be of great help in producing quick results.

Unstructured data can be extracted as embedding values on an artificial intelligence model.

There are various artificial intelligence models that extract embedding values from unstructured data. For example, there may be an artificial intelligence model that extracts the color characteristics of an image, while there may be an artificial intelligence model that extracts the actions a person is doing in the image, and there may also be a model that extracts the shape itself.

According to an embodiment of the present invention, the CONVERT function can embed unstructured data. By changing the model name (or artificial intelligence model name) in the convert function, the desired characteristics can be extracted as an embedding value, and the artificial intelligence model that determines the embedding value can be added to the data processing system when a new model is developed and commercialized.

According to an embodiment of the present invention, the embedding value obtained using the CONVERT function can be used in an unstructured search together with the SEARCH function. The SEARCH function can be used with the artificial intelligence model provided by the embedding and data processing system to quickly search for unstructured data.

In other words, the data processing system according to an embodiment of the present invention can accelerate the process of embedding unstructured data when using CONVERT and SEARCH syntax and the process of performing unstructured search using embedding using GPU. .

Therefore, not only can processing of unstructured data and structured data be performed with one system/operation platform and one database, but also faster results for queries can be derived compared to existing systems.

Figure 16 is a conceptual diagram showing an unstructured embedding method for unstructured data according to an embodiment of the present invention.

In Figure 16, a process of embedding unstructured data when using CONVERT and SEARCH syntax in a data processing system according to an embodiment of the present invention and a process of performing an unstructured search using embedding are disclosed.

Referring to FIG. 16, unstructured data 1600 can be converted into embedding values through artificial intelligence model 1 (1610) to artificial intelligence model n using a convert function. For example, the unstructured data 1600 is an image, and the embedding value may be a value extracted based on the feature map for each image of artificial intelligence model 1 (1610) to artificial intelligence model n.

For example, the unstructured data 1600 may be a specific image, artificial intelligence model 1 (1610) converts information about the number of people in the image into a first embedding value (1615), and artificial intelligence model 2 (1620) converts information about the number of people in the image into a first embedding value (1615). ) converts information about the place constituting the image into a second embedding value (1625), and artificial intelligence model 3 (1630) can convert information about the color of the image into a third embedding value (1635).

In the same manner as above, the first to nth embedding values 1615 for the image may be stored in the database 1650.

Next, the user may send a query to the data processing system to inquire about the number of people included in the image using a search syntax based on the user device, and the data processing system may query the data processing system based on the first embedding value 1615. The response can be delivered to the user.

A group containing at least one artificial intelligence model that changes unstructured data into an embedding value can be expressed by the term embedding artificial intelligence model group, and an embedding artificial intelligence model group converts one unstructured data into various embedding values depending on the purpose. It can change.

Figure 17 is a conceptual diagram showing the results of using the convert syntax and search syntax according to an embodiment of the present invention.

Figure 17 (a) shows the result of embedding using the convert syntax.

Referring to (a) of FIG. 17, the result of embedding text for a movie review is disclosed. There are multiple reviews for a movie, and each of the multiple reviews can be embedded based on the feature map of the artificial intelligence model and converted into a vector value.

Figure 17(b) shows the results of performing a search using a search syntax.

Referring to (b) of FIG. 17, 10 other movie reviews with the highest similarity to the movie review ('This movie was my favorite movie of all time') are prioritized. A movie review ('This movie was my favorite movie of all time') is converted into an embedding value, and the 10 priorities with the most similar embedding value to the movie review can be provided based on the similarity score value.

That is, according to an embodiment of the present invention, the method of embedding unstructured data in a data processing system includes steps in which an embedding artificial intelligence model group of the data processing system changes unstructured data into an embedding value and a database of the data processing system stores the embedding value. May include steps.

The embedding artificial intelligence model group includes at least one different artificial intelligence model that generates the necessary embedding value, and the embedding value may be determined based on the feature map of the at least one different artificial intelligence model. Additionally, the embedding value is generated by a convert statement based on the extended SQL engine, and the embedding value can be searched by a search statement based on the extended SQL engine.

Figure 18 is a conceptual diagram showing a method for binarizing an embedding value according to an embodiment of the present invention.

In Figure 18, a method for additionally increasing processing speed through binarization of additional embedding values is disclosed.

Referring to FIG. 18, the conversion process in a data processing system requires a considerable amount of time and computing resources, so it needs to be stored in a database for reuse.

Since the embedding itself is a dense array of real numbers or list of lists of floats, storing it in a database would take a lot of time and resources, so as an additional option, the embedding value must be formatted as a byte array. You can save it.

By storing the embedding value as a byte array, the time and resources required to store the data table in the database can be reduced.

In other words, the converted and generated embedding value can be converted once again into a binary value that is a byte array and stored in the database. When searching, the binary value, which is a byte array loaded into the database, can be converted back into an embedding value and a search operation can be performed.

Figure 18 shows a process of embedding unstructured data in a data processing system according to an embodiment of the present invention, a process of generating a binary value by binarizing the embedding value, a process of storing the binary value, and converting the binary value into an embedding value. The process of outputting the result based on the converted embedding value is initiated.

Referring to FIG. 18, unstructured data can be converted into embedding values through artificial intelligence model 1 to artificial intelligence model n using the convert function. For example, the unstructured data is an image, and the embedding value may be information about the feature map for each image of artificial intelligence model 1 to artificial intelligence model n.

For example, unstructured data may be a specific image, artificial intelligence model 1 converts information about the number of people in the image into a first embedding value, and artificial intelligence model 2 provides information about the places that make up the image. 2 embedding value, and artificial intelligence model 3 can convert information about the color of the image into a third embedding value.

The binary module may convert the first embedding value into a first binary value, the second embedding value into a second binary value, and the third embedding value into a third binary value.

The first to nth binary values converted in the above manner may be stored in the database.

Next, the user may transmit a query inquiring about the number of people included in the image through a search phrase through the user device to the data processing system, and the binary module of the data processing system converts the first binary value into the first embedding value. It is converted, and a response to the query can be delivered to the user based on the first embedding value.

Figure 19 is an example of binarization according to an embodiment of the present invention.

Referring to FIG. 19, the binary module can change the embedding value into a binary value.

The binary module can change the movie review text into an embedding value, and then change the embedding value back into a binary value.

That is, according to an embodiment of the present invention, the method of binarizing unstructured data in a data processing system includes the steps of the binary module of the data processing system changing the embedding value for the unstructured data into a binary value, and the database of the data processing system converting the embedding value into a binary value. It may include the step of storing.

The embedding artificial intelligence model group of the data processing system may further include a step of changing unstructured data into embedding values. The embedding artificial intelligence model group includes at least one different artificial intelligence model that generates the necessary embedding value, and the embedding value may be determined based on the feature map of the at least one different artificial intelligence model.

Figure 20 discloses a method for storing an embedding value or a binary value according to an embodiment of the present invention.

In Figure 20, a method for a data processing system to process unstructured data into embedding values or binary values is disclosed.

Referring to FIG. 20, the data processing system primarily stores unstructured data as binary values, and considers the frequency of use of binary values of unstructured data and divides the unstructured data into first type unstructured data (2010) and second type unstructured data. (2020) and can be stored in the database in different formats.

Unstructured data initially stored as binary values can be expressed in terms of default unstructured data (2000).

The default unstructured data 2000 may be data before conversion into the first type unstructured data 2010 or the second type unstructured data 2020, and may be data before accumulating statistics on the frequency of use of binary values for a critical time. The default unstructured data 2000 may be converted into first type unstructured data 2010 or second type unstructured data 2020 based on the frequency of use of the binary value during the critical time.

The first type unstructured data 2010 may be stored as an embedding value, and the second type unstructured data 2020 may be stored as a binary value.

The first type of unstructured data (2010) may be data with a relatively high frequency of use. The first type of unstructured data (2010) may be unstructured data that has changed from a binary value to an embedding value more than a threshold number of times within a set time for various reasons such as learning, prediction, and search.

The second type of unstructured data (2020) may be data with a relatively low frequency of use. The first type of unstructured data (2010) may be unstructured data that has changed less than a threshold number of times within a threshold time from a binary value to an embedding value within a set time for various reasons such as learning, prediction, or search.

The frequency of use for each cycle of the first type unstructured data (2010) and the second type unstructured data (2020) may be measured again for each specific cycle. Considering the frequency of use by cycle, the first type unstructured data 2010 may remain as the first type unstructured data 2010 or may be changed into the second type unstructured data 2020. Likewise, considering the frequency of use by cycle, the second type unstructured data 2020 may remain as the second type unstructured data 2020 or may be changed to the first type unstructured data 2010.

A specific cycle is set based on the point in time when the switching frequency is highest, taking into account the frequency with which users switch between the first type of unstructured data (2010) and the second type of unstructured data (2020), and changes are determined according to the cycle setting. The threshold frequency of use may vary. The specific cycle may be adjusted to additionally consider the frequency of data utilization in the user's workspace.

Figure 21 is a conceptual diagram showing a workspace backup method according to an embodiment of the present invention.

In Figure 21, a method for a user to back up his or her own workspace on a data processing system is disclosed.

Referring to FIG. 21, the user can perform a backup of the workspace 2160. The backup of the workspace (2160) refers to storing both the user's user data at the time of backup and the user metadata about the user's workspace information. If the user accidentally deletes or loses his or her data, the backup file that was backed up is stored. It can be restored using

According to an embodiment of the present invention, a workspace backup controller 2120 may exist within the master controller 2110 within the master server 2100. The workspace backup controller 2120 may transmit the backup request to the server controller 2130, which controls the workspace 2160 corresponding to the user, based on the user's request.

The server controller 2130 may include a workspace storage checker 2140 and a storage difference hasher 2150.

The workspace storage checker 2140 can check the existence of workspace backup data in the workspace backup storage 2170. If existing workspace backup data exists in the workspace backup storage 2170, user data and user metadata on the storage corresponding to the user's workspace 2160 can be compared with the existing workspace backup data. User data and user metadata currently on storage may be expressed in terms of target workspace backup data. That is, the workspace storage checker 2140 can perform a comparison between existing workspace backup data and target workspace backup data. If there is no existing workspace backup data in the workspace backup storage 2170, the target workspace backup data can be backed up to the workspace backup storage 2170.

The storage difference hasher 2150 may be implemented to perform hashing on changed data. The storage difference hasher (2150) hashes only the changed data between the existing workspace backup data and the target workspace backup data, and the workspace backup storage (2170) updates the existing workspace backup data based on the hash value. You can do it.

In this way, user data and user metadata can be backed up and stored in workspace backup storage 2170.

Figure 22 is a flowchart showing a workspace backup method according to an embodiment of the present invention.

In Figure 22, a method for a user to back up his or her own workspace on a data processing system is disclosed.

A workspace backup controller within the master controller within the master server of the data processing system may receive the user backup request (step S2200).

The workspace backup controller may forward the user backup request to the server controller based on the user backup request (step S2210).

The workspace storage checker of the server controller may check the existence of existing workspace backup data in the workspace backup storage (step S2220).

If existing workspace backup data exists in the workspace backup storage, the target workspace backup data of the user's workspace and the existing workspace backup data of the user's workspace can be compared (step S2230).

If existing workspace backup data does not exist in the workspace backup storage, the target workspace backup data may be backed up to the workspace backup storage (step S2240).

The storage difference hasher generates a hash value based on the changed data between the existing workspace backup data and the target workspace backup data (step S2250).

Workspace backup storage can update existing workspace backup data based on the hash value (step S2260).

That is, the workspace backup method in the data processing system according to the embodiment of the present invention includes the steps of the data processing system determining target workspace backup data and existing workspace backup data, and the data processing system determining target workspace backup data and existing workspace backup data. It may include a step of generating a hash value based on comparison of space backup data and a step of the data processing system backing up the existing workspace to the target workspace based on the hash value. The existing workspace is the workspace that the user previously created, and the target workspace is the workspace where the user will back up the existing workspace.

The workspace storage checker of the server controller of the data processing system checks the existence of existing workspace backup data in the workspace backup storage, and if existing workspace backup data exists in the workspace backup storage, the target workspace backup data and the existing workspace backup data are stored in the workspace backup storage. You can compare workspace backup data.

Figure 23 is a conceptual diagram showing a server backup method according to an embodiment of the present invention.

In Figure 23, a method for a user to back up a server on a data processing system is disclosed.

Referring to FIG. 23, server backup is an automatic backup that backs up all server-related data. Server backup is a backup performed at the service operation level and can be performed periodically. It can back up not only workspace-level user data, but also server data and server metadata related to the entire workspace related to the server. Server backups are for when there is a problem with the server itself or when the server is not working due to a specific problem.

A server backup controller 2320 may exist within the master controller 2310 of the master server 2300. The server backup controller 2320 may periodically receive a backup request for the server and transmit the backup request to the server controller 2330 of the server.

The server controller 2330 may include a server storage checker 2340 and a storage difference hasher 2350.

The server storage checker 2340 checks whether there is existing server backup data in the server backup storage 2360 and compares it with the server data and server metadata of the current server to check whether there is any added/changed/deleted data. If there is changed server data or changed server metadata, the existing server backup data in the server backup storage 2360 can be updated by generating a hash value based on the changed data.

Server data and server metadata on the current server may be expressed in terms of target server backup data. That is, the server storage checker 2340 can perform a comparison between existing server backup data and target server backup data. If existing server backup data does not exist in the server backup storage 2360, the target server backup data may be backed up to the server backup storage 2360.

The storage difference hasher 2350 may be implemented to perform hashing on changed data. If there is changed data between the existing server backup data and the target server backup data, the storage difference hasher 2350 generates a hash value by performing hashing based on the changed data, and the server backup storage 2360 generates a hash value based on the hash value. Server backup data can be updated.

In this way, server data and server metadata can be backed up and stored in server backup storage 2360.

Figure 24 is a flowchart showing a server backup method according to an embodiment of the present invention.

24 discloses a method for performing server backup on a data processing system.

A server backup controller within the master controller within the master server of the data processing system may receive the server backup request (step S2400).

The server backup controller may forward the server backup request to the server controller based on the server backup request (step S2410).

The server storage checker of the server controller may check the existence of existing server backup data in the server backup storage (step S2420).

If existing server backup data exists in the server backup storage, the target server backup data of the server and the existing server backup data of the user's workspace can be compared (step S2430).

If existing server backup data does not exist in the server backup storage, the target server backup data may be backed up to the server backup storage (step S2440).

The storage difference hasher may generate a hash value determined based on the changed data between the existing server backup data and the target server backup data (step S2450).

Server backup storage can update existing server backup data based on the hash value (step S2460).

That is, the server backup method according to an embodiment of the present invention includes the steps of the data processing system determining target server backup data and existing server backup data, and the data processing system hashing the target server backup data and existing server backup data based on comparison. It may include a step of generating a value and a step of the data processing system backing up the existing server to the target server based on the hash value.

The server storage checker of the server controller of the data processing system can check the existence of existing server backup data in the server backup storage, and compare the target server backup data with the existing server backup data if existing server backup data exists in the server backup storage. there is. The hash value may be determined based on the changed data between the existing server backup data and the target server backup data by the storage difference hasher.

Figure 25 is a conceptual diagram showing a workspace migration method according to an embodiment of the present invention.

In Figure 25, a method of performing migration for a workspace is disclosed.

Referring to Figure 25, migration refers to the act of moving a workspace from an existing server to a new server.

During migration, the storage distributed to the user or the computing resources (CPU, GPU, memory) distributed to the user may be updated according to the user migration specification.

When migrating, the user requests storage size upgrade, computing resource upgrade, etc. as user migration specifications, and the user's new workspace can be defined by considering these user migration specifications.

Hereinafter, the server containing the existing user's workspace may be expressed as the existing server 2503, and the server to which the workspace will be migrated may be expressed as the migration server 2506.

The workspace migration controller 2520 of the master controller 2510 of the master server 2500 may receive a migration request from the user.

The workspace migration controller 2520 may receive a user's migration request and query the migration server 2506 that can transfer the user's workspace based on the migration request. The workspace migration controller 2520 can determine the migration server 2506 and deliver the migration request to the existing server 2503 and the migration server 2506. The migration request may include information about user migration specifications.

The workspace migration controller 2520 transmits the first migration request requesting migration to the migration server 2506 to the existing server 2503 and requests to receive data according to the migration from the existing server 2503. A second migration request may be transmitted to the migration server 2506.

The server controller 2530 of the existing server 2503 may include a workspace migration sender 2540 and a storage hasher encoder 2550. The workspace migration sender 2540 may be implemented to transmit backup data of user migration data (user data, user metadata, etc.).

The workspace migration sender (2540) can back up the migration data of the existing server and deliver it to the storage hasher encoder (2550).

The storage hasher encoder 2550 may hash the migration data and transmit the hashed migration data to temporary migration storage.

Temporary migration storage 2560 is a storage that stores hashed migration data of the existing server 2503, and can transmit hashed migration data to the migration server 2506. Migration data stored in the temporary migration storage 2560 may be deleted at the same time the migration process is completed.

The server controller 2530 of the migration server 2506 includes a workspace migration receiver 2570, a storage hasher decoder 2580, and a workspace spawner 2590. can do.

The workspace migration receiver 2570 may receive hashed migration data from the temporary migration storage 2560, and the hashed migration data may be transmitted to the storage hasher decoder 2580.

The storage hasher decoder 2580 may generate migration data by decoding the hashed migration data. Additionally, the storage hasher decoder 2580 may generate storage and metadata related to the workspace on the migration server 2506 by considering user migration specifications.

The workspace spawner 2590 can create a new workspace on the migration server 2506 based on storage and metadata related to the created workspace.

When a new workspace is created on the migration server 2506, migration data stored in the temporary migration storage 2560 may be deleted.

Figure 26 is a flowchart showing a workspace migration method according to an embodiment of the present invention.

In Figure 26, a method of performing migration for a workspace is disclosed.

Referring to FIG. 26, the workspace migration controller of the master controller of the master server receives a migration request from the user device (step S2600).

The workspace migration controller receives the user's migration request and determines a migration server that can transfer the user's workspace based on the migration request (step S2610).

The workspace migration controller may forward the migration request to the existing server and the migration server (step S2620).

The migration request may include information about user migration specifications. The workspace migration controller may transmit a first migration request requesting to perform migration to the migration server to the existing server, and transmit a second migration request requesting to receive migration-related data from the existing server to the migration server. .

The workspace migration sender backs up the migration data of the existing server and delivers it to the storage hasher encoder (step S2630).

The storage hasher encoder hashes the migration data and transmits the hashed migration data to the temporary migration storage (step S2640).

The workspace migration receiver receives hashed migration data from the temporary migration storage and delivers the hashed migration data to the storage hasher decoder (step S2650).

The storage hasher decoder decodes the hashed migration data to generate migration data, and generates storage and metadata related to the workspace on the migration server by considering the user migration specifications (step S2660).

The workspace spawner creates a new workspace on the migration server based on storage and metadata related to the created workspace (step S2670).

That is, the workspace migration method in the data processing system according to an embodiment of the present invention includes the steps of the data processing system backing up migration data of the first workspace located in the existing server and transferring the migration data to the migration server, the data processing system It may include creating a second workspace on the migration server based on migration data.

The step of transferring migration data to the migration server is where the workspace migration sender backs up the migration data from the existing server and delivers it to the storage hasher encoder. The storage hash encoder hashes the migration data and transfers the hashed migration to the temporary migration storage. It may include a step of transmitting data and a step of the temporary migration storage transmitting hashed migration data from the existing server to the migration server.

In the step of creating a second workspace, the workspace migration receiver receives hashed migration data from the temporary migration storage, forwards the hashed migration data to the storage hasher decoder, and the storage hasher decoder performs the hashed migration data. A step of decoding data to generate migration data, a storage hasher decoder generating storage and metadata related to the workspace on the migration server considering the user migration specification, and a step of generating storage and metadata related to the workspace by the workspace spawner. It may include the step of creating a second workspace on the migration server based on .

Figure 27 is a conceptual diagram showing a server migration method according to an embodiment of the present invention.

In Figure 27, a server migration method for migrating the server itself is disclosed.

Referring to FIG. 27, server migration is the act of moving the existing server 2703 to a new migration server 2706. For example, it may be necessary to move the server itself to a new server that changes the size of the storage and computing resources of the existing server 2703. In these cases, server migration may be performed. Hereinafter, for convenience of explanation, the new migrated server may be expressed as a migration server 2706.

The master controller 2710 of the master server 2700 may include a server status monitor 2720 and a server migration controller 2730.

Server status monitor 2720 may be implemented to monitor server status. The server status monitor 2720 may communicate with a status checker 2750 located in the server controller 2740 based on scheduling. The status checker 2750 may check the status of the storage and/or resources of the existing server 2703. Server status monitor 2720 may receive storage status information and resource status information. The server status monitor 2720 may transmit a server migration request to the server migration controller 2730 when storage and/or resources are insufficient.

The server migration request may include a server migration specification, and the server migration specification may include information about specifications (eg, storage, computing resources) requested for the migration server 2706.

The server migration controller 2730 may determine the migration server 2706 to migrate the existing server to based on the server migration request. The server migration controller 2730 can search for a new migration server 2706 that can be migrated based on the server migration request and forward the request.

The server migration controller 2730 may transmit a server migration request to the server controller 2740.

The server controller 2740 may include a status checker 2750, a server migration sender 2760, and a storage hash encoder 2770.

The server migration sender 2760 can generate server migration data through backup of the server. The server migration sender 2760 may transmit server migration data to the storage hash encoder 2770.

The storage hash encoder 2770 may hash the server migration data and transmit the hashed server migration data to the temporary migration storage 2780.

Temporary migration storage 2780 can store hashed server migration data.

Temporary migration storage 2780 may transmit hashed migration data to the server migration receiver 2765 of the migration server 2706.

The hashed migration data may be decoded through the storage hash decoder 2775 and converted into migration data.

The storage hash decoder 2775 may generate storage and metadata to be applied to the migrating server (or storage and metadata related to all workspaces on the existing server) based on the server migration specification.

The workspace spawner 2780 can create all workspaces existing in the existing server on the migration server 2706 based on storage and metadata related to the server.

Temporary migration storage 2780 can delete hashed server migration data.

Figure 28 is a flowchart showing a server migration method according to an embodiment of the present invention.

In Figure 28, a method of performing migration to a server is disclosed.

Referring to Figure 28, if the server status monitor is low on storage and/or resources, a server migration request may be transmitted to the server migration controller (step S2800).

The server migration controller may determine a migration server to migrate the existing server to based on the server migration request (step S2810).

The server migration controller may transmit a server migration request to the server controller (step S2820).

The server migration sender creates server migration data through a backup of the existing server (step S2830).

The server migration sender transmits the server migration data to the storage hash encoder (step S2840).

A storage hash encoder hashes the server migration data (step S2850).

The storage hash encoder transmits the hashed server migration data to the temporary migration storage (step S2855).

Temporary migration storage stores hashed server migration data (step S2860).

Temporary migration storage transmits hashed migration data to the server migration receiver of the migration server (step S2865).

The hashed migration data is decoded through a storage hash decoder and converted into migration data (step S2870).

The storage hash decoder generates storage and metadata to be applied to the migration server (or storage and metadata related to all workspaces on the existing server) based on the server migration specification (step S2875).

The workspace spawner creates all workspaces existing in the existing server on the migration server based on storage and metadata related to the server (step S2880).

Temporary migration storage 2780 can delete hashed server migration data.

That is, the server migration method in the data processing system according to the embodiment of the present invention includes the steps of the data processing system backing up the migration data of the existing server and transmitting the migration data to the migration server, and the data processing system backing up the migration data of the existing server and transferring the migration data to the migration server based on the migration data. This may include the step of creating all workspaces existing on the existing server.

The steps for transmitting migration data to the migration server are: the server migration sender creates server migration data through a backup of the existing server, the server migration sender passes the server migration data to the storage hash encoder, and the storage hash encoder sends the server migration data to the server. Hashing the migration data, the storage hash encoder transmitting the hashed server migration data to the temporary migration storage, and the temporal migration storage transmitting the hashed migration data to the server migration receiver of the migration server. It can be included.

In addition, the step of creating all workspaces existing in the existing server includes the storage hash decoder decoding the hashed migration data and converting it into migration data, the storage hash decoder determining the storage and storage to be applied to the migration server based on the server migration specification. It may include a step of creating metadata and a step of the workspace spawner creating all workspaces existing on the existing server on the migration server based on storage and metadata.

Figure 29 is a conceptual diagram showing an unstructured model caching method according to an embodiment of the present invention.

In Figure 29, a method for caching an unstructured model is disclosed.

Referring to FIG. 29, cache 2950 is a high-speed data storage layer that stores subsets of data that are generally transient in nature. Accordingly, when data resides in cache 2950, requests can be processed more quickly than when accessing the data's primary storage location.

Caching refers to uploading data onto the cache 2950 and processing the data. When caching is used, data on cache 2950 that has previously been retrieved or computed can be efficiently reused. Caching can be used to more quickly request and process functions or data that are typically computationally expensive or frequently called.

The data processing system of the present invention can be used by uploading an artificial intelligence model to the cache 2950 through a technology for caching the artificial intelligence model and a caching algorithm for determining which artificial intelligence model to cache.

Caching of artificial intelligence models can have the following effects. For example, the computing resources and time required to load an artificial intelligence model can be reduced. Additionally, the results of the artificial intelligence model can be output more quickly through caching of the artificial intelligence model.

According to an embodiment of the present invention, caching of the artificial intelligence model is carried out using clauses 2900 (e.g., SEARCH, CONVERT, PREDICT, EVALUATE, etc.) inside the SQL engine (e.g., extended SQL engine). It is built into the back. Therefore, caching of an algorithm-based artificial intelligence model can be performed without a separate procedure for the user to trigger and cache.

The following algorithm can be used for caching of artificial intelligence engines.

For example, the first caching algorithm may be used as the default caching algorithm. The first caching algorithm is an algorithm that provides artificial intelligence models in the cache in the order of those that have not been used recently. When the cache is full, the oldest artificial intelligence model is removed, and space can be created to load a new artificial intelligence model. In other words, the first caching algorithm allows frequently used artificial intelligence models to be accessed frequently, so performance can be improved as artificial intelligence models that frequently input or output data are identified.

In an embodiment of the present invention, a first caching algorithm is used so that the artificial intelligence model is uploaded to the cache so that recently used artificial intelligence models can be loaded on the cache, and the more recently unused artificial intelligence models are added, the more they are added. A load on the cache of an artificial intelligence model can be deleted from the cache when necessary.

Resources are used for caching of artificial intelligence models according to an embodiment of the present invention, and for efficient management of resources, computing resources (CPU, GPU, memory) distributed based on the above-described workspace creation and workspace structure are taken into consideration. The caching algorithm of the most efficient artificial intelligence model can be selected.

For caching of artificial intelligence models, the number of artificial intelligence models cached in the cache and the caching algorithm used during caching can be variously considered. Additionally, the number of artificial intelligence models that can be stored in the cache will inevitably vary depending on the computing resources allocated to the user. Accordingly, various caching algorithms other than the first caching algorithm may be used for each user to load the artificial intelligence model on the cache.

As another caching algorithm that can be used in the present invention, the caching algorithm below can be used.

The second caching algorithm may additionally assign a time stamp to the artificial intelligence model loaded on the cache and delete the artificial intelligence model from the cache by additionally considering the time the artificial intelligence model was loaded on the cache. The second caching algorithm can manage the cache by deleting the most recently unused artificial intelligence model among artificial intelligence models that have not been used for a specific period of time. A secondary caching algorithm can be used to make the most of cache space by more aggressively removing older items. That is, in the second caching algorithm, the last accessed artificial intelligence model is deleted after the set threshold time, whereas in the first caching algorithm, the algorithm loads artificial intelligence models up to a specified number, not time, into the cache, so the next model The last accessed AI model may remain in the cache until it is loaded into this cache.

The third caching algorithm tracks the access status of the artificial intelligence model based on the bitmask, and the oldest accessed artificial intelligence model can be deleted. A bitmask is a data structure used to track the access status of all models stored in a cache line. The bit mask has one or more bits for each artificial intelligence model, and the position of the bit corresponds to the position of each artificial intelligence model. Each bit represents the access status of the model and is set to 0 or 1. In other words, the access status of the model is tracked using a bitmask for each artificial intelligence model in the cache line, and can be used to find a recently accessed model.

The fourth caching algorithm is an algorithm that maintains the used artificial intelligence model in the cache, and when the cache is full and a new artificial intelligence model is added, the artificial intelligence model most recently added to the cache is replaced or deleted. Artificial intelligence models are not deleted from the cache until the specified number of artificial intelligence models are loaded into the cache, but if the specified number is exceeded, the most recently added artificial intelligence model may be deleted from the cache.

That is, the unstructured model caching method in the data processing system according to the embodiment of the present invention includes the steps of the data processing system caching a plurality of artificial intelligence models in the cache, and the data processing system selecting one of the plurality of artificial intelligence models based on the caching algorithm. It may include replacing at least one artificial intelligence model.

Figure 30 is a conceptual diagram showing a first caching algorithm according to an embodiment of the present invention.

In Figure 30, replacement of the artificial intelligence engine is initiated based on the first caching algorithm.

Referring to Figure 30, the first caching algorithm may sequentially load artificial intelligence model A, artificial intelligence model B, artificial intelligence model C, and artificial intelligence model D in consideration of the artificial intelligence model used by the user on the cache. .

In addition, artificial intelligence model E must be added to the cache, and if the most recently unused artificial intelligence model among the artificial intelligence models loaded on the cache is artificial intelligence model A, artificial intelligence model A is deleted from the cache, and the corresponding Artificial intelligence model E can be added in its place.

Afterwards, artificial intelligence model F needs to be additionally loaded on the cache, and if the most recently unused artificial intelligence model among the artificial intelligence models loaded on the cache is artificial intelligence model D, the artificial intelligence model D can be deleted, and artificial intelligence model F can be loaded into the cache.

Figure 31 is a conceptual diagram showing an unstructured model caching method according to an embodiment of the present invention.

In Figure 31, a method for caching an unstructured model is disclosed.

Referring to FIG. 31, as described above, the workspace hub may include a shared workspace hub implemented based on a shared architecture and a dedicated workspace hub implemented based on a dedicated architecture. . Shared workspace hubs and dedicated workspace hubs can contain workspaces. The shared workspace hub may include a plurality of workspaces of a plurality of users, and the shared workspace hub may share an allocated data processing engine and allocated processing resources. A dedicated workspace hub includes a workspace for one user, and a data processing engine and processing resources for exclusive use by one user may be allocated to the dedicated workspace hub.

In an embodiment of the present invention, the default cache algorithm may be set differently depending on the characteristics of the workspace.

In a shared workspace hub (or shared workspace), the TLRU cache algorithm can be used.

The TLRU cache algorithm can assign a timestamp to the artificial intelligence model, which is a cache object loaded into the cache. By assigning a timestamp, not only the least used artificial intelligence models are deleted from the cache, but also models whose assigned time has elapsed can also be deleted from the cache.

In other words, since multiple users can share the same computing resources in a shared workspace, artificial intelligence models on the cache can be deleted more efficiently through this TLRU cache algorithm.

In a dedicated workspace hub (or dedicated workspace), the LRU cache algorithm can be used as the default cache algorithm. In a dedicated workspace, the LRU cache algorithm can be used as the default cache algorithm. In a dedicated workspace, the cache algorithm can be changed and applied based on the user's model usage data and model usage log.

The embodiments according to the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. A hardware device can be converted into one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described in terms of specific details, such as specific components, and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. Anyone with ordinary knowledge in the technical field to which the invention pertains can make various modifications and changes from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

Claims (6)

The embedding method of unstructured data in a data processing system is:
An embedding artificial intelligence model group of a data processing system converts the unstructured data into an embedding value; and
A database of the data processing system stores the embedding value,
The data processing system includes a workspace,
The workspace processes structured and unstructured data based on nested queries,
The unstructured data is processed based on unstructured data processing queries,
The structured data is processed based on structured data processing queries,
The nested query is a query that combines a first query for the unstructured data and a second query for the structured data,
The unstructured data processing query is a query for processing only the unstructured data,
The structured data processing query is a query for processing only the structured data. According to paragraph 1,
The data processing system performs artificial intelligence engine modeling using the unstructured data and the structured data on a single platform without a separate batch process based on extended structured query language (SQL),
Processing of the nested query is,
The data processing system performing processing on the unstructured data based on an extended SQL engine that processes the extended SQL; and
The data processing system is performed based on the step of performing processing on the structured data based on a general SQL engine that processes Postgre SQL (structured query language),
The extended SQL is a query language defined to process the unstructured data,
A method characterized in that the Postgre SQL is a query language defined to process the structured data. According to paragraph 1,
The embedding artificial intelligence model group includes at least one different artificial intelligence model that generates the necessary embedding values,
The embedding value is determined based on the feature map of the at least one different artificial intelligence model,
The embedding value is generated by a convert statement based on an extended SQL engine,
A method characterized in that the embedding value is searched by a search statement based on an extended SQL engine. A data processing system that performs embedding of unstructured data,
The embedding artificial intelligence model group changes the unstructured data into embedding values,
A database is implemented to store the embedding value,
The data processing system includes a workspace,
The workspace processes structured and unstructured data based on nested queries,
The unstructured data is processed based on unstructured data processing queries,
The structured data is processed based on structured data processing queries,
The nested query is a query that combines a first query for the unstructured data and a second query for the structured data,
The unstructured data processing query is a query for processing only the unstructured data,
A data processing system, characterized in that the structured data processing query is a query for processing only the structured data. According to clause 4,
The data processing system performs artificial intelligence engine modeling using the unstructured data and the structured data on one platform without a separate batch process based on extended structured query language (SQL),
Processing of the nested query is,
The data processing system performing processing on the unstructured data based on an extended SQL engine that processes the extended SQL; and
The data processing system is performed based on the step of performing processing on the structured data based on a general SQL engine that processes Postgre SQL (structured query language),
The extended SQL is a query language defined to process the unstructured data,
The Postgre SQL is a data processing system characterized in that it is a query language defined to process the structured data. According to paragraph 4,
The embedding artificial intelligence model group includes at least one different artificial intelligence model that generates the necessary embedding values,
The embedding value is determined based on the feature map of the at least one different artificial intelligence model,
The embedding value is generated by a convert statement based on an extended SQL engine,
A data processing system wherein the embedding value is searched by a search statement based on an extended SQL engine.

Images