KR102530254B1 - Npu adaptable network platform for artificial intelligent contents service - Google Patents

Abstract

Disclosed is a user terminal equipped with an NPU adaptive edge framework supporting AI content installation and execution according to an embodiment of the present invention. In the user terminal according to an embodiment of the present invention, the NPU adaptive edge framework includes an application manager for managing AI applications, a content installation manager for downloading and installing AI contents from a content server, and executing the AI contents. and an OS platform including an AI content edge engine, an NPU driver, and a sensor driver for accelerating the first NPU provided in the user terminal.

Description

NPU Adaptable Network Platform for AI Content Service {NPU ADAPTABLE NETWORK PLATFORM FOR ARTIFICIAL INTELLIGENT CONTENTS SERVICE}

The present invention relates to a network platform for AI content service, and more particularly, to an NPU adaptive network platform for AI content distribution, installation, and execution.

A Neural Processing Unit (NPU) is a processor that learns and processes information like our brain, similar to human brain neurons that are connected to countless nerve cells through synapses to send and receive signals and perform tasks at the same time. Because it proceeds with work, it is called an AI chip because it is grafted with artificial intelligence that can learn and judge itself. With the advent of these NPUs, it is possible to accelerate low-power, high-efficiency AI recognition and reasoning functions in the mobile terminal part.

In particular, there is an increasing need for various types of AI recognition processes for general videos or videos. The GPU is efficient in the learning process using machine learning technology, but the NPU is more efficient in the recognition process, so the importance of the NPU is growing in the development of services or applications that apply to already created learning models. Therefore, most chipset manufacturers are starting to develop NPUs as a next-generation SoC (system on chip), and this will continue to increase in the future.

Currently, learning technology based on machine learning is generalized, but the development of artificial intelligence devices that provide direct service to terminals by applying algorithms (learning models) is not meeting market demands, and various NPU chipsets appear in the market. However, the content to use it is lacking. In addition, compared to the high camera performance, AI-applied devices and services are also limited.

In particular, due to the NPU dependency and data monopoly of the algorithm, there is no terminal platform capable of testing various learning models, and there is a lack of learning data that can use NPU.

An object of the present invention for solving the above problem is to provide a user terminal capable of supporting a user to install and run desired AI content without being dependent on the NPU.

Another object of the present invention for solving the above problem is to provide a method capable of distributing AI content without being dependent on a test platform of a terminal.

In order to achieve the above object, the present invention provides a user terminal equipped with an NPU adaptive edge framework that supports AI content installation and execution, wherein the NPU adaptive edge framework includes an application manager for managing AI applications, and content An OS platform including a content installation manager for downloading and installing AI content from a server, an AI content edge engine for executing the AI content and accelerating the first NPU provided in the user terminal, an NPU driver, and a sensor driver. Provides a user terminal that includes.

Here, the AI content includes a learning model and an environment parameter, the learning model is provided in the form of a hardware independent intermediate expression or an executable code supported by a specific NPU, and the environment parameter corresponds to the environment of the user terminal. It includes libraries and resources, and conversion server information in which a cloud compiler to be converted into executable code when the learning model is in the form of an intermediate expression is located, and the content installation manager receives a list of AI content downloadable from the content server, and selects It is characterized in that it includes a content downloader for downloading AI content and a cloud request unit for converting the learning model into execution code supportable by the first NPU by requesting a cloud compiler on the conversion server.

In addition, the AI content includes a learning model and environment parameters, and the learning model is provided in the form of an intermediate expression independent of hardware or in the form of executable code supported by a specific NPU, and the content installation manager configures the learning model as A local compiler for converting into a first local executable code supportable by the first NPU and conversion information including the first NPU information and information of the user terminal are added to the environment parameter and uploaded to the content server, and the content It may further include an authentication requesting unit requesting authentication of the first local executable code from a server.

In addition, when the learning model is in the intermediate expression form, the environment parameter includes information of a peer terminal having a second local execution code supportable by the first NPU, and the content installation manager and a peer communication unit for receiving the second local execution code by requesting a peer terminal.

In addition, the peer communication unit may transmit the first local execution code to the third peer terminal and transmit the transmission information to the contents server when there is a request from the third peer terminal.

Further, here, the second local execution code is an execution code converted by a local compiler of the peer terminal, and transmission information of the local execution code between peer terminals is settled based on a billing policy of the content server.

In addition, here, the AI content edge engine is characterized by including a model executor that decodes and infers downloaded AI content, and a content service protocol for controlling communication between the NPU driver and the model executor.

In order to achieve the above other object, the present invention provides an AI content distribution method for distributing AI content in an AI content production studio, generating AI content including a learned learning model and environmental parameters using a machine learning tool. and uploading the AI content to a content server, wherein the learning model is a hardware independent intermediate expression form or an execution code form converted to a specific NPU, and the environment parameter is a library corresponding to the environment of the user terminal. and a resource, and also includes information on a conversion server where a cloud compiler that will convert the learning model into an executable code when the learning model is in the form of an intermediate expression is located.

Here, when the learning model is in the form of an intermediate expression, further comprising authenticating the local executable code converted by the local compiler of the user terminal, wherein the authenticating step is performed on the local executable code from the user terminal or the content server. When an authentication request is received, the local executable code is authenticated based on the authentication policy of the content server, and authentication information of the local executable code is added to the environment parameter and uploaded to the content server.

In addition, here, the local executable code that has obtained the authentication is transmitted between user terminals in a P2P manner based on the information of the environment parameter, and the transmission information of the local executable code is settled based on the billing policy of the content server to be characterized

When using a user terminal equipped with an NPU adaptive AI contents service edge framework for installing and running AI contents of the present invention, by downloading various AI contents, converting them into execution codes suitable for the NPU provided in the terminal, and executing or testing the , can accelerate efficient AI recognition/inference capabilities. In addition, AI content producers have the advantage of distributing their own AI content through the NPU adaptive network platform and testing it on terminals in various environments.

1 is a conceptual diagram of an NPU adaptive network platform supporting AI content service according to an embodiment of the present invention.
2 is a conceptual diagram of an NPU adaptive edge platform supporting installation and execution of AI content according to an embodiment of the present invention.
3 is a block diagram showing an NPU adaptive edge framework supporting installation and execution of AI content according to an embodiment of the present invention.
4 is a flowchart illustrating a process of installing AI contents in a terminal equipped with an NPU adaptive edge framework supporting installation and execution of AI contents according to an embodiment of the present invention.
5 is a flowchart showing a process of producing and distributing AI content in a content production studio according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted in a meaning consistent with the contextual meaning of the related art, and are not interpreted in an ideal or excessively formal sense unless explicitly defined in this application.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of an NPU adaptive network platform supporting AI content service according to an embodiment of the present invention.

Referring to FIG. 1, in the NPU adaptive network platform according to an embodiment of the present invention, a content creator uploads various types of AI content produced to a content server 30 through a content production studio 20, and a user It is possible to provide a service platform that can download AI content anytime, anywhere and install it on the user terminal 10 .

In addition, referring to FIG. 1, the NPU adaptive network platform according to an embodiment of the present invention can distribute content learned through machine learning and convert the distributed content to be accelerated according to the environment of the user terminal. It may be a subscriber-based web service that provides an edge computing environment.

That is, the content production studio 20 develops and uploads AI content, builds the machine learning result into content that can be executed in the application provided in the user terminal 10, GUI-based UX (flow chart type Graphic Coding ) can be provided. At this time, the AI content may include a learning model and environmental parameters, and the learning model may be in the form of an intermediate expression independent of hardware, or may be in the form of executable code converted to be supported by a specific NPU in a production studio. In this case, the learning model may include an application having an independent user interface, or may operate in conjunction with an open recognition application.

In addition, referring to FIG. 1 , when the downloaded AI content is in the form of an intermediate expression, an edge computing environment capable of directly converting the downloaded AI content into executable code is provided in the user terminal 10 . That is, for example, the user terminal 10 converts the AI content downloaded through the cloud compiler on the conversion server 40 into an execution code suitable for the NPU provided in the terminal 10 or is installed in the peer terminal 11. Code can be sent. In addition, the user terminals 10 and 10 may be provided with an EDGE computing environment (NPU, sensor, application, etc.) capable of testing AI recognition services using various AI contents.

2 is a conceptual diagram of an NPU adaptive edge platform supporting installation and execution of AI content according to an embodiment of the present invention.

Referring to FIG. 2 , the NPU adaptive edge platform 100 according to an embodiment of the present invention supports a device-independent installation environment of AI content downloaded from the content server 30.

In the NPU adaptive edge platform 100 of the present invention, when the learning model included in the downloaded AI content is in the form of an intermediate expression, an environment that can be converted into an execution code suitable for the NPU 101 provided in the user terminal 10 is provided For example, the learning model may be converted into executable code supportable by the NPU 101 using a cloud compiler of the conversion server 40 or a local compiler provided in the terminal 10 . Alternatively, the converted execution code may be transmitted from the peer terminal 11 having the same environment. More details regarding the conversion of the learning model into executable code will be described later.

3 is a block diagram showing an NPU adaptive edge framework supporting installation and execution of AI content according to an embodiment of the present invention.

Referring to FIG. 3 , the NPU adaptive edge framework according to an embodiment of the present invention may be provided as an independent framework in the hardware platform 150 of the terminal. In addition, the NPU adaptive AI contents service edge framework according to an embodiment of the present invention accelerates the application manager 110 for managing applications, the NPU 101 provided in the user terminal 10 to execute AI contents and It may be composed of an AI content edge engine 130 and an OS platform 140 including an NPU driver and a sensor driver.

As described above, the AI content may include a learning model and environmental parameters, and the learning model may be provided in the form of an intermediate expression independent of hardware or in the form of executable code supported by a specific NPU, and the environmental parameters may include a user terminal (10 ) can include libraries and resources including installation information and execution information corresponding to the environment.

In addition, the environment parameter is supported by information on the conversion server 40 or NPU 101 where the cloud compiler to be converted into executable code supported by the NPU 101 provided in the terminal 10 when the learning model is in the form of an intermediate expression is located. It may include information of a peer terminal 11 having an executable code. More detailed information about the code conversion by the cloud compiler and the peer terminal will be described later.

The content installation manager 120 receives a downloadable AI content list from the content server 30, and converts the content downloader 121 for downloading the selected AI content and the learning model included in the AI content to the cloud on the conversion server 40. It may include a cloud request unit 122 for converting a request to the compiler into executable code supportable by the NPU 101 provided in the terminal 10 . The cloud compiler may include a web compiler, and code conversion using the cloud compiler is a known technology, so a detailed description thereof will be omitted.

In addition, the content installation manager 120 includes a local compiler 125 capable of converting the learning model included in the downloaded AI content into first local execution code supportable by the NPU 101 provided in the terminal 10, and , and may include an authentication request unit 123 requesting authentication for the first local executable code.

At this time, the authentication request unit 123 adds conversion information including NPU 101 information and user terminal 10 information (eg, user terminal identification information, physical address, network address, etc.) to the environmental parameters. and upload it to the content server 30, and request authentication for the first local executable code from the content server 30.

The authentication information is registered in the content server 30, and the authenticated execution code can be directly transmitted between peer terminals, and can be settled according to the server's billing policy.

In addition, the content installation manager 120 requests the peer terminal 11 having the second local execution code supportable by the NPU 11 included in the terminal 10 with reference to the environment parameter to request the second local execution code. It may include a peer communication unit 124 that receives. In this case, the second local execution code provided in the peer terminal 11 may be an execution code converted by a local compiler of the peer terminal 11 .

The content installation manager 120 may select a method for converting AI content (using a cloud compiler, using a local compiler, or requesting a peer terminal, etc.) according to the environment of the terminal.

When there is a request from the third peer terminal, the peer communication unit 124 transmits the first local execution code converted in the terminal 10 to the third peer terminal and transmits transmission information to the content server 30. may be At this time, the transmission information of the local executable code between the peer terminals 10 and 11 may be settled based on the billing policy of the content server 30 .

For example, when the first local execution code converted in the user terminal 10 is transmitted to a third peer terminal, a portion of the AI contents fee charged to the third peer terminal is a coupon or a coupon to the user terminal 10. It may be given in the form of points or the like. Conversely, a portion of the AI content fee charged to the user terminal 10 may be given as points to the peer terminal 11 that has transmitted the second local execution code to the user terminal 10 .

The AI content edge engine 130 may include a model executor 131 that decodes and infers downloaded AI content, and a content service protocol 132 for controlling communication between the NPU driver 141 and the model executor 131. there is.

For example, the AI content edge engine 130 integrates parts supported by different SDKs for each NPU company, and can use vast AI contents designed for different NPUs.

Hereinafter, a process of installing AI contents in a terminal according to an embodiment of the present invention will be described.

4 is a flowchart illustrating a process of installing AI contents in a terminal equipped with an NPU adaptive edge framework supporting installation and execution of AI contents according to an embodiment of the present invention.

Referring to FIG. 4, the process of installing AI contents in a terminal equipped with an NPU adaptive edge framework according to an embodiment of the present invention includes displaying a list of downloadable AI contents (S110), and AI contents download step ( S120), execution code conversion step using local compiler (S130) and converted code authentication request step (S140), execution code conversion step using cloud compiler (S150), execution code receiving step from peer terminal (S160), AI content It may be configured to include a terminal installation step (S170).

Hereinafter, each step of a process of installing AI contents in a terminal according to an embodiment of the present invention will be described in more detail.

First, the user can search for downloadable AI content through the content downloader installed in the user terminal (S110). In this case, the content downloader may be an application basically installed in the terminal or downloaded from the market. In addition, the AI content retrieved through the content downloader may include only AI content supported by the NPU provided in the user terminal, or may also include a learning model in the form of an intermediate expression independent of the NPU. In addition, AI content may include learning models and environmental parameters learned using deep learning tools.

In the case of AI content including a learning model in the form of an intermediate expression, a means capable of converting it into an executable code may be provided in the AI content search list. For example, it can be converted through the server's cloud compiler, or it can receive execution code from a peer terminal, and if the terminal has a local compiler, it can convert itself. The environment parameter may include information about a peer terminal having an address of a server where a cloud compiler is located or a certified executable code. Information on various conversion means may be provided for the same AI content.

The content downloader may download AI content suitable for the environment of the terminal (S120). If the AI content is in the form of execution code supported by the NPU provided in the terminal, it can be installed in the user terminal S170 without undergoing a separate conversion. Alternatively, if it is in the form of an intermediate code and the terminal is equipped with a local compiler, the terminal can use the local compiler to convert the learning model included in the downloaded AI content into a first local execution code that can be supported by the NPU provided in the terminal. (S130).

At this time, the terminal converts information including NPU information related to the first local execution code and information of the user terminal (eg, identification information of the user terminal, physical address, network address, etc.) to the environmental parameters included in the AI content. It can be added and uploaded to the content server, and authentication of the first local execution code can be requested from the content server (S140). Authentication information is registered in the content server, and the authenticated execution code can be directly transmitted between peer terminals, and can be settled according to the server's billing policy. For example, a portion of the fee paid by the terminal that has downloaded the AI content may be paid to the peer terminal that has transmitted the execution code.

If the AI content is in the form of an intermediate expression, but there is no local compiler in the terminal, the terminal executes the learning model included in the AI content downloaded using the cloud compiler on the conversion server 40 supported by the NPU provided in the terminal. It can be converted into a code (S150).

Alternatively, the terminal may receive the second local execution code by requesting the peer terminal having the second local execution code supportable by the NPU provided in the terminal with reference to the environment parameter (S160). In this case, the second local execution code provided in the peer terminal is an execution code converted by a local compiler of the peer terminal, and may be a code certified by a content server or a content production studio.

The AI content converted through the above process may be installed in the user terminal (S170).

Hereinafter, the process of producing and distributing AI contents will be described.

5 is a flowchart showing a process of producing and distributing AI content in a content production studio according to an embodiment of the present invention.

Referring to FIG. 5 , the process of producing and distributing AI content according to an embodiment of the present invention includes a recognition application selection step (S210), a learning model creation step (S220), a learning model conversion step (S230), and an AI content production step. (S240), content upload step (S250) and local execution code authentication step (S260) may be configured.

Hereinafter, a process of producing and distributing AI content in a content production studio will be described in more detail with reference to FIG. 5 .

First, a content production studio selects a recognition application to which AI content is to be applied (S210). The recognition application may refer to an application that receives various signals through various sensors (eg, a camera, a sound sensor, a gravity sensor, a fire sensor, etc.) and operates based on the received signals. In addition, the recognition application may be an application pre-installed in a user terminal that downloads AI content or an application that is downloaded together with AI content.

A content production studio may refer to a terminal used by various entities that generate and distribute learning models through deep learning and big data analysis, such as schools, research institutes, companies, and individuals.

Next, a learning model to be used in the selected recognition application can be created (S220). The learning model may be a model generated by machine learning through big data analysis. Big data analysis and machine learning are known methods, so detailed descriptions are omitted. The learning model can be created in a device-independent intermediate expression form. Alternatively, it may be in the form of executable code that is not supported by the NPU and is manufactured according to the OS of the terminal.

Next, the generated learning model can be converted into executable code supported by a specific NPU (S230). In this step, a plurality of execution codes converted to suit a plurality of various NPUs may be generated. Alternatively, this step (S230) may be omitted.

Next, AI content including the learning model and environmental parameters created in the previous step can be produced (S240). At this time, the learning model may be provided in the form of hardware independent intermediate expression or execution code supported by a specific NPU as described above, and the environment parameter is a library including installation information and execution information corresponding to the environment of the user terminal. and resources.

In addition, the environment parameter includes information on a conversion server where a cloud compiler to be converted into an NPU-supportable executable code provided in the terminal when the learning model is in the intermediate expression form is located, or information on a peer terminal having an executable code supportable by the NPU. can do. However, information on peer terminals may not be generated in the content production stage.

Next, the created AI content including the learning model and environmental parameters can be uploaded to the server (S250).

Next, when the learning model is in the form of an intermediate expression, the local execution code converted by the local compiler of the user terminal can be authenticated (S260). For example, when a user terminal requests authentication for a local executable code from a content server, the content server may transmit the content of the authentication request to the content production studio.

The content production studio receiving the requested contents can authenticate the local executable code based on the authentication policy of the content server, add authentication information of the local executable code to an environment parameter, and upload the content to the content server. Meanwhile, authentication of local execution code may be performed directly in the content server. This may vary according to the authentication policy of the content server. Such authentication information may be initialized when content is updated to ensure content integrity.

In addition, the authenticated local execution code is transmitted between user terminals in a P2P manner based on environment parameter information, and the transmission information of the local execution code may be settled based on the billing policy of the content server.

As described above, using a user terminal equipped with an NPU adaptive AI contents service edge framework for installing and executing AI contents of the present invention, by downloading various AI contents and converting them into execution codes suitable for the NPU provided in the terminal By running or testing, efficient AI recognition/inference capabilities can be accelerated. In addition, AI content creators have the advantage of distributing self-produced AI content through the NPU adaptive network platform of the present invention and testing it on terminals in various environments.

Meanwhile, a device according to an embodiment of the present invention may be provided in the form of an application running in a user terminal. At this time, the user terminal may include various computing devices including a smart phone, a tablet PC, a laptop computer, a desktop computer, a PDA, and the like.

Meanwhile, the method of the present invention described above may be produced as a program code to be executed on a computer and stored in a computer readable medium, and a program executing the method of the present invention may be produced as an application and stored in a recording medium. It can be. The recording medium may include a removable storage medium such as a hard disk, memory, auxiliary memory, compact disk, etc. of a computer, mobile terminal, or server.

In addition, the method and apparatus of the present invention described so far may actually be implemented by a computer program, and may be stored in a computer-readable recording medium when executed on a computer. Computer-readable recording media include all types of recording media in which programs and data are stored so that they can be read by a computer system, and include ROM, RAM, CD, DVD-ROM, magnetic tape, floppy disk, optical data storage device, etc. And, it may also include what is implemented in the form of transmission through the Internet. That is, such a medium can be distributed to computer systems connected through a network, so that computer readable codes can be stored and executed in a distributed manner.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

Claims (10)

In a user terminal equipped with an NPU adaptive edge framework that supports AI content installation and execution, the NPU adaptive edge framework comprises
A content installation manager for downloading and installing AI content from the content server;
It is configured to include an AI content edge engine for executing the AI content and accelerating the first NPU provided in the user terminal.
The AI content includes a learning model and environmental parameters, and the learning model is provided in the form of an intermediate expression independent of hardware or in the form of executable code supported by a specific NPU,
The environment parameter includes library and resources corresponding to the environment of the user terminal and conversion server information in which a cloud compiler to be converted into executable code when the learning model is in the form of an intermediate expression is located. in claim 1
The content installation manager
A content downloader for receiving a list of AI content downloadable from the content server and downloading selected AI content; and
A terminal characterized in that it comprises a cloud request unit for requesting a cloud compiler on the conversion server to convert the learning model into executable code supportable by the first NPU. According to claim 1 or claim 2,
The content installation manager
A local compiler for converting the learning model into a first local execution code supportable by the first NPU;
An authentication request for adding conversion information including information on the first NPU and information on the user terminal to the environment parameter, uploading the converted information to the content server, and requesting the content server to authenticate the first local executable code. A terminal characterized in that it further comprises a unit. The method of claim 3,
The environment parameter includes information of a peer terminal having a second local execution code supportable by the first NPU when the learning model is in an intermediate expression form,
The content installation manager
and a peer communication unit for receiving the second local execution code by making a request to the peer terminal. The method of claim 4,
The peer communication unit transmits the first local execution code to the third peer terminal when there is a request from a third peer terminal, and transmits information about transmission of the first local execution code to a content server. Terminal to. The method of claim 5,
The second local execution code is an execution code converted by a local compiler of the peer terminal, and transmission information of the local execution code between peer terminals is settled based on a billing policy of the content server. delete In the AI content distribution method for distributing AI content in an AI content production studio,
The steps of generating AI content including the learning model and environmental parameters learned using machine learning tools;
Uploading the AI content to a content server,
The learning model is in the form of an intermediate expression independent of hardware or in the form of executable code converted to suit a specific NPU,
The environment parameter includes libraries and resources corresponding to the environment of the user terminal, and also includes conversion server information where a cloud compiler that will convert the learning model into an executable code is located when the learning model is in the form of an intermediate expression. AI content distribution method. The method of claim 8,
If the learning model is in the form of an intermediate expression, further comprising authenticating the local execution code converted in the local compiler of the user terminal,
In the authentication step, upon receiving an authentication request for the local executable code from the user terminal or the content server, the local executable code is authenticated based on the authentication policy of the content server, and the local executable code is set to the environment parameter. AI content distribution method characterized by adding authentication information of and uploading it to the content server. The method of claim 9,
Characterized in that the authenticated local execution code is transmitted between user terminals in a P2P manner based on the environment parameter information, and the transmission information of the local execution code is settled based on the billing policy of the content server. How to distribute AI content.

Images