KR102444932B1 - Electronic device and Method for controlling the electronic device - Google Patents

Abstract

The present disclosure relates to an artificial intelligence (AI) system using a machine learning algorithm such as deep learning and its application. In particular, the control method of an electronic device for encrypting an image using an artificial intelligence learning model and uploading it to an external server is to acquire an image, receive a user command for uploading the acquired image to an external server, and respond to the user command. In response, an image obtained by inputting an image obtained to an encryption model learned to obtain a feature value by using an image and a user key as input data and a user key of the electronic device to obtain a feature value corresponding to the image obtained, Identification information and feature values may be transmitted to an external server.

Description

Electronic device and method for controlling the electronic device for encrypting an image and uploading it to an external server

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device for encrypting an image using a user key and uploading the encrypted image to an external server, and a control method thereof.

In addition, the present disclosure relates to an artificial intelligence (AI) system for simulating functions such as cognition and judgment of a human brain by using a machine learning algorithm and an application thereof.

Recently, artificial intelligence systems that implement human-level intelligence have been used in various fields. Unlike the existing rule-based smart system, an artificial intelligence system is a system in which a machine learns, judges, and becomes smarter by itself. The more the artificial intelligence system is used, the better the recognition rate and the more accurate understanding of user preferences, and the existing rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

Artificial intelligence technology consists of machine learning (eg, deep learning) and elemental technologies using machine learning.

Machine learning is an algorithm technology that categorizes/learns the characteristics of input data by itself, and element technology uses machine learning algorithms such as deep learning to simulate functions such as cognition and judgment of the human brain. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

Various fields where artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answer, and speech recognition/synthesis. Visual understanding is a technology for recognizing and processing objects like human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, and the like. Inferential prediction is a technology for logically reasoning and predicting by judging information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge construction (data generation/classification) and knowledge management (data utilization). Motion control is a technology for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), manipulation control (action control), and the like.

Meanwhile, in recent years, as communication technology develops and the capacity of a high-definition image gradually increases, an image captured by the electronic device may be stored in an external device such as a cloud server as well as simply stored inside the electronic device.

In this case, when transmitting an image to an external device such as a cloud server, privacy problems due to hacking may occur, and time and costly problems may occur when transmitting a high-quality image itself to an external device.

Accordingly, there is a demand for a method for preventing personal information leakage in the process of uploading or downloading an image to an external device and uploading an image with a small capacity.

An object of the present disclosure is to prevent leakage of personal information by encrypting and compressing an image using a user key when uploading and downloading an image to an external server, and to upload and download a compressed image of a small capacity. And to provide a control method thereof.

According to an embodiment of the present disclosure, a control method of an electronic device for encrypting an image using an artificial intelligence learning model and uploading it to an external server includes: acquiring an image; receiving a user command for uploading the acquired image to the external server; In response to the user command, by inputting the acquired image and the user key of the electronic device into an encryption model trained to acquire a feature value using an image and a user key as input data, a feature value corresponding to the acquired image obtaining a; and transmitting the identification information of the acquired image and the feature value to the external server.

In addition, the user key of the electronic device may be one of a password input by the user of the electronic device and identification information of the electronic device, or a new value generated therefrom.

In addition, the transmitting includes, when there are a plurality of user keys of the electronic device, version information of a user key corresponding to the obtained image among the plurality of user keys, identification information of the obtained image and the feature It can be transmitted to the external server together with the value.

In addition, in response to the user command, obtaining a thumbnail image corresponding to the obtained image; matching and storing the obtained thumbnail image and identification information of the obtained image; and deleting the acquired image.

And, when a user command for displaying the acquired image is received from the user, providing the acquired thumbnail image; may include.

In addition, the external server matches and stores the identification information of the acquired image with the characteristic value, compares the similarity between the acquired image and another image based on the characteristic value, and based on the similarity comparison, the The acquired images can be classified.

And, receiving a keyword input; receiving a user command for searching for an image related to the keyword; transmitting a signal for requesting a search related to the keyword to the external server; and receiving, from the external server, a feature value corresponding to at least one search image related to the keyword among images classified based on the similarity comparison in response to the signal.

In addition, the at least one search image is obtained by inputting a feature value corresponding to the at least one search image and a user key of the electronic device into a decoding model that has been trained to restore an image using a feature value and a user key as input data. Restoring; may include.

And, when a user's download command for receiving a feature value corresponding to the acquired image is input, transmitting a signal requesting a feature value corresponding to the acquired image to the external server; receiving a feature value corresponding to the acquired image from the external server in response to the signal; and inputting a feature value corresponding to the acquired image and a user key of the electronic device into a decoding model trained to restore an image by using the feature value and the user key as input data to restore the acquired image. may include.

In addition, the receiving includes receiving version information of a user key of the electronic device used to encrypt the acquired image together with a feature value corresponding to the acquired image from the external server, and restoring the The step may include inputting a feature value corresponding to the acquired image and a user key of the electronic device corresponding to the version information to the decoding model to restore the acquired image.

Meanwhile, according to an embodiment of the present disclosure, an electronic device includes: a communication unit; display; a memory for storing an encryption model learned to obtain a feature value using an image and a user key as input data and a program for performing an operation of the electronic device; and a processor electrically connected to the communication unit, the display, and the memory to control the electronic device, wherein the processor obtains an input signal according to a user command for uploading the acquired image to the external server, , in response to the input signal, input the acquired image and the user key of the electronic device to the encryption model to acquire a feature value corresponding to the acquired image, identification information of the acquired image and the feature value may control the communication unit to transmit to the external server.

In addition, the user key of the electronic device may be one of a password input by the user of the electronic device and identification information of the electronic device, or a new value generated therefrom.

In addition, when there are a plurality of user keys of the electronic device, the processor is configured to combine version information of a user key corresponding to the obtained image among the plurality of user keys with identification information of the obtained image and the feature value. The communication unit may be controlled to transmit to the external server together.

In addition, in response to the user command, the processor acquires a thumbnail image corresponding to the acquired image, matches the acquired thumbnail image with identification information of the acquired image, and stores it in the memory, and the acquisition images can be deleted from the memory.

And, when a user command for displaying the acquired image is received from the user, the processor may control the display to provide the acquired thumbnail image.

In addition, the identification information of the acquired image and the characteristic value are matched and stored in the external server, and the external server compares the similarity between the acquired image and another image based on the characteristic value, and compares the similarity Based on , the acquired image may be classified.

The processor controls the communication unit to obtain an input signal according to a user command for searching for an image related to the input keyword, and to transmit a signal for requesting a search related to the keyword to the external server, the communication unit Through , a feature value corresponding to at least one search image related to the keyword among images classified based on the similarity comparison may be received from the external server.

In addition, the memory stores a decoded model trained to restore an image by using a feature value and a user key as input data, and the processor includes: a feature value corresponding to the at least one search image in the decryption model; The at least one search image may be restored by inputting a user key of the electronic device.

And, the memory stores a decryption model learned to restore an image by using the feature value and the user key as input data, the processor,

When a user's download command for receiving a feature value corresponding to the acquired image is input, the communication unit is controlled to transmit a signal requesting a characteristic value corresponding to the acquired image to the external server, and the communication unit receive a feature value corresponding to the obtained image from the external server through .

In addition, version information of a user key of the electronic device used to encrypt the acquired image is received together with a feature value corresponding to the acquired image from the external server, and the processor is configured to include the The acquired image may be restored by inputting a feature value corresponding to the acquired image and a user key of the electronic device corresponding to the version information.

According to various embodiments of the present disclosure as described above, when a user uploads or downloads an image acquired by an electronic device to an external server, it is possible to prevent leakage of personal information as well as to transmit an image to an external device with a small capacity. can be saved

1 is a view for explaining a system capable of compressing and encrypting an image using a user key and storing it in an external server according to an embodiment of the present disclosure;
2 is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
3 is a block diagram illustrating the configuration of an electronic device in detail according to an embodiment of the present disclosure;
4 is a block diagram including a module for encrypting an image and a module for decrypting an image, according to an embodiment of the present disclosure;
5A and 5B are diagrams for explaining a learned encryption model and a decryption model according to an embodiment of the present disclosure;
6A and 6B are diagrams for explaining a method of generating input data input to an encryption model according to an embodiment of the present disclosure;
7 is a block diagram schematically illustrating the configuration of an external server according to an embodiment of the present disclosure;
8A and 8B are diagrams for explaining a method of encrypting and uploading an image using an encryption model, according to an embodiment of the present disclosure;
9A and 9B are diagrams for explaining a method of downloading and decoding an image using a decoding model, according to an embodiment of the present disclosure;
10A and 10B are diagrams for explaining a method of searching for and decoding an image using a decoding model, according to an embodiment of the present disclosure;
11 is a flowchart illustrating a method for encrypting an image using an encryption model and uploading it to an external server according to an embodiment of the present disclosure;
12 is a block diagram illustrating a configuration of an electronic device for learning and using an encryption model and a decryption model according to an embodiment of the present disclosure;
13A to 13C are block diagrams showing specific configurations of a learning unit and an encryption unit/decryption unit according to an embodiment of the present disclosure;
14 and 15 are flowcharts of a network system using an encryption model or a decryption model, according to various embodiments of the present disclosure.

Hereinafter, various embodiments of the present disclosure will be described with reference to the accompanying drawings. However, this is not intended to limit the technology described in the present disclosure to specific embodiments, and it should be understood that various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure are included. . In connection with the description of the drawings, like reference numerals may be used for like components.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this disclosure, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as “first,” “second,” “first,” or “second,” may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it should be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this disclosure depends on the context, for example, "suitable for," "having the capacity to" ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a coprocessor configured (or configured to perform) A, B, and C” may refer to a dedicated processor (eg, an embedded processor), or one or more software programs stored in a memory device, to perform the corresponding operations. By doing so, it may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

An electronic device according to various embodiments of the present disclosure may be, for example, a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a PDA, a PMP. It may include at least one of a portable multimedia player, an MP3 player, a medical device, a camera, and a wearable device. A wearable device may be an accessory (e.g., watch, ring, bracelet, anklet, necklace, eyewear, contact lens, or head-mounted-device (HMD)), a textile or clothing integral (e.g. electronic garment); It may include at least one of a body-worn (eg, skin pad or tattoo) or bioimplantable circuit In some embodiments, the electronic device may include, for example, a television, a digital video disk (DVD) player, Audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set-top box, home automation control panel, security control panel, media box (e.g. Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game console (eg, Xbox™, PlayStation™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

In another embodiment, the electronic device may include various medical devices (eg, various portable medical measuring devices (eg, a blood glucose meter, a heart rate monitor, a blood pressure monitor, or a body temperature monitor), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), Computed tomography (CT), imagers, or ultrasound machines, etc.), navigation devices, global navigation satellite systems (GNSS), event data recorders (EDRs), flight data recorders (FDRs), automotive infotainment devices, marine electronic equipment (e.g. navigation devices for ships, gyro compasses, etc.), avionics, security devices, head units for vehicles, industrial or household robots, drones, ATMs in financial institutions, point of sale (POS) in stores of sales) or IoT devices (eg, light bulbs, various sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heaters, boilers, etc.).

According to an embodiment of the present disclosure, the external server may be implemented as a cloud server for storing images, as well as various devices such as a personal cloud device, an external storage device, and an electronic device as described above.

In the present disclosure, the term user may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

1 is a diagram for explaining a system capable of compressing and encrypting an image using a user key and storing it in an external server according to an embodiment of the present disclosure. The system may include a first electronic device 100 - 1 , a second electronic device 100 - 2 , and a server 200 . In this case, the first electronic device 100 - 1 and the second electronic device 100 - 2 may be electronic devices used by the same user, but this is only an example and may be implemented as the same electronic device. That is, one electronic device may upload and download an image.

The first electronic device 100 - 1 may acquire an image. In this case, the image may be an image captured by the first electronic device 100 - 1 , but this is only an example, and may be various images such as an image received from another electronic device or an image downloaded from the web. .

The first electronic device 100 - 1 may receive a user command for uploading an image to the server 200 . In this case, the first electronic device 100 - 1 may receive a user command for uploading an image through a menu, but this is only an example, and uploading an image to the server 200 through various user inputs You can receive user commands for

The first electronic device 100-1 may obtain a user key of the first electronic device 100-1 for encrypting and compressing an image. In this case, the user key may be identification information of the electronic device (eg, serial number of the electronic device, IMEI information, etc.), but this is only an example and may be a password input by the user. In addition, the user key may be generated by a combination of identification information of the electronic device and a password input by the user. The password input by the user may be an initially input password, but this is only an example and may be changed every time an image is transmitted or at a preset period.

The first electronic device 100 - 1 may acquire a feature value for uploading to the server 200 using the image and the user key in response to a user command for uploading the image. In particular, the first electronic device 100-1 obtains an image and a user key of the first electronic device 100-1 by inputting an image and a user key of the first electronic device 100-1 into an encryption model trained to obtain a feature value using the image and the user key as input data. It is possible to obtain a feature value for the image. In this case, the encryption model may be a deep neural network model, but is not limited thereto. In addition, the encryption model may have the effect of encrypting the image using the image and the user key, but at the same time may have the effect of compressing the image, and thus may be referred to as a compression model.

In addition, the feature value is an image representation value extracted by passing an image and a user key through a learned neural network as input, and can be used in various terms such as attribute values, feature vectors, and feature information. In particular, the feature value can be obtained through representation learning of a deep neural network model.

In particular, a feature value obtained by inputting an image and a user key into the encryption model can be decrypted into a normal image only when not only the feature value but also the user key used for encryption must be used when decrypting the image again into an image. When a feature value obtained by inputting an image and a user key into the encryption model is decrypted back into an image, if only the feature value is input to the decryption model, a normal image cannot be restored. Accordingly, the encryption model and the decryption model can be trained in pairs. In this case, since the decoding model may have an effect of restoring the compressed image, it may also be referred to as a restoration model.

The first electronic device 100 - 1 may upload the acquired feature value to the server 200 . In this case, the first electronic device 100-1 may transmit not only the acquired feature value but also the acquired image identification information (eg, image name, image creation date, image creation location, etc.) to the server 200 together. have. Also, the first electronic device 100 - 1 may transmit the acquired feature value and the acquired image identification information to the server 200 , and then generate a thumbnail corresponding to the image. In this case, the original image may be deleted.

The server 200 may store and manage the transmitted feature value. Specifically, the server 200 may match and store the feature value obtained from the first electronic device 100 - 1 and the identification information of the image. Also, the server 200 may compare the obtained image based on the feature value with the similarity of other images, and classify the obtained image based on the similarity comparison. In this case, the server 200 may classify the feature values corresponding to the acquired images in a vector form. Accordingly, the server 200 may provide the user with an image obtained in response to a keyword or image search request of the first electronic device 100 - 1 or the second electronic device 100 - 2 .

The second electronic device 100 - 2 may receive a download command for the acquired image from the user. In this case, the second electronic device 100 - 2 may receive a user command for downloading an image through a menu, but this is only an exemplary embodiment. You can receive user commands for

The second electronic device 100 - 2 may transmit a download request for the acquired image to the server 200 in response to the download command. In this case, the download request may include identification information of the acquired image.

The server 200 may search for a feature value corresponding to the acquired image based on identification information of the acquired image and transmit the found feature value to the second electronic device 100 - 2 .

The second electronic device 100 - 2 may restore the acquired image using the feature value received from the server 200 and the user key of the second electronic device 100 - 2 . Specifically, the second electronic device 100 - 2 includes the feature value received from the server 200 and the second electronic device 100 - 2 in a decoding model trained to obtain an image using the feature value and the user key as input data. The acquired image can be restored using the user key of . In this case, the image may be restored normally only when the user key of the first electronic device 100 - 1 and the user key of the second electronic device 100 - 2 are the same.

The encryption model and the decryption model learned in the present disclosure may be constructed in consideration of the field of application of the model or the computer performance of the device. For example, the learned encryption model may be set to acquire feature values using a user key and an image as input data, and the learned decryption model may be set to restore an image using the user key and feature values as input data have. In this case, the user key used to learn the encryption model and the user key used to learn the decryption model may be the same user key. In order to generate a feature value corresponding to the image by using the user key, and to restore the image using the generated feature value, the learned encryption model and decryption model are, for example, a model based on a neural network. can The encryption model and the decryption model may be designed to simulate a human brain structure on a computer and may include a plurality of network nodes having weights that simulate neurons of a human neural network. The plurality of network nodes may each form a connection relationship so that the neuron simulates a synaptic activity of a neuron that transmits and receives a signal through a synapse. In addition, the encryption model and the decryption model model may include, for example, a neural network model or a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes may exchange data according to a convolutional connection relationship while being located at different depths (or layers). Examples of the encryption model and the decryption model may include, but are not limited to, a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).

In addition, the first electronic device 100-1 and the second electronic device 100-2 encrypt the image selected by the user as a feature value as described above, upload it to the server 200, and An artificial intelligence agent may be used to download and restore the feature value. In this case, the artificial intelligence agent is a dedicated program for providing AI (Artificial Intelligence)-based services (eg, voice recognition service, secretarial service, translation service, search service, etc.), and an existing general-purpose processor (eg, CPU) or a separate AI-only processor (eg, GPU, etc.). In particular, the AI agent can control various modules to be described later.

Specifically, when a user input for uploading an image or a user input for downloading an image is received, the artificial intelligence agent may operate. Then, the artificial intelligence agent may encrypt the image into a feature value based on the user input and upload it to the external server 200 or download the feature value stored in the external server 200 and restore it as an image.

Of course, when a user input for uploading an image or a user input for downloading an image is received, the artificial intelligence agent may operate. Alternatively, the artificial intelligence agent may be in a pre-executed state before a user input for uploading an image or a user input for downloading an image is received. In this case, after a user input for uploading an image or a user input for downloading an image is received, the artificial intelligence agent may encrypt the image into a feature value and upload it or download the feature value and restore it as an image. Also, the artificial intelligence agent may be in a standby state before receiving a user input for uploading an image or a user input for downloading an image. Here, the standby state is a state in which a predefined user input is detected to control the start of the operation of the artificial intelligence agent. When a user input for uploading an image or a user input for downloading an image is received while the artificial intelligence agent is in the standby state, the first electronic device 100-1 or the second electronic device 100-2 is the artificial intelligence agent , and uploading the image by encrypting the feature value or downloading the feature value to restore the image. On the other hand, the artificial intelligence agent can control various modules to be described later. This will be described in detail later.

In addition, specific examples in which the electronic device 100 uploads or downloads an image using the learned encryption model or decryption model will be described below through various embodiments.

2 is a block diagram schematically illustrating a configuration of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 2 , the electronic device 100 includes a communication unit 110 , a display 120 , a memory 130 , and a processor 140 . The configurations shown in FIG. 2 are exemplary diagrams for implementing embodiments of the present disclosure, and appropriate hardware/software configurations at a level obvious to those skilled in the art may be additionally included in the electronic device 100 .

The communication unit 110 may communicate with an external device or an external server according to various communication methods. In particular, the communication unit 110 may transmit a feature value corresponding to the image to an external server in response to the user's image upload request, and receives the feature value corresponding to the image from the external server in response to the user's image download request can do.

The display 120 may provide various screens. In particular, the display 110 may display a screen including at least one image (eg, a photo). Also, the display 110 may display a UI for receiving various user commands, such as an image upload command, an image download command, an image search command, and the like.

The memory 130 may store various programs and data for performing an operation of the electronic device 100 . In particular, the MEMIRO 130 is an encryption model learned to obtain feature values for an image by using an image and a user key as input data, and a decryption model trained to restore an image by using the feature value and the user key as input data. At least one can be stored. In addition, the memory 130 may include an encryption module 410 and a decryption module 420 as shown in FIG. 4 . This will be described in detail later with reference to FIG. 4 .

The processor 140 may be electrically connected to the communication unit 110 , the display 120 , and the memory 130 to control overall operations and functions of the electronic device 100 . In particular, the processor 140 may encrypt the image as a feature value through the encryption module 410 and upload it to an external server. Specifically, when an input signal according to a user command for uploading the acquired image to the external server 200 is received, the processor 140 responds to the input signal, and the image acquired in the encryption model 413 and the electronic device ( 100 ) to obtain a feature value corresponding to the acquired image, and control the communication unit 110 to transmit identification information and feature value of the acquired image to the external server 200 .

Also, the processor 140 may decode a feature value downloaded from an external server into an image through the decoding module 420 . Specifically, when an input signal according to a user command for downloading a feature value corresponding to an image uploaded to the external server 200 is received, the processor 140 receives the feature value from the external server 200 in response to the input signal. It is possible to make a request, receive a feature value through the communication unit 110 , and input the received feature value and user key into the decryption model 422 to restore an image corresponding to the feature value.

3 is a detailed block diagram illustrating the configuration of the electronic device 100 according to another embodiment of the present disclosure. 3 , the electronic device 100 includes a communication unit 110 , a display 120 , a memory 130 , a processor 140 , a user input unit 150 , an audio output unit 160 , and a camera 170 . ) may be included.

The communication unit 110 may communicate with various types of external devices according to various types of communication methods. The communication unit 110 may include at least one of a Wi-Fi chip 111 , a Bluetooth chip 112 , a wireless communication chip 113 , and an NFC chip 114 . The processor 140 may communicate with an external server or various external devices using the communication unit 110 . In particular, the communication unit 110 may communicate with an external cloud server.

The display 120 may display image data processed by an image processing unit (not shown) on a display area (or a display). The display area may refer to at least a portion of the display 120 exposed on one surface of the housing of the electronic device 100 .

At least a portion of the display 120 may be located in at least one of a front area, a side area, and a rear area of the electronic device 100 in the form of a flexible display. The flexible display may be characterized by being able to bend, bend, or roll without damage through a thin and flexible substrate like paper.

The display unit 120 may be implemented as a touch screen having a layer structure in combination with a touch panel. The touch screen may have a function of detecting not only a display function but also a touch input position, a touched area, and even a touch input pressure, and also has a function of detecting a proximity touch as well as a real-touch. can

The memory 130 may store commands or data related to at least one other component of the electronic device 100 . In particular, the memory 130 may be implemented as a non-volatile memory, a volatile memory, a flash-memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 130 is accessed by the processor 140 , and reading/writing/modification/deletion/update of data by the processor 140 may be performed. In the present disclosure, the term "memory" refers to a memory 130, a ROM (not shown) in the processor 140, a RAM (not shown), or a memory card (not shown) mounted in the electronic device 100 (eg, micro SD). card, memory stick). In addition, programs and data for configuring various screens to be displayed on the display area of the display unit 110 may be stored in the memory 130 .

The user input unit 150 may receive various user inputs and transmit them to the processor 140 . In particular, the user input unit 150 may include a touch sensor, a (digital) pen sensor, a pressure sensor, or a key. The touch sensor may use, for example, at least one of a capacitive type, a pressure sensitive type, an infrared type, and an ultrasonic type. The (digital) pen sensor may be, for example, a part of a touch panel or may include a separate recognition sheet. The key may include, for example, a physical button, an optical key, or a keypad.

In particular, the user input unit 150 may obtain an input signal according to a user input for uploading an image or a user input for downloading an image. In addition, the user input unit 150 may transmit an input signal to the processor 140 .

The audio output unit 160 is configured to output not only various audio data on which various processing operations such as decoding, amplification, and noise filtering have been performed by the audio processing unit (not shown) but also various notification sounds or voice messages. In particular, the audio output unit 160 may be implemented as a speaker, but this is only an exemplary embodiment and may be implemented as an output terminal capable of outputting audio data.

The camera 170 may capture an image through a lens. In this case, the camera 170 may be provided on at least one of a front and a rear side of the electronic device 100 . Meanwhile, the camera 170 may be provided inside the electronic device 100 , but this is only an exemplary embodiment, and it exists outside the electronic device 100 and may be connected to the electronic device 100 by wire or wireless.

The processor 140 (or the controller) may control the overall operation of the electronic device 100 using various programs stored in the memory 130 .

The processor 140 may include a RAM 141 , a ROM 142 , a graphics processing unit 143 , a main CPU 144 , first to n interfaces 145-1 to 145-n, and a bus 146 . have. In this case, the RAM 141 , the ROM 142 , the graphic processing unit 143 , the main CPU 144 , the first to n-th interfaces 145-1 to 145-n, etc. may be connected to each other through the bus 146 . .

4 is a block diagram including an encryption module for encrypting an image and a decryption module for decrypting an image according to an embodiment of the present disclosure. First, as shown in FIG. 4 , the electronic device 100 may include both the encryption module 410 and the decryption module 420 , but this is only an embodiment, and the encryption module 410 and the decryption module It may contain only one of (420).

The encryption module 410 may include an image acquisition module 411 , an image encryption module 412 , an encryption model 413 , a user key management module 414 , and a thumbnail image generation module 415 .

The image acquisition module 411 may acquire an image for uploading from the electronic device 100 to the external server 200 . The image acquisition module 411 may acquire an image captured by the camera 170 , acquire an image transmitted from an external electronic device, and acquire an image downloaded from a website.

The image encryption module 412 may encrypt an image using the encryption model 413 . In this case, the encryption model 413 may be a model trained to obtain the feature value 530 using the image 510 and the user key 520 as input data, as shown in FIG. 5A . At this time, the encryption model 413 may be a Convolution Neural Network (CNN), but this is only an embodiment, and a Generative Adversarial Network (GAN), Various neural networks such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Bidirectional Recurrent Deep Neural Network (BRDNN), etc. may be used.

Specifically, the image encryption module 412 may generate input data using an image and a user key. The image encryption module 412 may generate input data using an image and a user key in various ways.

As an example, the image encryption module 412 may configure the image 610 as three channels (RGB) having a width and a height, as shown in FIG. 6A , and a user key 620 ) can also be composed of N channels having a width and a height. In addition, the image encryption module 412 may generate input data 630 having (N+3) channels by concatenating the image 610 of 3 channels and the user key 620 of N channels. have. That is, the image encryption module 412 may generate the input data 630 by summing the channel of the image 610 and the channel of the user key 620 .

As another example, the image encryption module 412 may configure the image 610 as three channels (RGB) having a width and a height, as shown in FIG. 6b, and a user key ( 640) can also be configured as one channel having a width and a height. Then, the image encryption module 412 generates input data 650 having three channels by summing each pixel of the image 610 of three channels with the pixels of the user key 640 of one channel. can That is, the image encryption module 412 may generate the input data 650 by summing the pixel value of the image 610 and the pixel value of the user key 640 .

Meanwhile, the structure and setting values of the neural network may be changed according to the embodiments described with reference to FIGS. 6A and 6B , respectively. That is, depending on the type of input data, the input data size of the neural network, the number of layers constituting the neural network, the kernel size and number of channels in each layer, the number of pooling layers, the pooling size, and whether skip connection is applied , whether Intermediate Feature and Top-down Feature are applied may be changed.

The image encryption module 412 may input the generated input data to the encryption model 413 including a plurality of layers. In this case, the input data may be extracted as feature values of gradually smaller sizes by calculating a plurality of layers. That is, the input data is calculated as a first feature value by passing the first layer, the first feature value is computed as a second feature value by passing the second layer, and the second feature value passes through the pooling layer, so that the size may be calculated with the reduced third feature value. If N layers are passed through in this way, it can be calculated as a final feature value with a reduced size.

That is, the image encryption module 412 may pass the input data 630 or 650 through a plurality of layers to obtain the finally calculated feature value 530 . In this case, the acquired feature value 530 is encrypted through a plurality of layers. That is, the acquired feature value 530 cannot be restored to the original image without a user key. In addition, the size of the acquired feature value 530 may be reduced through the pooling layer, so that it may have a compressed effect compared to the existing image.

The user key management module 414 may manage user keys. Specifically, when the user key is the same as identification information (eg, serial number, IMEI information) of the electronic device 100 , the user key may not be changed. However, when the user key is a password directly input by the user, the user key may be changed periodically or non-periodically. Accordingly, the user key management module 414 may manage version information about the user key changed by the user. That is, the user key management module 414 may manage the history of the changed user key and use the changed user key when decrypting the feature value into an image later.

According to another embodiment, when the user key is changed, the image encryption module 413 may re-encrypt previously encrypted images using the changed user key and transmit the encrypted images to an external server.

The thumbnail image generating module 415 may generate a thumbnail image corresponding to the encrypted image. That is, since the encrypted image is uploaded to the server 200 in a compressed state, the electronic device 100 does not need to store the high-capacity image as it is. Accordingly, the thumbnail image generating module 415 may generate a low-capacity thumbnail image corresponding to the encrypted image, and match the generated thumbnail image and identification information of the image to the electronic device 100 and store it. In this case, the thumbnail image generating module 415 divides the plurality of pixels constituting the image into groups, and generates a thumbnail image based on the representative values (eg, average, median, mode, etc.) of the pixels included in the group. can do. For example, when an encrypted image is transmitted to the server 200 and the thumbnail image generating module 415 generates a thumbnail image, the original image may be deleted.

That is, when a request for displaying a low-resolution image is received by the user, the electronic device 100 may search for the thumbnail image generated by the thumbnail image generating module 415 and provide it to the user, and a high-resolution image may be provided to the user by the user. When a request for displaying an image is received, the electronic device 100 downloads a feature value stored in the external server 200, inputs the downloaded feature value into a decoding model, restores it as an image, and then provides it to the user. have.

As shown in FIG. 4 , the decoding module 420 may include an image decoding module 421 , a decoding model 422 , and an image providing module 423 .

The image decoding module 421 may restore the feature value received from the external server 200 to the original image by using the decoding model 422 . In this case, the decryption model 422 may be a model trained to restore an image by using the feature value and the user key as input data.

Specifically, as shown in FIG. 5B , the decryption model 422 may be a model trained to acquire the image 510 using the feature value 530 and the user key 520 as input data. In this case, the decoding model 422 may be implemented as a deconvolution neural network. The deconvolution neural network is a neural network mirrored with the convolution neural network used as the encryption model 413 , and an unpooling layer may be used instead of a pooling layer. In this case, the deconvolution neural network may store pixel position values one by one, and perform an unpooling operation using the stored pixel position values. On the other hand, the deconvolution neural network used as the decryption model 422 disclosed in the present invention may always be trained in pairs with the convolution neural network used as the encryption model 413 .

Meanwhile, the decoding model 422 may use a deconvolution neural network, but this is only an example, and other neural networks such as a generative adversarial network (GAN) may be used. In particular, when GAN is used for image restoration, the electronic device 100 may restore an image having a higher resolution than the original image.

The image decryption module 421 may restore the image 510 by inputting the feature value 530 and the user key 520 downloaded from the server 200 into the decryption model 422 learned as input data.

The image providing module 423 may provide a restored image to the user. In this case, the image providing module 423 may provide the image restored to the full screen, but this is only an example and may provide the image restored to various screens such as a PIP screen. Also, the image providing module 423 may store the restored image in the memory 130 of the electronic device 100 .

7 is a block diagram schematically illustrating a configuration of a server according to an embodiment of the present disclosure. As shown in FIG. 7 , the server 200 includes a communication unit 210 , a memory 220 , and a processor 230 .

The communication unit 210 may communicate with external electronic devices. In this case, the communication unit 210 may receive identification information of the image and a feature value of the image from the external electronic devices 100 . Also, the communication unit 210 may transmit a feature value to the external electronic devices 100 in response to a search request or a download request.

The memory 220 may match and store identification information of an image transmitted from the external electronic devices 100 and a feature value of the image. In this case, the memory 220 may also store the information on the similarity of the image together with the identification information of the image and the feature value of the image.

The processor 230 may control the overall operation of the server 200 . In particular, when the identification information of the image and the characteristic value of the image are received from the electronic device 100 , the processor 230 may match the identification information of the image and the characteristic value of the image and store the matching information in the memory 220 . In this case, when version information of a user key corresponding to the acquired image among the plurality of user keys is further received from the electronic device 100 , the processor 230 may also store version information on the user key together.

In this case, the processor 230 may compare the obtained image based on the feature value with the similarity of other images, and classify the obtained image based on the similarity comparison. That is, the processor 230 may store a feature value for each image similarity.

When an image download request including identification information of an image is received from the electronic device 100 , the processor 230 may search for a feature value of an image to be downloaded based on identification information of the image. In addition, the processor 230 may control the communication unit 210 to transmit a feature value of the searched image to the electronic device 100 .

Meanwhile, the processor 230 may transmit only the feature value of the search image to the electronic device 100, but this is only an exemplary embodiment, and the communication unit ( 210) can be controlled.

When an image search request including a feature value or keyword of an image is received from the electronic device 100, the processor 230 may search for a search image related to the feature value or keyword of the received image based on the similarity of the image. .

8A and 8B are diagrams for explaining a method of encrypting and uploading an image using an encryption model, according to an embodiment of the present disclosure.

First, the electronic device 100 may receive an upload command for an image (S810). Specifically, as shown in (a) of FIG. 8B , the electronic device 100 may display an application screen including a plurality of images. In this case, the plurality of images may be images captured by the camera 170 , but this is only an example, and images received from an external device, images downloaded from the web, and screens of the electronic device 100 are captured. It may be various images such as images. When a user command for selecting one of a plurality of images is input, the electronic device 100 may display a menu 880 around the selected image as shown in (b) of FIG. 8B . When an item called “cloud storage” is selected from among the items included in the menu 880 , the electronic device 100 may receive an image upload command for saving the selected image to the external cloud server 200 . Meanwhile, as described above, an upload command for an image may be input using the menu 880, but this is only an exemplary embodiment, and an upload command for an image may be inputted in various ways. For example, when a preset button (eg, a button for executing an artificial intelligence program) included in the electronic device 100 is selected and a command for uploading an image is input, the electronic device 100 displays the image You can receive an upload command for .

The electronic device 100 may obtain a user key in response to an image upload command (S820). In this case, the electronic device 100 may obtain identification information of the electronic device 100 as the user key, and may obtain a password set by the user as the user key. In this case, the password set by the user may be a password for an application for uploading an image, or may be a password for a selected image. Also, the electronic device 100 may generate a new value based on the identification information of the electronic device 100 and the password set by the user.

The electronic device 100 may obtain a feature value by inputting an image and a user key into the encryption model (S830). Specifically, the electronic device 100 may generate input data using an image and a user key, and may obtain a feature value corresponding to the image by inputting the generated input data into an encryption model.

The electronic device 100 may transmit identification information and feature values of the image to the server 200 (S840). In this case, the image identification information may be various information such as an image name, an image creation date, an image creation location, and the like. In addition, the electronic device 100 may transmit only the identification information and the feature value of the image, but this is only an exemplary embodiment and may transmit version information of the user key together with the identification information and the feature value.

After transmitting the identification information and feature value of the image to the external server 200, the electronic device 100 may generate a thumbnail image (S850). That is, the electronic device 100 may compress and store a high-quality image in the external server 200, and may generate and store a low-quality thumbnail image. In this case, the electronic device 100 may match the low-quality thumbnail image with identification information of the image and store it.

The server 200 may match and store the image identification information and the feature value received from the electronic device 100 (S860). In addition, the server 200 may store the version information of the user key together with the image identification information and the feature value.

The server 200 may classify the image by comparing the similarity of the feature values (S870). Specifically, the server 200 may perform similarity comparison based on a feature value expressed as a vector, and classify and manage images through a classifier in a feature space. By classifying and managing images, it becomes possible to search for images based on feature values when searching for images later.

9A and 9B are diagrams for explaining a method of downloading and decoding an image using a decoding model, according to an embodiment of the present disclosure.

The electronic device 100 may receive an image download command (S910). Specifically, as shown in (a) of FIG. 9B , the electronic device 100 may display thumbnail images corresponding to a plurality of images stored in the server 200 . In this case, the thumbnail image may be a thumbnail image generated in step S850 of FIG. 8A . When one of the plurality of thumbnail images is selected, the electronic device 100 may display a menu 970 around the selected thumbnail image as shown in (b) of FIG. 9B . When a “download” item is selected among the items displayed on the menu 970 , the electronic device 100 may receive an image download command. However, as described above, receiving an image download command using the menu 970 is only an exemplary embodiment, and the image download command may be inputted using another method.

The electronic device 100 may transmit a download request signal for an image corresponding to the selected thumbnail image to the server 200 (S920). In this case, the download request signal may include identification information of an image corresponding to the selected thumbnail image.

The server 200 may search for a stored feature value matching the identification information of the image (S930). Specifically, the server 200 may search for a feature value matching the identification information of the received image among the feature values stored in the memory 220 .

The server 200 may transmit the searched feature value to the electronic device 100 (S940). In this case, the server 200 may transmit only the searched feature value to the electronic device 100 , but this is only an exemplary embodiment and may transmit version information of the user key together with the feature value to the electronic device 100 .

The electronic device 100 may restore the image by inputting the user key and the feature value into the decryption model ( S950 ). Specifically, the electronic device 100 may restore an image corresponding to the feature value by using the user key and the feature value as input data and inputting it into the decoding model. In this case, if the user key is different from the user key (or user key version) used for image encryption, the electronic device 100 cannot restore the image corresponding to the feature value, but the user key is used for image encryption. When the same as the user key, the electronic device 100 may restore an image corresponding to the feature value using the decryption model. In this case, the decryption model may be trained in pairs with the encryption model. In this case, the electronic device 100 may restore the original image, but this is only an exemplary embodiment, and may be restored to an image of higher quality than the original image by using the GAN.

The electronic device 100 may provide a restored image (S960). Specifically, the electronic device 100 may display the restored image on the screen shown in (b) of FIG. 9B . Also, the electronic device 100 may store the restored image in the memory 110 of the electronic device 100 .

10A and 10B are diagrams for explaining a method of searching for and decoding an image using a decoding model, according to an embodiment of the present disclosure.

The electronic device 100 may receive a search target image or keyword ( S1010 ). For example, as shown in (a) of FIG. 10B , the electronic device 100 may receive a keyword input into the search window 1070 . As another example, as shown in (b) of FIG. 10B , the electronic device 100 may receive a search target image for searching for a similar image through the menu 1080 . When the search target image is selected, the electronic device 100 may obtain a feature value corresponding to the search target image by inputting the selected search target image together with the user key into the encryption model.

The electronic device 100 may transmit a feature value or a keyword corresponding to the search target image to the server 200 (S1020).

The server 200 may search for a feature value corresponding to the search image to be searched based on the feature value or keyword corresponding to the search target image (S1030). Specifically, the server 200 may classify and store the feature values according to the degree of similarity. The server 200 may search for a feature value classified similarly to a feature value corresponding to the received search target image. Alternatively, when classifying the feature values according to the degree of similarity, the server 200 may classify the image according to an object included in the image. When an image is classified according to an object included in the image, the server 200 may search for a feature value corresponding to an image having an object corresponding to the received keyword.

The server 200 may transmit the searched feature value to the electronic device 100 (S1040). In this case, the server 200 may transmit the user key version information used when encrypting the feature value together with the searched feature value.

The electronic device 100 may restore the search image by inputting the searched feature value and the user key into the decryption model ( S1050 ). Specifically, the electronic device 100 may restore a search image corresponding to the searched feature value by inputting the user key and the feature value as input data into the decryption model.

The electronic device 100 may provide a search image (S1060).

That is, as described above, by using the feature values classified based on the similarity, the user can search for an image desired by the user from among the encrypted images stored in the server 200 .

11 is a flowchart illustrating a method for encrypting an image using an encryption model and uploading it to an external server, according to an embodiment of the present disclosure.

The electronic device 100 may acquire an image (S1110). In this case, the image may be an image captured by the camera 170 , but this is only an example, and an image received from an external device, an image downloaded from the web, an image captured by the screen of the electronic device 100 , etc. It may be various images such as

The electronic device 100 may receive a user command for uploading the acquired image (S1120). In this case, the electronic device 100 may receive a user command for uploading an image through the menu 880 shown in (b) of FIG. You can receive a user command to do so.

The electronic device 100 may obtain a feature value by inputting an image and a user key into the encryption model (S1130). Specifically, the electronic device 100 may generate input data using an image and a user key. Then, the electronic device 100 may obtain a feature value of a small capacity by inputting the generated input data to the learned encryption model.

The electronic device 100 may transmit identification information and feature values of the acquired image to the server 200 (S1140). The electronic device 100 may also transmit version information on the user key used when generating the feature value to the server 200 . Even after transmitting identification information and feature values of the acquired image to the server 200 , the electronic device 100 may generate and store a low-quality thumbnail image to display the image to the user.

As described above, by generating a feature value using an image and a user key and transmitting the generated feature value to the server 200 , the electronic device 100 encrypts the image and transmits the compressed feature value to the server 200 . can be uploaded to Accordingly, when uploading an image to the external server 200, not only personal information is protected, but also it is possible to upload a feature value of a small capacity to the outside.

12 is a block diagram illustrating a configuration of an electronic device (particularly, a processor) for learning and using an encryption model and a decryption model according to an embodiment of the present disclosure.

Referring to FIG. 12 , the processor 1200 may include at least one of a learning unit 1210 , an encryption unit 1220 , and a decryption unit 1230 . The processor 1200 of FIG. 12 may correspond to the processor 140 of the electronic device 100 of FIGS. 2 and 2 or the processor of the data learning server (not shown).

The learning unit 1210 may generate or train an encryption model for obtaining a feature value by inputting an image and a user key, and a decryption model for restoring an image by inputting the feature value and the user key. As an example, the learning unit 1210 may generate, learn, or update an encryption model for obtaining a feature value corresponding to the image by using the image and the user key as training data. As another example, the learning unit 1210 may generate, learn, or update a decoding model for restoring an image by using the feature value and the user key as training data. In this case, the learning unit 1210 may train the encryption model and the decryption model in pairs. That is, when the feature value obtained by inputting the user key and image into the encryption model as input data is input to the decryption model together with the user key, the encryption model and the decryption model are paired so that they can be restored to the image input to the encryption model. can learn

Also, the learning unit 1210 may re-learn the encryption model and the decryption model already learned using the general image and the random user key using the image and the user key stored in the electronic device 100 . That is, the learning unit 1210 may learn an encryption model and a decryption model optimized for the user while maintaining the security of user data.

The encryption unit 1220 may obtain a feature value corresponding to the image by using the image and the user key as input data of the learned encryption model. Also, the decoder 1230 may restore an image corresponding to the feature value by using the feature value and the user key as input data of the learned decryption model.

At least a portion of the learning unit 1210 and at least a portion of the encryption unit 1220/decryption unit 1230 may be implemented as a software module or manufactured in the form of at least one hardware chip and mounted in an electronic device. For example, at least one of the learning unit 1210 and the encryption unit 1220/decryption unit 1230 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or a conventional general-purpose processor It may be manufactured as a part of (eg, CPU or application processor) or a graphics-only processor (eg, GPU) and mounted in the aforementioned various electronic devices or encryption/decryption devices. In this case, the dedicated hardware chip for artificial intelligence is a dedicated processor specialized in probability calculation, and has higher parallel processing performance than the existing general-purpose processor, so that it can quickly process computational tasks in the field of artificial intelligence, such as machine learning. When the learning unit 1210 and the encryption unit 1220/decryption unit 1230 are implemented as a software module (or a program module including instructions), the software module is a computer-readable non-transitory reading It may be stored in a non-transitory computer readable media. In this case, the software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, a part of the software module may be provided by an operating system (OS), and the other part may be provided by a predetermined application.

In this case, the learning unit 1210 and the encryption unit 1220/decryption unit 1230 may be mounted on one electronic device or may be mounted on separate electronic devices, respectively. For example, one of the learning unit 1210 and the encryption unit 1220/decrypting unit 1230 may be included in the electronic device 100 , and the other may be included in an external server. As another example, one of the encryption unit 1220 and the decryption unit 1230 may be included in the first electronic device 100-1, and the other may be included in the second electronic device 100-2. In addition, the learning unit 1210 and the encryption unit 1220 / decryption unit 1230 provide the model information built by the learning unit 1210 to the encryption unit 1220 / decryption unit 1230 through a wired or wireless connection. Alternatively, data input to the learning unit 1210 may be provided to the learning unit 1210 as additional learning data.

13A is a block diagram of a learning unit 1210 , an encryption unit 1220 , and a decryption unit 1230 according to various embodiments of the present disclosure.

Referring to (a) of FIG. 13A , the learning unit 1210 according to some embodiments may include a training data acquiring unit 1210 - 1 and a model learning unit 1210 - 4 . In addition, the learning unit 1210 may further selectively include at least one of the training data preprocessor 1210 - 2 , the training data selector 1210 - 3 , and the model evaluator 1210 - 5 .

The training data acquisition unit 1210-1 may acquire training data required for the encryption model and the decryption model. In an embodiment of the present disclosure, the training data acquisition unit 1210-1 may acquire a general image and a random user key as training data. As another example, the learning data acquisition unit 1210-1 may acquire an image stored in the electronic device 100 and a user key of the electronic device 100 as training data for user optimization.

The model learning unit 1210-4 uses the training data to obtain a feature value corresponding to the image by the encryption model using the user key, and the decryption model to restore the image corresponding to the feature value using the user key. can be learned As an example, the model learning unit 1210-4 inputs a user key and an image to the encryption model to obtain a feature value, and inputs the acquired feature value and the user key back to the decryption model to restore the image. and training a decryption model.

In addition, the model learning unit 1210-4 includes the number of layers constituting the encryption model according to the type of input data, the kernel size and number of channels of each layer, the number of pooling layers, the pooling size, and the skip connection (skip). connection), the encryption model can be trained by changing whether the Intermediate Feature and Top-down Feature are applied. Also, the model learning unit 1210 - 4 may train the decryption model in pairs with the learned encryption model. Since the encryption model and the decryption model are trained in pairs, the learning unit 1210 will be described below based on the encryption model.

In particular, the model learning unit 1210 - 4 may learn the encryption model through supervised learning using at least a portion of the training data as a criterion for determination. Alternatively, the model learning unit 1210-4 is, for example, through unsupervised learning of discovering a criterion for obtaining a feature value by self-learning using training data without any guidance, for example, an encryption model. can be learned Also, the model learning unit 1210 - 4 may train the encryption model through, for example, reinforcement learning using feedback on whether the result of the situation determination according to the learning is correct. Also, the model learning unit 1210 - 4 may train the encryption model using, for example, a learning algorithm including error back-propagation or gradient descent.

Also, the model learning unit 1210 - 4 may learn a selection criterion for which training data to use in order to obtain a feature value by using the input data.

When a plurality of pre-built encryption models exist, the model learning unit 1210 - 4 may determine an encryption model having a high correlation between the input training data and the basic training data as the encryption model to be learned. In this case, the basic learning data may be pre-classified for each type of data, and the encryption model may be previously built for each type of data. For example, the basic training data is pre-classified by various criteria such as the region where the training data is generated, the time when the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, the type of object in the training data, etc. may have been

When the encryption model is learned, the model learning unit 1210 - 4 may store the learned encryption model. In this case, the model learning unit 1210 - 4 may store the learned encryption model in the memory 130 of the electronic device 100 . Alternatively, the model learning unit 1210 - 4 may store the learned encryption model in a memory of a server connected to the electronic device 100 through a wired or wireless network.

The learning unit 1210 includes a training data preprocessing unit 1210-2 and a training data selection unit 1210-3 in order to improve the processing result of the encryption model or to save resources or time required for generating the encryption model. It may include more.

The learning data preprocessor 1210 - 2 may preprocess the acquired data so that the acquired data can be used for learning for acquiring a feature value. The training data preprocessor 1210 - 2 may process the acquired data into a preset format so that the model learner 1210 - 4 can use the acquired data for learning to acquire a feature value. As an example, the training data preprocessor 1210 - 2 may process a user key and an image as input data as shown in FIGS. 6A and 6B .

The training data selection unit 1210-3 may select data required for learning from the data acquired by the training data acquisition unit 1210 - 1 or data preprocessed by the training data preprocessor 1210 - 2 . The selected training data may be provided to the model learning unit 1210 - 4 . The learning data selector 1210 - 3 may select learning data necessary for learning from among the acquired or pre-processed data according to a preset selection criterion. Also, the training data selection unit 1210 - 3 may select training data according to a preset selection criterion by learning by the model learning unit 1210 - 4 .

The learning unit 1210 may further include a model evaluation unit 1210 - 5 in order to improve the processing result of the encryption model.

The model evaluation unit 1210-5 inputs evaluation data to the encryption model, and when a processing result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit 1210-4 may re-learn it. . In this case, the evaluation data may be predefined data for evaluating the encryption model.

For example, the model evaluation unit 1210 - 5 satisfies a predetermined criterion when, among the processing results of the learned encryption model for the evaluation data, the number or ratio of evaluation data for which the processing result is not accurate exceeds a preset threshold. It can be evaluated as not being done.

On the other hand, when there are a plurality of learned encryption models, the model evaluation unit 1210-5 evaluates whether a predetermined criterion is satisfied for each learned encryption model, and determines a model that satisfies the predetermined criterion as the final encryption model. can In this case, when there are a plurality of models satisfying the predetermined criteria, the model evaluation unit 1210 - 5 may determine one or a predetermined number of models preset in the order of the highest evaluation score as the final encryption model.

Also, as described above, when the encryption model is learned, the learning unit 1210 may learn the decryption model to be mirrored in the encryption model.

Referring to (b) of FIG. 13A , the encryption unit 1220 according to some exemplary embodiments may include an input data acquisition unit 1220-1 and a feature value providing unit 1220-4. Also, the encryption unit 1220 may selectively further include at least one of an input data preprocessor 1220 - 2 , an input data selector 1220 - 3 , and a model updater 1220 - 5 .

The input data acquisition unit 1220-1 may acquire data necessary for acquiring a feature value. Specifically, the input data acquisition unit 1220-1 may acquire an image and a user key of the electronic device 100 as input data.

The feature value providing unit 1220 - 4 may obtain a feature value corresponding to the image by applying the data obtained by the input data obtaining unit 1220-1 to an encryption model learned as an input value. The feature value providing unit 1220 - 4 may provide feature values corresponding to the image and the user key. The feature value providing unit 1220-4 may obtain a feature value by applying the data selected by the input data preprocessing unit 1220-2 or the input data selecting unit 1220-3 to the encryption model as an input value, which will be described later. have.

The encryption unit 1220 includes the input data preprocessor 1220-2 and the input data selection unit 1220-3 to improve the processing result of the encryption model or to save resources or time for providing the processing result. It may include more.

The input data preprocessor 1220 - 2 may preprocess the acquired data so that the acquired data can be used to acquire a feature value. The input data preprocessor 1220 - 2 may process the input data into a predefined format so that the feature value providing unit 1220 - 4 can use the input data to obtain the feature value. For example, the input data preprocessor 1220 - 2 may process the input data as shown in FIGS. 6A and 6B .

The input data selection unit 1220 - 3 may select data required for acquiring a feature value from the data acquired by the input data acquiring part 1220-1 or the data preprocessed by the input data preprocessing part 1220 - 2 . The selected data may be provided to the feature value providing unit 1220 - 4 . The input data selection unit 1220 - 3 may select some or all of the acquired or pre-processed data according to a preset selection criterion for acquiring a feature value. Also, the input data selection unit 1220 - 3 may select data according to a predetermined selection criterion by learning by the model learning unit 1210 - 4 .

The model updater 1220 - 5 may control the encryption model to be updated based on the evaluation of the feature value provided by the feature value provider 1220 - 4 . For example, the model updating unit 1220-5 provides the feature value provided by the feature value providing unit 1220-4 to the model learning unit 1210-4, so that the model learning unit 1210-4 is You can request that the cryptographic model be further trained or updated.

Referring to (c) of FIG. 13A , the decoding unit 1230 according to some embodiments may include an input data obtaining unit 1230-1 and an image providing unit 1230-4. Also, the decoder 1230 may further selectively include at least one of an input data preprocessor 1230 - 2 , an input data selector 1230 - 3 , and a model updater 1230 - 5 .

The input data acquisition unit 1230-1 may acquire data necessary for image restoration. For example, the input data obtaining unit 1230-1 may obtain a user key and a feature value obtained from the server 200 as input data. The image providing unit 1230-4 may provide an image corresponding to a feature value by applying the data acquired by the input data acquiring unit 1230-1 to a decoding model learned as an input value. The image providing unit 1230 - 4 may restore an image corresponding to the feature value and the user key. The image providing unit 1230 - 4 may restore an image by applying the data selected by the input data preprocessor 1230 - 2 or the input data selector 1230 - 3 to the decoding model as an input value, which will be described later.

The decoding unit 1230 includes an input data preprocessor 1230-2 and an input data selection unit 1230-3 to improve the processing result of the decoding model or to save resources or time for providing the processing result. It may include more.

The input data preprocessor 1230 - 2 may preprocess the acquired data so that the acquired data can be used for image restoration. The input data preprocessor 1230 - 2 may process the input data into a predefined format so that the image providing unit 1230 - 4 can use the input data for image restoration.

The input data selection unit 1230 - 3 may select data required for image restoration from among the data acquired by the input data acquiring unit 1230-1 or the data preprocessed by the input data preprocessing unit 1230 - 2 . The selected data may be provided to the image providing unit 1230 - 4 . The input data selection unit 1230 - 3 may select some or all of the acquired or pre-processed data according to a preset selection criterion for image restoration.

The model updating unit 1230 - 5 may control the decoding model to be updated based on the evaluation of the image provided by the image providing unit 1230 - 4 . For example, the model update unit 1230 - 5 provides the image provided by the image providing unit 1230 - 4 to the model learner 1210 - 4 , so that the model learner 1210 - 4 performs the decoding model. may request further learning or renewal.

13B is a diagram illustrating an example in which the electronic device 100 and an external server S work together to learn an encryption model and obtain a feature value according to an embodiment.

Referring to FIG. 13B , the external server S may learn a criterion for obtaining the feature value of the encryption model, and the electronic device 100 provides an encryption model generated based on the learning result by the server S. can be used to obtain feature values.

In this case, the model learning unit 1210 - 4 of the server S may perform the function of the learning unit 1210 illustrated in FIG. 12 . In addition, the feature value providing unit 1220 - 4 of the electronic device 100 applies the data selected by the input data selection unit 1220 - 3 to the encryption model generated by the server S to provide a feature corresponding to the image. value can be obtained.

Alternatively, the feature value providing unit 1220 - 4 of the electronic device 100 receives the encryption model generated by the server S from the server S, and uses the received encryption model to provide a feature value corresponding to the image. can be obtained. In this case, the feature value providing unit 1220 - 4 of the electronic device 100 applies the input data selected by the input data selection unit 1220 - 3 to the encryption model received from the server S to correspond to the image. feature values can be obtained.

13C is a diagram illustrating an example in which the electronic device 100 and an external server S work together to learn a decoding model and restore an image, according to an embodiment.

Referring to FIG. 13C , the external server S may learn a criterion for the decoding model to reconstruct an image, and the electronic device 100 uses the decoding model generated based on the learning result by the server S. can be used to restore the image.

In this case, the model learning unit 1210 - 4 of the server S may perform the function of the learning unit 1210 illustrated in FIG. 12 . In addition, the image providing unit 1230 - 4 of the electronic device 100 applies the data selected by the input data selection unit 1230 - 3 to the decoding model generated by the server S to obtain an image corresponding to the feature value. can be obtained.

Alternatively, the image providing unit 1230 - 4 of the electronic device 100 receives the decryption model generated by the server S from the server S, and uses the received decryption model to generate an image corresponding to the feature value. can be restored In this case, the image providing unit 1230 - 4 of the electronic device 100 applies the input data selected by the input data selection unit 1230 - 3 to the decoding model received from the server S, The feature value can be restored.

14 to 15 are flowcharts of a network system using an encryption model or a decryption model, according to various embodiments of the present disclosure. 14 and 15 , a network system using an encryption model or a decryption model may include first components 1401 and 1501 and second components 1402 and 1502 .

Here, the first components 1401 and 1401 may be the electronic device 100 , and the second components 1402 and 1502 may be the server S in which an encryption model or a decryption model is stored. Alternatively, the first components 1401 and 1501 may be general-purpose processors, and the second components 1402 and 1502 may be artificial intelligence-only processors. Alternatively, the first components 1401 and 1501 may be at least one application, and the second components 1402 and 1502 may be an operating system (OS). That is, the second components 1402 and 1502 are more integrated than the first components 1401 and 1501, have a smaller delay, have superior performance, or have more resources than the first components 1401 and 1501. It may be a component capable of processing many operations required when generating, updating, or applying an encryption model or a decryption model more quickly and effectively than the first components 1401 and 1501 .

In this case, an interface for transmitting/receiving data between the first components 1401 and 1501 and the second components 1402 and 1502 may be defined.

For example, an API (application program interface) having training data to be applied to an encryption model or a decryption model as a factor value (or a parameter value or a transfer value) may be defined. An API is a set of subroutines or functions that can be called for any processing of any one protocol (eg, the protocol defined in the electronic device 100) to another protocol (eg, the protocol defined in the server S). can be defined. That is, an environment in which an operation of another protocol can be performed in one protocol can be provided through the API.

The third components 1403 and 1503 may be implemented as an external server or a cloud device that can match and store a feature value corresponding to an image and identification information of the image.

In FIG. 14 , the first component 1401 may receive an upload command for an image ( S1410 ). In this case, the upload command for the image may be obtained through the menu for the image, but this is only an example and may be obtained by other methods.

The first component 1401 may obtain a user key (S1420). In this case, the user key may be identification information of the electronic device 100 or a password input by the user.

The first component 1401 may transmit the image and the user key to the second component 1402 (S1430). In this case, when a plurality of user keys exist, the first component 1401 may transmit version information on a user key corresponding to an image among the plurality of user keys together.

The second component 1402 may obtain a feature value by inputting an image and a user key into the encryption model (S1440). In this case, the encryption model may be a model trained to obtain a feature value corresponding to the image by using the image and the user key as input data.

The second component 1402 may transmit a feature value corresponding to the image to the first component 1401 (S1450).

The first component 1401 may transmit identification information of the image and a feature value corresponding to the image to the third component 1403 (S1460). In this case, the first component 1401 may transmit version information on the user key together to the third component 1403 .

The third component 1403 may match and store the identification information of the image transmitted from the first component 1401 with a feature value corresponding to the image (S1470). In this case, the third component 1403 may store version information about the user key together.

Also, the first component 1401 may generate a thumbnail image corresponding to the image (S1480). When a user command for displaying an image is input, the first component 1401 may display the generated thumbnail image.

In FIG. 15 , the third component 1503 may match and store identification information and feature values ( S1510 ).

The first component 1501 may receive a download command for the image (S1520). In this case, the download command may be input through a menu for an image, but this is only an exemplary embodiment and may be inputted by other methods.

The first component 1501 may transmit a download request signal to the third component 1503 (S1530). In this case, the download request signal may include identification information of an image to be downloaded.

The third component 1503 may search for a feature value corresponding to the identification information of the image (S1540). Specifically, it is possible to search for a feature value matching the identification information of the image among a plurality of feature values stored in the third component 1503 .

The third component 1503 may transmit the searched feature value to the first component 1501 (S1550). In this case, the third component 1503 may transmit the retrieved feature value to the first component 1501 , but this is only an example, and the searched feature value may be directly transmitted to the second component 1502 . .

The first component 1501 may transmit the user key and the feature value to the second component 1502 (S1560).

The second component 1502 may restore the image by inputting the user key and the feature value into the decryption model (S1570). In this case, the decryption model is a model trained to restore an image using a user key and a feature value as input data, and may be learned in pairs with the encryption model.

The second component 1502 may transmit the restored image to the first component 1501 (S1580).

The first component 1501 may provide a restored image (S1590). Alternatively, the first component 1501 may store the restored image.

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media readable by a machine (eg, a computer). As a device capable of calling and operating according to the called command, it may include an electronic device (eg, the electronic device 100) according to the disclosed embodiments. When the command is executed by the processor, the processor directly Alternatively, a function corresponding to the instruction may be performed using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter. , may be provided in the form of a non-transitory storage medium, where 'non-transitory' means that the storage medium does not include a signal and is tangible, but data is stored on the storage medium It does not distinguish between semi-permanent or temporary storage.

According to an exemplary embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be It may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, parallel, iteratively, or heuristically, or at least some operations are executed in a different order, are omitted, or other operations are added. can be

110: communication unit 120: display
130: memory 140: processor
150: user input unit 160: audio output unit
170: camera

Claims (20)

In a control method of an electronic device for encrypting an image using an artificial intelligence neural network model and uploading it to an external server
acquiring an image;
receiving a user command for uploading the acquired image to the external server;
In response to the user command, a feature value corresponding to the acquired image by inputting the acquired image and the user key of the electronic device into a neural network model trained to acquire a feature value by using an image and a user key as input data obtaining a; and
and transmitting the identification information of the acquired image and the feature value to the external server. According to claim 1,
The user key of the electronic device,
The control method according to claim 1, wherein the password is one of a password input by a user of the electronic device and identification information of the electronic device. According to claim 1,
The transmitting step is
When there are a plurality of user keys of the electronic device, version information of a user key corresponding to the obtained image among the plurality of user keys is transmitted to the external server together with identification information and the characteristic value of the obtained image A control method, characterized in that. According to claim 1,
obtaining a thumbnail image corresponding to the obtained image in response to the user command;
matching and storing the obtained thumbnail image and identification information of the obtained image; and
A control method comprising a; deleting the acquired image. 5. The method of claim 4,
and providing the obtained thumbnail image when a user command for displaying the obtained image is received from a user. According to claim 1,
The external server,
matching and storing the identification information of the acquired image with the feature value, comparing the similarity between the acquired image and another image based on the feature value, and classifying the acquired image based on the similarity comparison Characterized control method. 7. The method of claim 6,
receiving keywords;
receiving a user command for searching for an image related to the keyword;
transmitting a signal for requesting a search related to the keyword to the external server; and
and receiving, from the external server, a feature value corresponding to at least one search image related to the keyword among images classified based on the similarity comparison in response to the signal. 8. The method of claim 7,
Restoring the at least one search image by inputting a feature value corresponding to the at least one search image and a user key of the electronic device into a decoding model that is trained to restore an image by using a feature value and a user key as input data Control method comprising; According to claim 1,
transmitting a signal requesting a feature value corresponding to the acquired image to the external server when a user's download command for receiving a feature value corresponding to the acquired image is input;
receiving a feature value corresponding to the acquired image from the external server in response to the signal; and
Restoring the acquired image by inputting a feature value corresponding to the acquired image and a user key of the electronic device into a decoding model trained to restore an image by using the feature value and the user key as input data; control method including. 10. The method of claim 9,
The receiving step is
Receives version information of a user key of the electronic device used to encrypt the acquired image together with a feature value corresponding to the acquired image from the external server,
The restoration step is
A control method for restoring the acquired image by inputting a feature value corresponding to the acquired image and a user key of the electronic device corresponding to the version information to the decoding model. In an electronic device,
communication department;
display;
a memory for storing a neural network model trained to obtain a feature value by using an image and a user key as input data and a program for performing an operation of the electronic device;
a processor electrically connected to the communication unit, the display, and the memory to control the electronic device; and
The processor is
Obtaining an input signal according to a user command for uploading the acquired image to an external server,
In response to the input signal, input the acquired image and the user key of the electronic device to the neural network model to obtain a feature value corresponding to the acquired image,
The electronic device for controlling the communication unit to transmit the identification information of the acquired image and the feature value to the external server. 12. The method of claim 11,
The user key of the electronic device,
The electronic device of claim 1, wherein the password is one of a password input by a user of the electronic device and identification information of the electronic device. 12. The method of claim 11,
The processor is
When there are a plurality of user keys of the electronic device, version information of a user key corresponding to the obtained image among the plurality of user keys is transmitted to the external server together with identification information and the characteristic value of the obtained image An electronic device, characterized in that for controlling the communication unit to do so. 12. The method of claim 11,
The processor is
In response to the user command, obtain a thumbnail image corresponding to the obtained image,
Matching the obtained thumbnail image and identification information of the obtained image and storing it in the memory,
and deleting the acquired image from the memory. 15. The method of claim 14,
The processor is
The electronic device of claim 1, wherein when a user command for displaying the acquired image is received from a user, the display is controlled to provide the acquired thumbnail image. 12. The method of claim 11,
The identification information of the acquired image and the characteristic value are matched and stored in the external server,
The external server,
and comparing the similarity between the acquired image and another image based on the feature value, and classifying the acquired image based on the similarity comparison. 17. The method of claim 16,
The processor is
Obtaining an input signal according to a user command for searching for an image related to the input keyword,
controlling the communication unit to transmit a signal requesting a search related to the keyword to the external server,
and receiving, from the external server, a feature value corresponding to at least one search image related to the keyword among images classified based on the similarity comparison through the communication unit. 18. The method of claim 17,
The memory is
Stores the decryption model trained to restore the image using the feature value and the user key as input data,
The processor is
and restoring the at least one search image by inputting a feature value corresponding to the at least one search image and a user key of the electronic device into the decryption model. 12. The method of claim 11,
The memory is
Stores the decryption model trained to restore the image using the feature value and the user key as input data,
The processor is
When a user's download command for receiving a feature value corresponding to the acquired image is input, the communication unit is controlled to transmit a signal requesting a feature value corresponding to the acquired image to the external server,
Receives a feature value corresponding to the acquired image from the external server through the communication unit,
and reconstructing the acquired image by inputting a feature value corresponding to the acquired image and a user key of the electronic device to the decoding model. 20. The method of claim 19,
Version information of the user key of the electronic device used to encrypt the acquired image is received together with a feature value corresponding to the acquired image from the external server,
The processor is
The electronic device restores the acquired image by inputting a feature value corresponding to the acquired image and a user key of the electronic device corresponding to the version information in the decryption model.

Images