WO2022119128A1

WO2022119128A1 - Electronic device and control method therefor

Info

Publication number: WO2022119128A1
Application number: PCT/KR2021/015233
Authority: WO
Inventors: 탁윤식; 신재욱; 여해동; 임애란; 임준호
Original assignee: 삼성전자주식회사
Priority date: 2020-12-03
Filing date: 2021-10-27
Publication date: 2022-06-09
Also published as: KR20220078176A

Abstract

An electronic device and a control method therefor are provided. An electronic device control method comprises the steps of: acquiring an image including a document with a modified form; inputting the acquired image in a trained neural network model to acquire a mapping table for indicating the positional correlation between pixels of a document whose form included in the image is modified and pixels of a result image in which the form of the document has been restored; estimating an aspect ratio of the document on the basis of the position information about the pixels mapped to the mapping table; adjusting the ratio of the mapping table in the estimated aspect ratio; and restoring the shape of the document on the basis of the adjusted mapping table.

Description

Electronic device and control method thereof

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device capable of reconstructing a document with a deformed shape included in an image, and a control method thereof.

Recently, there is an increasing demand for photographing a high-resolution image by using an electronic device such as a smart phone and scanning a document included in the photographed high-resolution image.

However, when photographing a document, a document whose shape is deformed due to wrinkling or folding of the document may be photographed. Conventionally, in order to solve this problem, there has been a technique for restoring a document whose shape is deformed through image processing to its original shape.

However, in the case of a technique of restoring the shape of a document through conventional image processing, since the original horizontal/vertical ratio of the document cannot be known, the shape of the document is restored at a ratio other than the original horizontal/vertical ratio of the document There were limits to what could be done.

An object of the present disclosure is to provide an electronic device capable of restoring a circular shape of a document by estimating a horizontal and vertical ratio of the document, and a method for controlling the same.

According to an embodiment of the present disclosure, a method of controlling an electronic device includes: acquiring an image including a document with a deformed shape; inputting the obtained image into a trained neural network model to obtain a mapping table indicating a positional association between pixels of a document whose shape is deformed included in the image and pixels of a result image of which the shape of the document is restored; estimating a horizontal/vertical ratio of the document based on location information of pixels mapped to the mapping table; adjusting a ratio of the mapping table to the estimated horizontal and vertical ratio; and restoring a document based on the adjusted mapping table.

Meanwhile, according to an embodiment of the present disclosure, an electronic device includes: a camera; a memory including at least one instruction; and a processor, wherein the processor executes the at least one instruction, obtains an image including a document whose shape is deformed through the camera, and inputs the obtained image to a trained neural network model to A mapping table indicating a positional correlation between pixels of the document whose shape included in the image is deformed and pixels of the resulting image in which the shape of the document is restored is obtained, and based on the location information of pixels mapped to the mapping table, The ratio of the mapping table is adjusted based on the estimated horizontal and vertical ratio by estimating the horizontal and vertical ratio, and the document is restored based on the adjusted mapping table.

According to an embodiment of the present disclosure as described above, even when the electronic device captures a document with a deformed shape, the electronic device can restore the deformed document at a more accurate ratio close to reality.

1 is a block diagram illustrating a configuration for reconstructing a shape-deformed document, according to an embodiment of the present disclosure;

2 is a diagram for explaining a method for a mapping table generation module to generate a mapping table using a neural network model, according to an embodiment of the present disclosure;

3 is a diagram for explaining a method for a document ratio estimation module to estimate an actual ratio of a document according to an embodiment of the present disclosure;

4 is a view for explaining a method for a document restoration module to restore the shape of a document based on an estimated actual ratio, according to an embodiment of the present disclosure;

5 is a view for explaining an image captured by a camera, a document image restored by a conventional method, and a document image restored by a method according to an embodiment of the present disclosure;

6 is a flowchart for explaining a method of controlling an electronic device, according to an embodiment of the present disclosure;

7 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Since the present embodiments can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, like reference numerals may be used for like components.

In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted.

In addition, the following examples may be modified in various other forms, and the scope of the technical spirit of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to more fully and complete the present disclosure, and to fully convey the technical spirit of the present disclosure to those skilled in the art.

The terms used in the present disclosure are used only to describe specific embodiments, and are not intended to limit the scope of rights. The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this disclosure, expressions such as "A or B," "at least one of A and/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as “first,” “second,” “first,” or “second,” may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element).

On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression “configured to (or configured to)” as used in this disclosure, depending on the context, for example, “suitable for,” “having the capacity to” ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware.

Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C" refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

In an embodiment, a 'module' or 'unit' performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module and implemented with at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware.

Meanwhile, various elements and regions in the drawings are schematically drawn. Accordingly, the technical spirit of the present invention is not limited by the relative size or spacing drawn in the accompanying drawings.

Meanwhile, the electronic device according to various embodiments of the present disclosure may include, for example, at least one of a smart phone, a tablet PC, a desktop PC, a laptop PC, and a wearable device. A wearable device may be an accessory (e.g., watch, ring, bracelet, anklet, necklace, eyewear, contact lens, or head-mounted-device (HMD)), a textile or clothing integral (e.g. electronic garment); It may include at least one of body-attached (eg, skin pad or tattoo), or bioimplantable circuitry.

In some embodiments, the electronic device may include, for example, a television, digital video disk (DVD) player, audio, refrigerator, air conditioner, vacuum cleaner, oven, microwave oven, washing machine, air purifier, set-top box, home automation control panel, Secure at least one of a control panel, media box (eg Samsung HomeSync ^TM , Apple TV ^TM , or Google TV ^TM ), game console (eg Xbox ^TM , PlayStation ^TM ), electronic dictionary, electronic key, camcorder, or electronic picture frame. may include

Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings. 1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure. The electronic device 100 includes a camera 110 , a memory 120 , and a processor 130 . In this case, the electronic device 100 may be implemented as a smart phone. However, the electronic device 100 according to the present disclosure is not limited to a specific type of device, and may be implemented as various types of electronic devices 100 such as a tablet PC and a digital camera.

The camera 110 may capture an image. In particular, the camera 110 may capture an image including a document with a deformed shape. In this case, the deformed document may refer to a document that is wrinkled or folded and cannot maintain its original shape.

Also, the camera 110 may include a plurality of lenses different from each other. Here, that the plurality of lenses are different from each other may include a case in which a field of view (FOV) of each of the plurality of lenses is different from each other and a case in which positions at which each of the plurality of lenses are disposed are different, and the like.

The memory 120 may store data necessary for a module for restoring a document with a deformed shape included in the photographed image to perform various operations. The module for restoring the shape of the document may include a document detection module 131 , a mapping table generation module 133 , a document ratio estimation module 137 , and a document restoration module 139 . Also, the memory 110 may store the neural network model 135 for obtaining a mapping table indicating positions of pixels of a document included in an image.

Meanwhile, the memory 120 may include a non-volatile memory capable of maintaining stored information even when power supply is interrupted, and a volatile memory requiring continuous power supply to maintain the stored information. Data for performing various operations of the module for restoring the shape of a document may be stored in a non-volatile memory. In addition, the neural network model 135 for obtaining a mapping table indicating positions of pixels of a document included in an image may also be stored in the nonvolatile memory.

Also, the memory 120 may include at least one buffer for temporarily storing an image frame acquired through the camera 110 .

The processor 130 may be electrically connected to the memory 120 to control overall functions and operations of the electronic device 100 .

In the processor 130, when a camera application is executed or a user command for restoring the shape of a document included in the captured image is input, the module for restoring the shape of the document stored in the non-volatile memory performs various operations. Data to be used may be loaded into the volatile memory. Then, the processor 130 may load the neural network model for obtaining the mapping table into the volatile memory. The processor 130 may perform various operations through various modules and neural network models based on data loaded into the volatile memory. Here, the loading refers to an operation of loading and storing data stored in the nonvolatile memory into the volatile memory so that the processor 130 can access it.

The processor 130 may use the camera 110 to obtain an image including a document whose shape is deformed. In this case, the processor 130 may acquire an image through the camera 110 , but this is only an exemplary embodiment and may acquire an image including a document whose shape is deformed from the outside.

When a user command for restoring the shape of a document included in the captured image is input, the processor 130 may detect a document in the captured image through the document detection module 131 . In this case, the document is a document with a deformed shape, and may be a folded or bent document. In addition, the document may be, for example, printed paper, a receipt, a book, a business card, etc., but is not limited thereto.

In this case, the document detection module 131 may detect a document in the image using the learned object recognition model. In this case, the object recognition model may be implemented as a convolutional neural network (CNN) model, but this is only an example, and a deep neural network (DNN), a recurrent neural network (RNN) and generative adversarial networks (GANs).

Also, when a document in the image is detected, the processor 130 may display a bounding box around the detected document in the image displayed on the display. In this case, the bounding box is a box surrounding the changed shape of the document, and may have a rectangular shape including the maximum horizontal length and the maximum vertical length of the changed shape document.

On the other hand, when a plurality of objects as well as a document are included in the captured image, the processor 130 may detect the document included in the image through the document detection module 131 , but this is only an embodiment, and the captured image When only a document is included in the image, the processor 130 may not separately detect the document.

The processor 130 may generate a mapping table for restoring the shape of a document through the mapping table generation module 133 . Specifically, when the processor 130 inputs the image (or the image of the document included in the image) obtained through the mapping table generation module 133 to the trained neural network model 135 to restore the deformed document A mapping table indicating positions of pixels in a document included in an image may be obtained.

At this time, as shown in FIG. 2 , the processor 130 inputs the information 210 on the image to the first network 220 of the neural network model 135 to include depth information of the document whose shape is deformed. A dimensional image 230 may be acquired. The shape of the document included in the three-dimensional image 230 is the same as the shape of the document included in the photographed image 210, but since it includes depth information as well as x-coordinate values and y-coordinate values, the three-dimensional coordinate value is included can do. As shown in FIG. 2 , the depth information of the 3D image 230 may be expressed in gray scale. Meanwhile, information on the 3D image 230 may be stored in a buffer.

In addition, as shown in FIG. 2 , the processor 130 inputs the three-dimensional image 230 to the second network 240 of the neural network model 135 and creates a mapping table 250 for restoring the shape of the document. can be obtained In this case, the mapping table 250 may be a table indicating the positional correlation between the pixels of the input image and the pixels of the reconstructed result image.

The horizontal/vertical ratio of the mapping table 250 may correspond to the horizontal/vertical ratio of the image 210 or the horizontal/vertical ratio of a rectangular bounding box including a document with a deformed shape among the images 210 . For example, when only a document is included in the image 210 , the horizontal/vertical ratio of the mapping table 250 may correspond to the horizontal/vertical ratio of the image 210 , and the image 210 includes at least one document as well as a document. When an object is included, the horizontal/vertical ratio of the mapping table 250 may correspond to the horizontal/vertical ratio of a bounding box surrounding the document. That is, since the horizontal/vertical ratio of the image 210 or the horizontal/vertical ratio of a rectangular bounding box including a deformed document among the images 210 is different from the original horizontal/vertical ratio of the document, the mapping table ( 250) may be different from the original aspect ratio of the document.

Meanwhile, the first network 220 may be trained using, as input data and output data, a learning image including a document with a deformed shape and a 3D image obtained by photographing the learning image with a depth camera. Also, the second network 240 may be learned by using, as input data and output data, a mapping table corresponding to a three-dimensional image obtained by photographing with a depth camera and a learning image, respectively. In this case, the first network 220 and the second network 240 may be implemented as U-net, DeepLab, PSPNet (Pyramid Scene Parsing Network), FCN (Fully Convolutional Networks), etc. that can be utilized for image conversion/generation. have.

On the other hand, since the horizontal/vertical ratio of the mapping table 250 is different from the original horizontal/vertical ratio of the document, when the shape of the document is restored using the mapping table 250, the original horizontal/vertical ratio of the document is not restored. it may not be Accordingly, the processor 130 may restore the shape of the document by estimating the original aspect ratio of the document before restoring the shape of the document using the mapping table.

The processor 130 may estimate the original aspect ratio of the document through the document ratio estimation module 137 . In particular, the processor 130 obtains 3D position information of pixels mapped to the mapping table 250 using the 3D image 230 , and based on the obtained 3D position information of the pixels, the original width / The vertical ratio can be estimated. Specifically, the processor 130 compares the 3D coordinate value of the pixel of the 3D image 230 mapped to each point included in the horizontal axis of the mapping table 250 and the 3D coordinate value of the pixel mapped to the previous point. You can estimate the width by adding the difference. In addition, the processor 130 estimates the vertical length by adding the difference between the 3D coordinate value of the pixel of the 3D image mapped to each point included in the vertical axis of the mapping table and the 3D coordinate value of the pixel mapped to the previous point can do.

More specifically, the mapping table 250 includes a plurality of points (

,

), N: the horizontal length of the mapping table, M: the vertical length of the mapping table). In this case, a point of the mapping table 250 may be mapped to each pixel of the 3D image 230 as shown in FIG. 3 .

At this time, the point included in the horizontal axis of the mapping table 250 of row j (

inside

), the original horizontal length of the document may be estimated by adding the difference between the 3D coordinate value of the pixel of the 3D image 230 mapped to each and the 3D coordinate value of the pixel mapped to the previous point. That is, by Equation 1 below, the original horizontal length (

) can be estimated. The horizontal length of the document may be calculated based on the calculated horizontal length of all rows, and may be calculated based on representative values such as the lowest value, the maximum value, and the average value of the calculated horizontal length of the entire row. .

In addition, the point (

inside

), the original vertical length of the document may be estimated by adding the difference between the 3D coordinate value of the pixel of the 3D image 230 mapped to each and the 3D coordinate value of the pixel mapped to the previous point. That is, by Equation 2 below, the original vertical length of column i in the document (

) can be estimated. The vertical length of the document may be calculated based on the calculated vertical length of all columns, and may be calculated based on representative values such as the lowest value, the maximum value, and the average value of the calculated vertical length of all columns.

The processor 130 may restore the document based on the horizontal and vertical ratios estimated using the document ratio estimation module 137 through the document recovery module 139 .

Specifically, the processor 130 may adjust the ratio of the mapping table 250 obtained using the neural network model 135 based on the horizontal and vertical ratio estimated using the document ratio estimation module 137 . For example, as shown in FIG. 4 , the processor 130 sets the ratio of the mapping table 250 in which the aspect ratio is X:Y to X, which is the aspect ratio estimated using the document ratio estimation module 137 . It is possible to obtain the mapping table 410 in which the ratio is adjusted by adjusting it to ':Y'.

In addition, the processor 130 may restore the document 420 by mapping pixel values of the document included in the image to each of the plurality of points included in the adjusted mapping table.

As described above, by reconstructing the document by estimating the original shape of the document, the electronic device 100 may reconstruct the deformed document at a more accurate ratio close to the actual shape. Specifically, as shown on the left side of FIG. 5 , when the existing method is used to restore the business card 510 with a deformed shape, as shown in the center of FIG. 5 , the horizontal/vertical ratio is the original horizontal/vertical ratio. Since it is restored to correspond to the horizontal/vertical ratio of the bounding box, the business card 520 may be restored in a distorted shape that is not real. However, as shown on the left side of FIG. 5 , when the method according to an embodiment of the present disclosure is used to restore the business card 510 with a deformed shape, as shown on the right side of FIG. 5 , the horizontal/vertical ratio is It may be restored to the business card 530 having a shape close to the original horizontal/vertical ratio.

6 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

First, the electronic device 100 may acquire an image including a document with a deformed shape ( S610 ). In this case, the electronic device 100 may acquire an image through the camera 110 , but this is only an example and may receive an image from the outside.

The electronic device 100 may obtain a three-dimensional image including depth information of a document whose shape is deformed by inputting information about the image into a first network among the neural network models ( S620 ). In this case, the 3D image may include 3D coordinate values including depth information as well as X and Y coordinate values. Also, the first network may be learned by using, as input data and output data, each of the three-dimensional image obtained by photographing the training image and the training image of the document having a deformed shape with a depth camera.

The electronic device 100 may obtain a mapping table by inputting a 3D image to a second network among the neural network models ( S630 ). In this case, the mapping table is a table indicating the positional correlation between the pixels of the document whose shape included in the image is deformed and the pixels of the result image in which the shape of the document is restored. It may correspond to the proportion of the bounding box of the shape. Meanwhile, the second network may be learned by using, as input data and output data, a mapping table corresponding to a three-dimensional image obtained by photographing with a depth camera and a learning image, respectively.

The electronic device 100 may estimate the horizontal to vertical ratio of the document based on the 3D position information of pixels mapped to the mapping table ( S640 ). Specifically, the electronic device 100 adds the difference between the 3D coordinate value of the pixel of the 3D image mapped to each point included in the horizontal axis of the mapping table and the 3D coordinate value of the pixel mapped to the previous point to obtain a horizontal length , and the vertical length can be estimated by adding the difference between the 3D coordinate value of the pixel of the 3D image mapped to each point included in the vertical axis of the mapping table and the 3D coordinate value of the pixel mapped to the previous point. .

The electronic device 100 may adjust the ratio of the mapping table to the estimated horizontal and vertical ratio (S650). That is, the electronic device 100 may adjust the ratio of the mapping table to be close to the original horizontal and vertical ratio of the document.

The electronic device 100 may restore the document based on the adjusted mapping table (S660). Specifically, the electronic device 100 may restore the document by mapping pixels of the document included in the image to each of the plurality of points included in the adjusted mapping table.

7 is a block diagram illustrating in more detail a hardware configuration of an electronic device according to an embodiment of the present disclosure. 7 , the electronic device 700 according to the present disclosure includes a display 710 , a speaker 720 , a camera 730 , a memory 740 , a communication interface 750 , an input interface 760 , It may include a sensor 770 and a processor 780 . However, such a configuration is an example, and it goes without saying that a new configuration may be added or some configuration may be omitted in addition to such a configuration in carrying out the present disclosure. Meanwhile, the camera 730 , the memory 740 , and the processor 780 have the same configuration as the camera 110 , the memory 120 , and the processor 130 described with reference to FIG. 1 , and thus overlapping descriptions will be omitted.

The display 710 may display an image captured by the camera 730 . Also, the display 710 may display a bounding box surrounding a document whose shape is deformed in the captured image. Also, the display 710 may display a UI for receiving a user command for restoring the shape of the document.

On the other hand, the display 710 may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), etc., and the display 710 may be implemented as a flexible display, a transparent display, etc. in some cases. . However, the display 710 according to the present disclosure is not limited to a specific type.

The speaker 720 may output a voice message. In particular, the speaker 720 may be included in the electronic device 700 , but this is only an exemplary embodiment, and may be electrically connected to the electronic device 700 and located outside. In this case, the speaker 720 may output a voice message guiding that the shape of the document included in the captured image is restored.

The communication interface 750 includes a circuit and may communicate with an external device. Specifically, the processor 780 may receive various data or information from an external device connected through the communication interface 750 , and may transmit various data or information to the external device.

The communication interface 750 may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, and an NFC module. Specifically, each of the WiFi module and the Bluetooth module may perform communication using a WiFi method and a Bluetooth method. In the case of using a WiFi module or a Bluetooth module, various types of connection information such as an SSID may be first transmitted and received, and various types of information may be transmitted and received after communication connection using this.

In addition, the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), 5th Generation (5G), and the like. In addition, the NFC module may perform communication using a Near Field Communication (NFC) method using a 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz.

In particular, according to various embodiments of the present disclosure, the communication interface 750 may receive various types of information, such as data related to the neural network model 135 , from an external device. In addition, the communication interface 750 may receive an image including a document with a deformed shape from an external terminal or server.

The input interface 760 includes a circuit, and the processor 780 may receive a user command for controlling the operation of the electronic device 700 through the input interface 760 . Specifically, the input interface 760 may be implemented in a form included in the display 710 as a touch screen, but this is only an exemplary embodiment, and is composed of a button, a microphone, and a remote control signal receiver (not shown). can get

In particular, in various embodiments according to the present disclosure, the input interface 760 provides various functions such as a user command for executing a camera application, a user command for taking an image, a user command for restoring the shape of a document through the UI, and the like. User commands can be input.

The sensor 770 may acquire various information related to the electronic device 700 . In particular, the sensor 770 may include a GPS capable of acquiring location information of the electronic device 700 , and a biometric sensor (eg, a heart rate sensor) for acquiring biometric information of a user using the electronic device 700 . , PPG sensor, etc.) and various sensors such as a motion sensor for detecting a motion of the electronic device 700 .

Meanwhile, the functions related to the neural network model as described above may be performed through a memory and a processor. The processor may consist of one or a plurality of processors. In this case, one or a plurality of processors are general-purpose processors such as CPUs and APs, GPUs. It may be a graphics-only processor, such as a VPU, or an artificial intelligence-only processor, such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the non-volatile memory and the volatile memory. The predefined action rule or artificial intelligence model is characterized in that it is created through learning.

Here, being made through learning means that a predefined operation rule or artificial intelligence model of a desired characteristic is created by applying a learning algorithm to a plurality of learning data. Such learning may be performed in the device itself on which artificial intelligence according to the present disclosure is performed, or may be performed through a separate server/system.

A neural network model (or artificial intelligence model) may be composed of a plurality of neural network layers. Each layer has a plurality of weight values, and the layer operation is performed through the operation of the previous layer and the operation of the plurality of weights. Examples of neural networks include Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), GAN. There are Generative Adversarial Networks and Deep Q-Networks, and the neural network in the present disclosure is not limited to the above-described examples, except as otherwise specified.

The learning algorithm is a method of training a predetermined target device (eg, a robot) using a plurality of learning data so that the predetermined target device can make a decision or make a prediction by itself. Examples of the learning algorithm include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, and the learning algorithm in the present disclosure is specified when It is not limited to the above example except for.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to an embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store™) or on two user devices (eg, It can be distributed (eg downloaded or uploaded) directly or online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product (eg, a downloadable app) is stored at least on a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.

As described above, each of the components (eg, a module or a program) according to various embodiments of the present disclosure may be composed of a singular or a plurality of entities, and some of the above-described corresponding sub-components are omitted. Alternatively, other sub-components may be further included in various embodiments. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration.

According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

Meanwhile, the term “unit” or “module” used in the present disclosure includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, part, or circuit. can A “unit” or “module” may be an integrally formed component or a minimum unit or a part of performing one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium readable by a machine (eg, a computer). The device calls the stored instructions from the storage medium. and, as a device capable of operating according to the called command, the electronic device (eg, the electronic device 100) according to the disclosed embodiments may be included.

When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and it is common in the technical field to which the disclosure pertains without departing from the gist of the disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

Claims

A method for controlling an electronic device, comprising:

acquiring an image including a document whose shape is deformed;

inputting the obtained image into a trained neural network model to obtain a mapping table indicating a positional association between pixels of a document whose shape is deformed included in the image and pixels of a result image of which the shape of the document is restored;

estimating a horizontal/vertical ratio of the document based on location information of pixels mapped to the mapping table;

adjusting a ratio of the mapping table to the estimated horizontal and vertical ratio; and

and restoring a document based on the adjusted mapping table.
According to claim 1,

Obtaining the mapping table comprises:

Obtaining a three-dimensional image including depth information of the document whose shape is deformed by inputting the information on the image into a first network of the neural network model,

A control method for obtaining the mapping table by inputting the three-dimensional image to a second network among the neural network models.
3. The method of claim 2,

The estimating step is

The horizontal length is estimated by adding the difference between the 3D coordinate value of the pixel of the 3D image mapped to each point included in the horizontal axis of the mapping table and the 3D coordinate value of the pixel mapped to the previous point,

A control method for estimating the vertical length by adding a difference between a 3D coordinate value of a pixel of the 3D image mapped to each point included in the vertical axis of the mapping table and a 3D coordinate value of a pixel mapped to a previous point.
3. The method of claim 2,

The first network,

It is learned by using each of the three-dimensional image obtained by photographing the learning image and the learning image with a depth camera as input data and output data for a document with a deformed shape,

The second network,

A control method in which a 3D image obtained by photographing with the depth camera and a mapping table corresponding to the learning image are respectively used as input data and output data are learned.
According to claim 1,

The ratio of the mapping table corresponds to a ratio of the image or a ratio of a rectangular bounding box including a document with a deformed shape among the images.
According to claim 1,

The restoration step is

A control method for restoring the document by mapping pixels of the document included in the image to each of a plurality of points included in the adjusted mapping table.
In an electronic device,

camera;

a memory including at least one instruction; and

processor; including;

The processor, by executing the at least one instruction,

Obtaining an image including a document whose shape is deformed through the camera,

By inputting the obtained image into the learned neural network model, a mapping table indicating the positional association between the pixel of the document whose shape included in the image is deformed and the pixel of the result image in which the shape of the document is restored is obtained,

adjusting the ratio of the mapping table to the estimated horizontal and vertical ratio by estimating a horizontal and vertical ratio of the document based on location information of pixels mapped to the mapping table;

An electronic device for restoring a document based on the adjusted mapping table.
8. The method of claim 7,

The processor is

Obtaining a three-dimensional image including depth information of the document whose shape is deformed by inputting the information on the image into a first network of the neural network model,

The electronic device obtains the mapping table by inputting the 3D image to a second network among the neural network models.
9. The method of claim 8,

The processor is

The horizontal length is estimated by adding the difference between the 3D coordinate value of the pixel of the 3D image mapped to each point included in the horizontal axis of the mapping table and the 3D coordinate value of the pixel mapped to the previous point,

An electronic device for estimating the vertical length by adding a difference between a 3D coordinate value of a pixel of the 3D image mapped to each point included in the vertical axis of the mapping table and a 3D coordinate value of a pixel mapped to a previous point.
9. The method of claim 8,

The first network,

It is learned by using each of the three-dimensional image obtained by photographing the learning image for a document with a deformed shape and the learning image with a depth camera as input data and output data,

The second network,

An electronic device that is learned by using each of the mapping table corresponding to the three-dimensional image obtained by photographing with the depth camera and the learning image as input data and output data.
8. The method of claim 7,

The ratio of the mapping table corresponds to a ratio of the image or a ratio of a rectangular bounding box including a document with a deformed shape among the images.
8. The method of claim 7,

The processor is

The electronic device restores the document by mapping pixels of the document included in the image to each of the plurality of points included in the adjusted mapping table.