KR20200088682A - Electronic apparatus and controlling method thereof - Google Patents

Abstract

Provided are an electronic device capable of more efficiently inspecting a defect of a product, and a control method thereof. The electronic device of the present invention comprises: a camera; a memory including at least one instruction; and a processor connected to the camera and the memory, and controlling the electronic device. By executing the at least one instruction, the processor obtains a first image photographing an object through the camera, obtains a second image including an object not having a defect by using the obtained first image as input data of a learned first model, and determines whether or not the object has a defect based on the first image and the second image.

Description

Electronic device and its control method {ELECTRONIC APPARATUS AND CONTROLLING METHOD THEREOF}

The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that can more efficiently and conveniently determine whether a defect exists in an object using an image of the object .

In addition, the present disclosure relates to an artificial intelligence (AI) system that simulates functions such as cognition and judgment of a human brain using a machine learning algorithm and its application.

Recently, smart factory-related technologies that automate the manufacturing and processing of products by applying ICT (Information and Communications Technologies) technology to equipment and machinery in factories are increasing.

In particular, among technologies related to smart factories, technologies related to automation of product inspection based on machine vision are emerging. Specifically, a technique for automatically determining whether a defect is present on a shape or a surface of a product using a machine vision, which is a computer-equipped imaging device, has emerged as a technology related to a smart factory.

In this regard, the shape of the product or the pattern of defects occurring on the surface is becoming very diverse, and as the types of parts included in the product are diversified, the location or background of defects in the product is also very diverse and complicated. In addition, the machine vision-based inspection system should distinguish the defects of the product from the actual product defects such as scratches, scratches, cracks, etc., in that the defects of the product are judged based on the image acquired using the camera. There is also a problem.

Meanwhile, artificial intelligence systems that realize human-level intelligence have been used in various fields. The artificial intelligence system is a system in which a machine learns, judges, and becomes smart unlike a rule-based smart system. As the artificial intelligence system is used, the recognition rate is improved and the user's taste can be more accurately understood, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

Artificial intelligence technology consists of machine learning (e.g. deep learning) and elemental technologies utilizing machine learning.

Machine learning is an algorithm technology that classifies/learns the characteristics of input data by itself, and element technology is a technology that simulates functions such as cognition and judgment of the human brain by using machine learning algorithms such as deep learning. It consists of technical fields such as understanding, reasoning/prediction, knowledge expression, and motion control.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes and applies/processes human language/characters, and includes natural language processing, machine translation, conversation system, question and answer, speech recognition/synthesis, and the like. Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image improvement. Inference prediction is a technique for logically inferring and predicting information by determining information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge building (data generation/classification), knowledge management (data utilization), and so on. Motion control is a technique for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), operation control (behavior control), and the like.

As the fields in which artificial intelligence technology is applied are diversified and the machine vision-based inspection system is complicated, attempts have been made to combine artificial intelligence technology with a machine vision-based inspection system.

However, in the case of a machine vision based inspection system incorporating artificial intelligence technology, a lot of data is required to learn machine vision. In particular, in order to detect a defect in a product, learning related to a normal product is required, but there is a problem that it is difficult to sufficiently secure data about the product before or shortly after the product is released.

The present disclosure has been devised to solve the above-mentioned problems, and the purpose of the present disclosure is to provide an electronic device capable of more efficiently checking whether a product is defective based on a learning model learned using a design image of a product (object). It provides an apparatus and a control method thereof.

An electronic device according to an embodiment of the present disclosure includes a camera; A memory including at least one instruction; And a processor connected to the camera and the memory to control the electronic device. The processor, by executing the at least one command, obtains a first image of an object captured through the camera, and uses the acquired object first image as input data of the learned first model, without defects. A second image including the object in a state is acquired, and it is determined whether the object is defective based on the first image and the second image.

In addition, the processor compares the first image and the second image, and in the first image, the object is an image of the object without the defect and an image of the object and the object included in the second image. Different regions can be judged between images.

In addition, the processor acquires an image corresponding to the determined different region from the first image, and uses the image corresponding to the obtained different region as input data of the learned second model, thereby characteristic of the different region. And determining whether the object is defective based on the determined characteristics of different regions.

The processor may determine that a defect exists in the object when the obtained image corresponding to the different region is determined as an image of an object having a defect according to the characteristics of the determined different region.

Also, the position of the object in the first image and the position of the object without defects in the second image may be the same.

In addition, the electronic device may further include a display, and the processor may control the display to display information on whether the determined object is defective.

In addition, when it is determined that a defect exists in the object, the processor may control the display to display a region in which the defect exists in the object in a distinctive manner from other regions in the first image.

Further, the first model is a model trained based on an image of a defect-free state object, and an image of an object free of defects is generated by a third model trained based on the design image of the object. Can.

Meanwhile, a method of controlling an electronic device according to another embodiment of the present disclosure includes: acquiring a first image photographing an object through the camera; Obtaining a second image including the object in a defect-free state by using the acquired first image as input data of the learned first model; And determining whether the object is defective based on the first image and the second image.

And, the determining whether the defect is, compares the first image and the second image to compare different areas between the image of the object included in the first image and the image of the object included in the second image. The determining step; may further include.

In addition, determining whether the defect includes: obtaining an image corresponding to the determined different region from the first image; Determining characteristics of the different regions by using the image corresponding to the obtained different regions as input data of the learned second model; And determining whether the object is defective based on the determined characteristics of the different regions.

The determining of the defect may include determining that a defect exists in the object when the image corresponding to the different region is determined as an image of an object having a defect according to the characteristics of the determined different region. Step; may further include.

Also, the position of the object in the first image and the position of the object without defects in the second image may be the same.

In addition, the method may further include displaying information on whether the determined object is defective on a display.

And, if it is determined that a defect exists in the object, displaying a region in which the defect is present in the object to be distinguished from other regions in the first image.

Here, the first model is a model trained based on an image of a defect-free state object, and the image of an object free of defects is obtained by a third model trained based on the design image of the object. Can be generated.

As described above, by determining whether an object has a defect using the learned artificial intelligence model, the user's convenience can be increased in that the user's need to inspect the surface of each object is reduced. In addition, in that the artificial intelligence model trained as the design image of the object is used, the user can reduce the burden of securing a sufficient image of the product in a normal state.

1 is a view for explaining a machine vision-based system including an electronic device according to an embodiment of the present disclosure,
2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;
FIG. 3A is a diagram for describing an artificial intelligence first model for learning an image of an object having no defect according to an embodiment of the present disclosure.
3B is a view for explaining an AI first model that provides an image of an object having no defect according to an embodiment of the present disclosure;
4 is a diagram for explaining an artificial intelligence third model that generates training data of an artificial intelligence first model according to an embodiment of the present disclosure;
5 is a block diagram illustrating in detail a configuration of an electronic device according to various embodiments of the present disclosure;
6 is a diagram for describing an electronic device according to an embodiment of the present disclosure;
7 is a block diagram illustrating an electronic device for learning and using an artificial intelligence model;
8 and 9 are block diagrams illustrating a learning unit and an analysis unit according to an embodiment of the present disclosure, and
10 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure, while considering the functions in the present disclosure, general terms that are currently widely used are selected, but this may vary according to the intention or precedent of a person skilled in the art or the appearance of new technologies. . Also, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, their meanings will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the contents of the present disclosure, not simply the names of the terms.

Since the embodiments of the present disclosure can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, it should be understood to include all conversions, equivalents, or substitutes included in the scope of the disclosed ideas and techniques. In the description of the embodiments, when it is determined that the detailed description of the related known technology may obscure the subject matter, the detailed description is omitted.

Terms such as first and second may be used to describe various components, but components should not be limited by terms. The terms are only used to distinguish one component from other components.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprises” or “consist of” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, one or more other It should be understood that features or numbers, steps, operations, components, parts, or combinations thereof are not excluded in advance.

In the present disclosure, "module" or "unit" performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "parts" are integrated into at least one module except for "modules" or "parts" that need to be implemented with specific hardware to be implemented with at least one processor (not shown). Can be.

In the present disclosure, “at least one of a, b, or c” denotes only a, only b, only c, both a and b, both a and c, both b and c, all a, b and c, or variations thereof Can be interpreted as

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily perform the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Hereinafter, an electronic device for determining whether an object is defective according to various embodiments of the present disclosure will be described.

1 is a diagram for describing a machine vision based system including an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1, the machine vision based system 1000 may include an electronic device 100 and a display device 200.

The electronic device 100 may inspect the object 10 passing through the electronic device 100 through the conveyor belt 300 in the smart factory. Specifically, the electronic device 100 may photograph the object 10 on the conveyor belt 300 to determine whether a defect exists in the object 10.

Defects of the object 10 may indicate irregularities such as irregular scratches, nicks, and cracks in the appearance of the product in the process of manufacturing and manufacturing the object 10.

Here, the object 10 may be any product that is produced, manufactured, and processed in a smart factory. For example, the object 10 includes a smartphone, a tablet PC (tablet personal computer), a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), Laptop personal computer (PC), netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or It may include at least one of a wearable device (wearable device). According to various embodiments, the wearable device may be an accessory type (for example, a watch, ring, bracelet, anklet, necklace, glasses, contact lens, or head-mounted device (HMD)), a fabric or clothing integrated type ( Examples may include at least one of an electronic garment), a body attachment type (eg, a skin pad or tattoo), or a bio-implantable type (eg, an implantable circuit).

In some embodiments, the object 10 may be a home appliance. Household appliances include, for example, televisions, digital video disk (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, and home automation controls. Home automation control panel, security control panel, TV box (e.g. Samsung HomeSync™, Samsung One Connect Box™), game console, electronic dictionary, electronic key, camcorder, or electronic picture frame It may include at least one of.

In addition, in some embodiments, the object 10 may be a component included in a finished product such as a display panel, a battery, and a button, rather than a finished product such as a smartphone, a tablet PC, and a home appliance. .

The electronic device 100 may photograph the object 10 placed on the conveyor belt 300 at various angles while moving up, down, left, and right and front of the conveyor belt 300. Then, as the object 10 moves on the conveyor belt 300, the electronic device 100 may acquire an image of the object 10 while tracking the object 10 on the conveyor belt 300.

To this end, the electronic device 100 may be designed to freely move the space on the conveyor belt 300.

The electronic device 100 may acquire various images of the object 10 while controlling various shooting conditions such as illumination, luminance, and focal length of the camera included in the electronic device 100.

The electronic device 100 may determine whether a defect exists in the photographed object 10 using the acquired image of the object 10. Specifically, the electronic device 100 may determine whether a defect exists in the object 10 using an artificial intelligence model learned based on an image associated with the object 10. Here, the image related to the object 10 may represent an image of the object 10 in a state in which defects such as irregular scratches, cuts, and cracks do not exist.

More specifically, the electronic device 100 photographs an image of the object 10 on the conveyor belt 300, inputs it to the trained model, obtains a normal image of the object 10, and photographs the image and the learning model It is possible to determine whether the object 10 is defective by comparing the images obtained from.

Specifically, the electronic device 100 determines whether a different region exists between the photographed image and the image obtained from the learning model, whether the cause of the different region is due to a defect generated in the object 10, or light bleeding, foreign matter It can be determined by external factors such as adhesion.

Meanwhile, although only one electronic device 100 is illustrated in FIG. 1, the present invention is not limited thereto. That is, at least one electronic device 100 photographing one object on the conveyor belt 300 may exist in the space above the conveyor belt 300.

The electronic device 100 may transmit the image of the object 10 and the determined result information to the display device 200.

The display apparatus 200 may display information such as a photographed image of the object 10, an image obtained from a learning model, and whether a defect has occurred and a defect location, and accordingly, the administrator of the machine vision system 1000 is a conveyor belt It is possible to know the image of the object 10 on the 300, the presence of a defect, and the location of the defect.

Although the display device 200 is implemented as a separate device from the electronic device 100 in FIG. 1, it goes without saying that the electronic device 100 may include the display device 200.

In addition, in FIG. 1, it has been described that the electronic device 100 directly transmits information related to an image and a defect of the object 10 to the display device 200, but the server between the electronic device 100 and the display device 200 is described. (Not shown), the electronic device 100 transmits information related to the image and the defect of the object 10 to the server (not shown), and the display device 200 displays the object 10 from the server (not shown). ) And information related to defects.

Through the electronic device 100 according to various embodiments of the present disclosure, the machine vision-based system may determine whether a defect exists in the object 10. Hereinafter, the electronic device 100 according to an embodiment of the present disclosure will be described with reference to FIG. 2.

2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device 100 includes a camera 110, a memory 120, and a processor 130.

The camera 110 may photograph the object 10 from various angles. The camera 110 may photograph the top, bottom, left, and right and front of the object 10 in the x, y, and z axes of the object 10. At this time, the camera 110 may photograph various images 10 of the object 10 at one location while varying the brightness conditions such as the illuminance and luminance of the lens or adjusting the focal length of the lens.

The camera 110 may transmit an image of the acquired object 10 to the processor 130, and the processor 130 may determine whether a defect exists in the object 10 based on the image received from the camera 110. Can.

The memory 120 may include, for example, internal memory or external memory. The internal memory includes, for example, volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (eg, OTPROM (one time programmable ROM (PROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g. NAND flash or NOR flash, etc.), hard drives, Or it may include at least one of a solid state drive (SSD).

The external memory may be a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), It may include a multi-media card (MMC) or a memory stick (memory stick). The external memory may be functionally and/or physically connected to the electronic device 100 through various interfaces.

The memory 120 is accessed by the processor 130, and data read/write/modify/delete/update may be performed by the processor 130. In the present disclosure, the term memory may include a memory card (eg, a micro SD card, a memory stick) mounted in the memory 120, a ROM, RAM in the processor 130, or an electronic device 100.

The memory 120 may include at least one computer-executable instruction for controlling the electronic device 100.

Also, the memory 120 may store artificial intelligence models according to various embodiments of the present disclosure.

The artificial intelligence model described in the present disclosure is a judgment model learned based on an artificial intelligence algorithm, and may be, for example, a model based on a neural network. The learned artificial intelligence model may be designed to simulate a human brain structure on a computer and may include a plurality of network nodes having weights, which simulate neurons of a human neural network. The plurality of network nodes may form a connection relationship so that neurons simulate synaptic activity of neurons that exchange signals through synapses. In addition, the trained artificial intelligence model may include, for example, a neural network model or a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes may be located at different depths (or layers) and exchange data according to a convolution connection relationship. Examples of the learned artificial intelligence model may include, but are not limited to, Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN).

The first model among the AI models stored in the memory 120 may be a model trained to provide an image of the object 10 without defects based on the image of the object 10 photographed by the camera 110. have.

In addition, the second model among the artificial intelligence models stored in the memory 120 is an artificial object that determines whether a defect such as scratching or stamping occurs on the object 10 based on the image of the object 10 photographed by the camera 110. It can be an intelligent model.

In addition, the third model among the AI models stored in the memory 120 may be a model trained to generate training data of the first model.

Meanwhile, FIG. 3 is a diagram for explaining a first model, and specifically, FIG. 3A is a diagram for explaining a first model for learning an image of an object 10 having no defects, and FIG. 3B is a defect. A diagram for explaining a first model that provides an image of an object 10 that is not.

Referring to FIG. 3A, the first model is an artificial intelligence model trained by an artificial intelligence algorithm, and may be a model trained to provide an image 220 of the object 10 without defects.

To this end, the first model may be trained by receiving a plurality of images 210 of the object 10 without defects. In this case, the first model may be a model trained by an image reconstruction algorithm such as Autoencoder, VAE (Variational Auto-encoder) GAN (Generative Adversarial Network), or Associate Memory.

That is, the first model generates an image of the object 10 without defects by the artificial intelligence-based image restoration algorithm, even if the image 310 of the object 10 with some defects is input as shown in FIG. 3B. , It is possible to create an image 320 of the object 10 without defects.

For example, as illustrated in FIG. 3A, the first model may be trained to generate a frame image 220 of a mobile phone having no defect based on the frame image 210 of the mobile phone having no defect. At this time, the first model may be learned based on a plurality of defect-free mobile phone frame images.

In addition, the first model learned based on the plurality of images, even if some of the scratched cell phone frame image 310 is input to the lower left of the cell phone frame as shown in FIG. 320).

On the other hand, the image 210 of the defect-free object 10 used as the training data of the first model may be an image obtained by photographing the object 10 without defects, but the third model, which is an artificial intelligence model, may be used. It may be an image of the obtained virtual object.

In this regard, FIG. 4 is a diagram for describing a third model that generates training data 210 of the first model.

The third model may be trained to provide images 421 and 422 of the object 10 based on the design image 410 of the object 10 using artificial intelligence algorithms. Here, the design image of the object 10 is an image generated before the object 10 is produced and manufactured, and may be a computer aided design (CAD) image, a computer aided engineering (CAE) image, or a 3D printed image. On the other hand, the design image of the object 10 is not necessarily limited thereto, and may be a design image of the present disclosure if it is a virtual image other than an image actually photographing the object 10.

The third model may provide images 421 and 422 of the object 10 without defects based on the design image 410 of the object 10.

4, it is assumed that the design image 410 of the object 10 is given. In this case, the design image 410 may be a design image of a part of the object 10.

At this time, the image of the generated object 10 may be the same image as the image of the object 10 manufactured based on the design image 410. In addition, since the image of the generated object 10 is generated based on the design image 410, it may be an image of the object 10 without defects.

For example, the third model may generate a plurality of images 421 and 422 for the holes in the mobile phone frame based on the CAD image 410 for the holes in the mobile phone frame. The generated images 421 and 422 are images identical to images of holes in a mobile phone frame manufactured according to the CAD image 410, and may be images of holes without defects.

Meanwhile, the third model may generate images 421 and 422 having various change factors using an image-to-image generation technique, an image-to-image synthesis technique, or an image-to-image synthesis technique. For example, the third model uses image-to-image generation technology, image-to-image synthesis technology, or image-to-image synthesis technology to image (421, 422) images of objects 10 having different colors or materials of frames. ).

Accordingly, the first model may be trained to generate various images of the object 10 without defects based on the images 421 and 422 of the object 10 generated from the third model.

Meanwhile, the artificial intelligence models described above may be learned from an external server (not shown) and provided to the electronic device 100. The electronic device 100 may download the AI model from an external server (not shown) and store it in the memory 120, and when the AI model is updated (or re-learned), the updated AI model is received from the external server. You can also save. To this end, the electronic device 100 may be connected to an external server through a local area network (LAN) and an Internet network.

According to an embodiment, all or part of the memory 120 may be implemented as an external server of the electronic device 100 such as a cloud server. That is, all or part of the above-described AI models may be included in an external server.

Returning to FIG. 2 again, the processor 130 is a component for controlling the overall operation of the electronic device 100. For example, the processor 130 may drive an operating system or an application program to control a plurality of hardware or software components connected to the processor 130, and may perform various data processing and calculations. The processor 130 may be a central processing unit (CPU) or a graphics-processing unit (GPU), or both. The processor 130 may be implemented with at least one general-purpose processor, digital signal processor, application specific integrated circuit (ASIC), system on chip (SoC), microcomputer (MICOM), or the like.

The processor 130 may perform operations of the electronic device 100 according to various embodiments of the present disclosure by executing computer executable instructions stored in the memory 120.

The processor 130 may be connected to the camera 110 and the memory 120 to control the electronic device 100.

The processor 130 acquires a first image obtained by photographing the object through the camera 110, and uses the obtained first image as input data of the learned first model to include a second object including a defect-free object You can acquire an image.

At this time, as described above in FIG. 3, in that the first model is a model trained to generate an image of the object 10 in the absence of a defect, the object 10 is photographed through the camera 110. Even if a defect is present, the second image may be an image of an object without defects.

Here, since the second image is an image generated based on the first image, the location of the object in the first image and the location of the object without defects in the second image may be the same.

The processor 130 may determine whether the object is defective based on the first image and the second image.

Specifically, the processor 130 may compare the first image and the second image to determine different areas between the image of the object included in the first image and the image of the object included in the second image. In the point that the position of the object in the first image and the position of the object without defects in the second image are the same, the processor 130 overlaps the first image and the second image to overlap the first image and the second image to each other. It is possible to determine whether different areas exist between each other.

Since the second image is an image of an object in a defect-free state, when a different region does not exist between the first image photographing the object 10 and the second image obtained from the first model, the first image The image of the included object 10 may be an image of the object 10 without defects.

On the other hand, when different areas exist between the first image and the second image, different areas may be areas where defects exist, but may be areas where defects do not exist. For example, the different areas between the first image and the second image are light smears of the camera 110, only the area containing foreign matters on the object 10 or the camera 110, and scratches and cracks included in the object 10 It may not be an area where defects such as these exist.

Accordingly, the processor 130 may use the learned second model to determine characteristics of different regions between the first image and the second image, that is, whether a defect exists in the different regions. Here, the second model may be an artificial intelligence model trained to determine whether a defect such as a scratch or a stamp has occurred in the object 10 included in the input image.

Specifically, the processor 130 may acquire an image corresponding to a different region from the first image, and use the acquired image as input data of the learned second model to determine characteristics of different regions. That is, the processor 130 may use the second model to determine whether an image corresponding to a different area is an image including light blurring or foreign matter, or an image having defects such as scratches and cracks.

Also, the processor 130 may determine whether the object 10 is defective based on the determined characteristics of different regions.

For example, if it is determined that the determined image of the different region is due to external factors such as light bleeding or foreign matter, the processor 130 may determine that the image corresponding to the different region is an image of an object having no defect. However, when the processor 130 determines that the determined image of the different area is due to scratches, cracks, or the like of the appearance of the object 10, it may be determined that the image corresponding to the different area is an image of an object having a defect.

In addition, when it is determined that the image corresponding to the different region is the image of the object having the defect according to the determined characteristics of the different regions, the processor 130 may determine that the defect exists in the object.

That is, when the processor 130 determines that an image corresponding to a different region is an image of an object having a defect based on the second model, the processor 130 determines that the defect exists and the kind of the defect present in the object (for example, , Scratch, crack, etc.).

Meanwhile, the components of FIG. 2 are not essential components of the electronic device 100. The electronic device 100 may be implemented by fewer components than those of FIG. 2, or may be implemented by more components than those of FIG. 2.

5 is a block diagram illustrating in detail a configuration of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 5, the electronic device 100 may include a camera 110, a memory 120, a processor 130, a display 140, a communication unit 150, and an input unit 160.

For convenience of description in relation to the camera 110, the memory 120, and the processor 130, description within the overlapping range with FIG. 4 will be omitted.

The display 140 may display various information under the control of the processor 130. The display 140 may display an image of the object 10 and information on whether the object 10 is defective under the control of the processor 130.

Here, the image of the object 10 may include at least one of an image of the object 10 photographed by the camera 110 and an image of the object 10 acquired by the first model.

In addition, the information on whether or not the object 10 is defective is information on an area in which the object 10 has a defect, and includes information on the area where the object 10 has a defect, information on the type of the defect, and the like. It can contain.

The communication unit 150 is a component for the electronic device 100 to communicate with an external server (not shown) and the display device 200. Through the communication unit 150, the electronic device 100 may receive the first to third models from an external server (not shown) in which the artificial intelligence model is stored. Also, the electronic device 100 may transmit an image of the object 10 and information on whether the object 10 is defective to the display device 200 through the communication unit 150.

To this end, the communication unit 140 may include various communication modules such as a wired communication module (not shown), a short-range wireless communication module (not shown), and a wireless communication module (not shown).

Here, the wired communication module is a module for performing communication with an external device (not shown) according to a wired communication method such as wired Ethernet. In addition, the short-range wireless communication module is a module for performing communication with an external device (not shown) located at a short distance according to a short-range wireless communication method such as Bluetooth (Bluetooth, BT), Bluetooth Low Energy (BLE), and ZigBee. In addition, the wireless communication module is a module that is connected to an external network according to wireless communication protocols such as WiFi and IEEE to communicate with an external device (not shown) and a voice recognition server (not shown). In addition, the wireless communication module is based on various mobile communication standards such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), and 5G Networks. It may further include a mobile communication module that performs communication by connecting to a mobile communication network.

The input unit 160 is a component for receiving input from a user, and an input interface may correspond thereto. Through the input unit 160, the electronic device 100 may receive a user's input on the shooting conditions of the camera 110. For example, through the input unit 160, the electronic device 100 may receive photographing conditions, such as brightness of the camera 100, a focal length, and the number of photographing in one minute.

Meanwhile, the processor 120 may control the display 140 to display an image of the object 10 or information on whether the determined object 10 is defective.

In this regard, FIG. 6 is a diagram for describing an electronic device according to an embodiment of the present disclosure. Specifically, FIG. 6 is a diagram for describing an electronic device displaying information on a defect of the object 10 when it is determined that a defect exists in the object 10.

When it is determined that a defect exists in the object 10, the processor 120 determines that an area where the defect exists in the object 10 on the image of the object 10 photographed by the camera 110 is the object 10. It can be marked to distinguish it from other areas.

For example, as illustrated in FIG. 6, the processor 120 controls the display 140 to display an identification mark such as a square or a circle in a region where a defect is present in the object 10 on the captured image of the object 10. can do.

In addition, the processor 120 displays the display 140 to additionally display information indicating the type of the defect, in addition to displaying an identification mark on the area where the defect exists in the object 10 on the image of the object 10. Can be controlled. For example, when it is determined that the object 10 is scratched, the processor 120 may be'scratched' or'scratched', in addition to displaying an area where a combination exists on the image of the photographed object 10 : 80%'.

Meanwhile, the configuration of FIG. 5 is only an embodiment, and some of the components of the electronic device 100 illustrated in FIG. 5 may be omitted according to an implementation example of the electronic device 100. For example, the electronic device 100 does not include the display unit 140 and is connected to the display device 200 and may transmit an image of the object photographed to the display device 200 and information about whether the object is defective, The display device 200 having received this may display an image of the object 10 as shown in FIG. 6 and an identification mark for an area where a defect exists.

Meanwhile, the electronic device 100 may include a learning unit and an analysis unit for learning and using the above-described recognition model.

7 is a block diagram illustrating an electronic device for learning and using an artificial intelligence model, according to an embodiment of the present disclosure.

Referring to FIG. 7, the processor 130 may include at least one of a learning unit 131 and a determination unit 132.

The learning unit 131 may generate an artificial intelligence model having a criterion for acquiring an image of an object having no defect based on the image of the object photographed using the learning data. For example, the learning unit 131 generates, learns, or re-learns an artificial intelligence first model to obtain an image of an object without defects from an image of an object that is taken by using an image of an object without defects as learning data. I can do it.

In addition, the learning unit 131 may generate an artificial intelligence second model having criteria for determining the presence and type of defects included in the image of the photographed object using the learning data. For example, the learning unit 131 may generate, learn, or retrain a second model to determine the existence and type of a defect included in the object by using various images of the object including the defect as learning data.

In addition, the learning unit 131 uses a virtual object image such as an object design drawing as training data to obtain training data of the first model, that is, an image of an object without defects, and has no defects. The artificial intelligence third model may be generated, trained, or re-trained to acquire an image of the object of.

The determination unit 132 may generate an image of the object 10 without defects by using a predetermined image as input data of the learned artificial intelligence first model. As another embodiment, the determination unit 132 uses the predetermined image as input data of the learned AI second model, and determines whether a defect exists in the object 10 included in the predetermined image and the type of the defect. I can judge. Here, the predetermined image may be an image of the object 10 photographed by the camera 110.

In addition, the determination unit 132 may generate an image of the object 10 without defects by using a virtual image such as a design drawing of the object 10 as input data of the learned artificial intelligence third model.

At least a portion of the learning unit 131 and at least a portion of the determining unit 132 may be implemented as a software module or manufactured in at least one hardware chip and mounted on the electronic device 100. For example, at least one of the learning unit 131 and the determining unit 132 may be manufactured in the form of a dedicated hardware chip for artificial intelligence, or an existing general-purpose processor (for example, a CPU or application processor) or a graphics-only processor. It may be manufactured as part of (eg, GPU) and mounted on various electronic devices. At this time, the dedicated hardware chip for artificial intelligence is a dedicated processor specialized in probability computation, and has higher parallel processing performance than the conventional general-purpose processor, and thus can rapidly process computational tasks in the field of artificial intelligence such as machine learning. When the learning unit 131 and the determination unit 132 are implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory readable recording medium (non- transitory computer readable media. In this case, the software module may be provided by an operating system (OS) or may be provided by a predetermined application. Alternatively, some of the software modules may be provided by an operating system (OS), and the other may be provided by a predetermined application.

Meanwhile, the learning unit 131 and the determining unit 132 may be mounted on one electronic device or may be mounted on separate electronic devices. In addition, the learning unit 131 and the determination unit 132 may provide the model information constructed by the learning unit 131 to the determination unit 132 through wired or wireless communication, or input to the learning unit 131. Data may be provided to the learning unit 132 as additional learning data.

8 and 9 are block diagrams of the learning unit 131 and the determining unit 132 according to various embodiments.

Referring to FIG. 8, the learning unit 131 may include a learning data acquisition unit 131-1 and a model learning unit 131-4. In addition, the learning unit 131 may further include at least one of a learning data pre-processing unit 131-2, a training data selection unit 131-3, and a model evaluation unit 131-5.

The learning data acquiring unit 131-1 may acquire learning data required for the artificial intelligence first model for acquiring an image of the object 10 without a defect. In this case, the learning data of the first model includes an image obtained by photographing an object without defects, a virtual image of an object without defects obtained from the learned third model, and a defect area by the electronic device 100. It can be an image of an object that is determined to not exist.

In addition, the training data acquisition unit 131-1 may acquire various images and various kinds of information on the types of defects in order to train the artificial intelligence second model. In this case, the learning data acquiring unit 131-1 may acquire information in which the image of the object in which the combination exists and information on the type of defect match each other.

In another embodiment, the learning data acquisition unit 131-1 is a design drawing of the object 10 (for example, to obtain a virtual image, not a captured image of the object 10 without defects) CAD drawings, CAE drawings, etc.).

The model learning unit 131-4 may train the first model to have a criterion for generating an image of an object having no defects, using the training data.

Alternatively, the model learning unit 131-4 may train the second model to have a specific criterion for determining whether or not a defect exists in the object 10 and the type of the object using the training data.

Then, the model learning unit 131-4 may train the third model to have a criterion for generating a virtual object image having no defects using the training data.

The model learning unit 131-4 may train the artificial intelligence model through supervised learning. Alternatively, the model learning unit 131-4 may, for example, train an artificial intelligence model through unsupervised learning by self-learning using learning data without much guidance. For example, the model learning unit 131-4 may train an artificial intelligence model using a Generative Adversarial Network (GAN) technology or a VAE (Variational Auto-encoder) technology. In addition, the model learning unit 131-4 may train the artificial intelligence model, for example, through reinforcement learning using feedback on whether a judgment result according to learning is correct. In addition, the model learning unit 131-4 may train the artificial intelligence model using, for example, a learning algorithm including an error back-propagation or a gradient descent method. .

In addition, the model learning unit 131-4 may also learn the selection criteria for what training data to use.

The model learning unit 131-4 may determine, as a plurality of pre-built artificial intelligence models, an artificial intelligence model for learning an artificial intelligence model having a high relationship between input learning data and basic learning data. In this case, the basic learning data may be pre-classified for each type of data, and the artificial intelligence model may be pre-built for each type of data. For example, the basic training data is pre-classified by various criteria such as the region where the training data is generated, the time when the training data is generated, the size of the training data, the genre of the training data, the creator of the training data, the type of the object in the training data, and the like. It may be.

When the artificial intelligence model is learned, the model learning unit 131-4 may store the learned artificial intelligence model. For example, the model learning unit 131-4 may store the learned artificial intelligence model in the memory 120 of the electronic device 100.

The learning unit 131 is a learning data pre-processing unit 131-2 and a learning data selection unit 131-3 in order to improve the determination result of the AI model or to save resources or time required to generate the AI model. ) May be further included.

The learning data pre-processing unit 131-2 may pre-process the acquired data so that the acquired data can be used for learning to acquire an image of an object having no defect. In addition, the learning data preprocessing unit 131-2 may preprocess the acquired data so that the acquired data can be used for learning to determine whether an object has a defect or not and the type of the defect.

The learning data selection unit 131-3 may select data required for learning from data acquired by the learning data acquisition unit 131-1 or data preprocessed by the learning data preprocessing unit 131-2. The selected learning data may be provided to the model learning unit 131-4.

The learning data selection unit 131-3 may select learning data necessary for learning from acquired or preprocessed data according to a preset selection criterion. In addition, the learning data selector 131-3 may select learning data according to a preset selection criterion by learning by the model learning unit 131-4.

The learning unit 131 may further include a model evaluation unit 131-5 to improve the determination results of the first to third models.

The model evaluation unit 131-5 may input the evaluation data to the artificial intelligence model, and if the determination result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit 131-4 may cause the model learning unit 131-4 to learn again. have. In this case, the evaluation data may be predefined data for evaluating the artificial intelligence model.

For example, the model evaluator 131-5, among the judgment results of the learned artificial intelligence model for the evaluation data, sets a predetermined criterion when the number or rate of evaluation data whose judgment result is not accurate exceeds a preset threshold. It can be evaluated as unsatisfactory.

On the other hand, when there are a plurality of learned AI models, the model evaluator 131-5 evaluates whether each of the learned AI models satisfies a predetermined criterion, and finalizes the model that satisfies a predetermined criterion. You can decide as a model. In this case, when there are a plurality of models satisfying a predetermined criterion, the model evaluator 131-5 may determine any one or a predetermined number of models preset in order of highest evaluation score as the final artificial intelligence model.

Referring to FIG. 9, the determination unit 132 according to some embodiments may include an input data acquisition unit 132-1 and a determination result providing unit 132-4.

Also, the determination unit 132 may further include at least one of the input data pre-processing unit 132-2, the input data selection unit 132-3, and the model update unit 132-5.

The input data acquisition unit 132-1 may acquire data required to acquire an image of an object without defects. At this time, the obtained data may be an image of the object 10 photographed by the camera 110 or a design image of the object 10. The determination result providing unit 132-4 applies the input data obtained from the input data acquisition unit 132-1 to the artificial intelligence first model or the third model learned as an input value to obtain an image of an object without defects. can do.

The input data acquisition unit 132-1 may acquire data necessary to determine whether a defect is present in the object included in the image and the type of the defect. The determination result providing unit 132-4 may apply the input data obtained from the input data acquisition unit 132-1 to the second model learned as an input value to determine whether an object is defective or not and the type of defect.

The determination result providing unit 132-4 applies the data selected by the input data preprocessing unit 132-2 or the input data selection unit 132-3, which will be described later, to the artificial intelligence model as input values to obtain a determination result. Can.

In one embodiment, the determination result providing unit 132-4 may apply the data acquired by the input data acquisition unit 132-1 to the trained artificial intelligence model to obtain an image of a defect-free object.

The determination unit 132 improves the determination result of the artificial intelligence model, or saves resources or time for providing the determination result, the input data pre-processing unit 132-2 and the input data selection unit 132-3 It may further include.

The input data pre-processing unit 132-2 may pre-process the acquired data so that the data acquired by the input data acquisition unit 132-1 can be used. Specifically, the input data pre-processing unit 132-2 may process the acquired data in a predefined format so that the acquired data can be used to acquire an image of an object without defects. Alternatively, the input data pre-processing unit 132-2 may pre-process the acquired data so that the acquired data can be used to determine whether an object is defective or not and the type of defect.

The input data selection unit 132-3 may select data required for providing a response from data acquired by the input data acquisition unit 132-1 or data pre-processed by the input data pre-processing unit 132-2. The selected data may be provided to the determination result providing unit 132-4. The input data selector 132-3 may select some or all of the obtained or preprocessed data according to a preset selection criterion for providing a response. Also, the input data selector 132-3 may select data according to a preset selection criterion by learning by the model learning unit 131-4.

The model updating unit 132-5 may control the artificial intelligence model to be updated based on the evaluation of the determination result provided by the determination result providing unit 132-4. For example, the model updating unit 132-5 provides the model learning unit 132-4 with the determination result provided by the determination result providing unit 132-4, so that the model learning unit 132-4 AI models can be asked to further learn or update. In particular, the model update unit 132-5 may retrain the artificial intelligence model based on feedback information according to user input.

10 is a flowchart illustrating a method of controlling an electronic device according to an embodiment of the present disclosure.

First, a first image photographing an object is acquired (S1110). A second image including an object in a defect-free state may be obtained by using the image of the acquired object as input data of the learned first model (S1120).

Then, it may be determined whether the object is defective based on the first image and the second image (S1130 ).

Specifically, by comparing the first image and the second image, different regions may be determined between an image of an object included in the first image and an image of an object included in the second image.

In addition, images corresponding to different regions determined in the first image may be acquired.

The characteristics of different regions may be determined by using the images corresponding to the obtained different regions as input data of the learned second model. For example, it may be determined whether the characteristics of different areas are due to causes such as light bleeding or foreign matter attachment or defects of objects such as object breakage or splitting.

In addition, it is possible to determine whether the object is defective based on the determined characteristics of different regions.

Specifically, when it is determined that the image corresponding to the different region is an image of an object having a defect according to the determined characteristics of the different regions, it may be determined that the defect exists in the object.

At this time, since the second image is obtained based on the first image by the learned first model, the location of the object in the first image and the location of the object in a defect-free state in the second image are the same. can do.

Meanwhile, information on whether the determined object is defective may be displayed on the display.

In this case, when it is determined that a defect exists in the object, an area in which the defect exists in the object may be displayed in the first image so as to be distinguished from other areas.

Meanwhile, the electronic device of the embodiments of the present disclosure may have the following effects. For example, it is not easy to determine whether a defect exists or not and whether a defect exists in the object through the image of the captured object, in that the image of the object on the captured image varies according to the angle and direction in which the object is photographed. As in one embodiment of the present disclosure, an image of an object having no defect is generated through an image of the photographed object, it is determined whether a different area exists, and whether a defect exists in a different area and the type of the defect is determined. According to the electronic device, the user's convenience can be improved in that the user only needs to photograph an image of the object.

In addition, in order to train an artificial intelligence model that acquires an image of a non-defective object, a large amount of an image of a non-defective object is required. As in one embodiment of the present disclosure, a design image of an object According to an electronic device that generates an image of an object in which a defect does not exist, the user's convenience can also be improved in that the user only needs to provide the design image of the object as learning data.

The various embodiments described above may be implemented in software, hardware, or a combination thereof. According to a hardware implementation, embodiments described in the present disclosure include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions. According to the software implementation, embodiments such as procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

The method according to various embodiments of the present disclosure may be implemented by software including instructions that can be stored in a machine-readable storage media. The device is a device that can call a stored command from a storage medium and is operable according to the called command, and may include an electronic device (eg, the electronic device 100) according to the disclosed embodiments. When the instruction is executed by the processor, the processor may perform a function corresponding to the instruction directly or using other components under the control of the processor. Instructions can include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here,'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily on the storage medium.

According to one embodiment, a method according to various embodiments disclosed in this document may be provided as being included in a computer program product. Computer program products can be traded between sellers and buyers as products. The computer program product may be distributed online in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or through an application store (eg Play Store™). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored on a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

Each component (eg, module or program) according to various embodiments may be composed of a singular or a plurality of entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, modules or programs) can be integrated into one entity, performing the same or similar functions performed by each corresponding component before being integrated. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallelly, repeatedly, or heuristically executed, at least some operations may be executed in a different order, omitted, or other operations may be added. Can.

Although the preferred embodiments of the present disclosure have been described and described above, the present disclosure is not limited to the specific embodiments described above, and it is usually in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications can be made by a person who has knowledge of, and these modifications should not be individually understood from the technical idea or prospect of the present disclosure.

100: electronic device 110: camera
120: memory 130: processor

Claims (16)

In the electronic device,
camera;
A memory including at least one instruction; And
And a processor connected to the camera and the memory to control the electronic device.
The processor, by executing the at least one instruction,
Acquiring a first image obtained by photographing an object through the camera, using the acquired object first image as input data of the learned first model, obtaining a second image including the object in a defect-free state, , Determining whether the object is defective based on the first image and the second image. According to claim 1,
The processor,
By comparing the first image and the second image, in the first image, the object is different from the image of the object without the defect and the image of the object included and the image of the object included in the second image. Electronic device to judge. According to claim 2,
The processor,
Acquiring an image corresponding to the determined different region from the first image, and using the image corresponding to the obtained different region as input data of the learned second model to determine characteristics of the different region, and determining The electronic device determines whether the object is defective based on characteristics of different regions. According to claim 3,
The processor,
When it is determined that the acquired image corresponding to the different region is an image of an object having a defect according to the determined characteristic of the different region, the electronic device determines that a defect exists in the object. According to claim 1,
The position of the object in the first image and the position of the object without defects in the second image are the same as each other. According to claim 1,
Display; further comprising,
The processor,
And controlling the display to display information on whether the determined object is defective. The method of claim 6,
The processor,
If it is determined that a defect exists in the object, the electronic device controls the display to display a region in which the defect exists in the object from the first image so as to be distinguished from other regions. According to claim 1,
The first model,
It is a model trained based on the image of a state object without defects,
The image of the object without the defect,
And an electronic device generated by a third model trained based on the design image of the object. In the control method of an electronic device including a camera,
Obtaining a first image of an object taken through the camera;
Obtaining a second image including the object in a defect-free state by using the acquired first image as input data of the learned first model; And
And determining whether the object is defective based on the first image and the second image. The method of claim 9,
The step of determining whether the defect is
And comparing the first image and the second image to determine different areas between the image of the object included in the first image and the image of the object included in the second image. The method of claim 9,
Determining whether the defect,
Obtaining an image corresponding to the determined different region from the first image;
Determining characteristics of the different regions by using the image corresponding to the obtained different regions as input data of the learned second model; And
And determining whether the object is defective based on characteristics of the determined different regions. The method of claim 11,
Determining whether the defect,
And determining that a defect exists in the object when the image corresponding to the different region is determined as an image of an object having a defect according to the determined characteristics of the different regions. The method of claim 9,
The position of the object in the first image and the position of an object without defects in the second image are the same as each other. The method of claim 9,
And displaying information on whether the determined object is defective on a display. The method of claim 14,
And if it is determined that a defect exists in the object, displaying a region in which the defect exists in the object to be distinguished from other regions in the first image. The method of claim 9,
The first model,
It is a model trained based on the image of a state object without defects,
The image of the object without the defect,
The control method is generated by a third model trained based on the design image of the object.

Images