WO2019059460A1

WO2019059460A1 - Image processing apparatus and method

Info

Publication number: WO2019059460A1
Application number: PCT/KR2017/015476
Authority: WO
Inventors: 최승혁
Original assignee: 주식회사 이넘넷
Priority date: 2017-09-22
Filing date: 2017-12-26
Publication date: 2019-03-28
Also published as: JP2019061642A; KR101867586B1

Abstract

The present invention relates to an image processing apparatus and method for extracting an object cut from an original image by using artificial intelligence technology which uses a neural network learning algorithm so as to simulate functions of recognition, determination, and the like of the human brain. The image processing apparatus according to one embodiment of the present invention comprises: a first analysis unit for acquiring image division information by which an original image is divided into a plurality of regions having similar characteristics; a learning unit for identifying an object from the original image on the basis of the result of learning using a neural network, generating, by using the image division information, a classified image in which the object extracted from the original image is classified into first to third regions, and outputting a corrected image in which any one region of the classified image is clearly corrected; and a processing unit for outputting an object cut generated by calculating the original image and the corrected image, and for including the original image and the classified image in learning using the neural network.

Description

Image processing apparatus and method

The present invention relates to an image processing apparatus and method for extracting object cuts from an original image using an artificial intelligence technology that simulates functions of recognition, judgment, etc. of a human brain using a neural network learning algorithm.

As data traffic increases in the form of exponential function with the development of computer technology, artificial intelligence has become an important trend to lead future innovation. Artificial intelligence is a way to mimic the way people think, so virtually all industries are infinitely adaptable.

Representative technologies of artificial intelligence include pattern recognition, machine learning, expert system, neural network, and natural language processing. Artificial intelligence has been developed with the aim of making reasonable decision making of devices through machine learning and artificial neural network technology, which enable to increase recognition rate of big data through self learning.

The field of artificial intelligence, which started to flow in the mid-1950s, aimed to develop general artificial intelligence beyond human intelligence by the year 2000, but the optimism was gradually declining. However, since the 1990s, there has been a steady accumulation of large amounts of data, improvements in the performance of related hardware such as CPU, and the development of self-learning algorithms such as deep learning, There has been a growing interest in devices that utilize such devices.

The above-described background technology is technical information that the inventor holds for the derivation of the present invention or acquired in the process of deriving the present invention, and can not necessarily be a known technology disclosed to the general public prior to the filing of the present invention.

It is an object of the present invention to extract an object cut from an original image by using a learning result using a neural network, which is devised to solve the above-mentioned problems and / or limitations.

An image processing apparatus according to an embodiment of the present invention includes a first analyzing unit for obtaining image segmentation information obtained by dividing an original image into a plurality of regions having similar characteristics to each other; A method for identifying an object from an original image based on a learning result using a neural network and segmenting the object extracted from the original image using the image segmentation information into a first region to a third region, And outputting a corrected image in which one of the divided images is clearly corrected; And a processing unit for outputting the object cut generated by calculating the original image and the corrected image, and including the original image and the classified image into learning using the neural network.

Wherein the image processing apparatus comprises: a receiver for receiving first user input information and second user input information from the object cut in response to receipt of an invalid signal for the object cut; And an additional segment image by reinforcing a part of the segmented image for the object cut based on the image segmentation information corresponding to the first user input information and the second user input information, And a second analyzing unit for outputting an additional corrected image in which an area is clearly corrected.

The processing unit may output an additional object cut generated by calculating the original image and the additional corrected image, and may include the original image and the additional classified image in the learning using the neural network.

The image processing apparatus may repeatedly perform the operations of the receiving unit, the second analyzing unit, and the processing unit until an acknowledgment signal for the additional object cut is received.

The receiving unit may receive the first user input information for a foreground area included in the object cut and receive the second user input information for a background area included in the object cut.

An image processing method according to an embodiment of the present invention includes: obtaining image segmentation information obtained by dividing an original image into a plurality of regions having similar characteristics; A method for identifying an object from an original image based on a learning result using a neural network and segmenting the object extracted from the original image using the image segmentation information into a first region to a third region, And outputting a corrected image in which one of the classified images is clearly corrected; And outputting an object cut generated by calculating the original image and the corrected image, and including the original image and the classified image in learning using the neural network.

The image processing method comprising: receiving first user input information and second user input information from the object cut in response to receiving an impossible signal for the object cut; And an additional segment image by reinforcing a part of the segmented image for the object cut based on the image segmentation information corresponding to the first user input information and the second user input information, And outputting an additional corrected image in which a certain region is clearly corrected.

The image processing method may further include outputting an additional object cut generated by operating the original image and the additional corrected image, and including the original image and the additional classified image in the learning using the neural network .

The image processing method may repeatedly perform the operations of receiving, outputting, and including until an acknowledgment signal for the additional object cut is received.

Wherein the receiving comprises: receiving the first user input information for a foreground region included in the object cut; And receiving the second user input information for the background area included in the object cut.

In addition, other methods for implementing the invention, other systems, and computer programs for executing the methods may be further provided.

Other aspects, features, and advantages will become apparent from the following drawings, claims, and detailed description of the invention.

According to the embodiments, although the object cut is extracted by manually inputting the existing user input information, in the present embodiment, the object cut is automatically extracted from the original image and provided by using the learning result using the neural network, It is convenient to extract object cuts conveniently without intervention.

In addition, the object cut is automatically extracted from the original image using the learning result using the neural network, and when the user's satisfaction with the extracted object cut is lowered, the user intervenes to extract the additional object cut, User satisfaction with cut can be improved.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

FIG. 1 is a view for schematically explaining an image processing system according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining a detailed configuration of an image processing apparatus in the image processing system of FIG. 1; FIG.

FIG. 3 is a diagram for explaining a detailed configuration of an artificial intelligent processing unit according to an embodiment of the image processing apparatus of FIG. 2. Referring to FIG.

FIG. 4 is a view for schematically explaining a detailed configuration of a first analysis unit of the artificial intelligence processing unit of FIG. 3; FIG.

FIG. 5 is a diagram for explaining a detailed configuration of an artificial intelligent processing unit according to another embodiment of the image processing apparatus of FIG. 2. FIG.

FIG. 6 is a diagram for explaining a detailed configuration of a second analysis unit of the artificial intelligence processing unit of FIG. 5; FIG.

FIG. 7 is a diagram for explaining a detailed configuration of a user terminal in the image processing system of FIG. 1. FIG.

8A to 8H are views showing examples of images processed by the image processing apparatus.

9 to 12 are flowcharts for explaining an image processing method according to an embodiment of the present invention.

Brief Description of the Drawings The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described in conjunction with the accompanying drawings. It should be understood, however, that the present invention is not limited to the embodiments set forth herein, but may be embodied in many different forms and includes all conversions, equivalents, and alternatives falling within the spirit and scope of the present invention . BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects of the present invention will become more apparent by describing in detail preferred embodiments thereof with reference to the attached drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof. The terms first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. .

FIG. 1 is a view for schematically explaining an image processing system according to an embodiment of the present invention. Referring to FIG. 1, the image processing system 1 may include an image processing apparatus 100, a user terminal 200, and a communication network 300.

The image processing apparatus 100 according to an embodiment of the present invention acquires image division information divided into a plurality of regions having similar characteristics to an original image and identifies the object from the original image based on the learning result using the neural network A division image in which the object extracted from the original image is divided into the first region to the third region by using the image division information, a corrected image in which any one of the division images is clearly corrected is output, The original image, the classified image, and the object cut can be included in the learning using the neural network in response to receipt of the confirmation signal for the generated object cut. Included in the learning using the neural network may include using the original image and the classified image as learning data using a neural network.

According to an embodiment, the image processing apparatus 100 receives the first user input information and the second user input information from the object cut in response to the receipt of the impossible signal for the generated object cut, The additional divided image is generated based on the first user input information and the image division information corresponding to the second user input information to generate an additional corrected image in which any one of the additional divided images is clearly corrected, And outputs the additional object cut generated by calculating the original image and the additional corrected image. In response to receipt of the confirmation signal for the generated additional object cut, the original image and the additional classified image can be included in the learning using the neural network have. Herein, the image processing apparatus 100 receives the first user input information and the second user input information until an acknowledgment signal for the additional object cut is received, generates an additional discriminated image for the foreground region, and outputs an additional corrected image And outputting additional object cuts are repeatedly performed, and the original image and the additional classified image can be included in the learning using the neural network.

The user terminal 200 may display an image processing web page and / or an image processing application provided by the image processing apparatus 100. To this end, the image processing apparatus 100 may transmit the image processing web page and / or image processing application to the user terminal 200 as the image display apparatus through the communication network 300. Upon receiving the user's connection identification information (ID) and password through the user terminal 200, the image processing apparatus 100 can perform user authentication to the image processing web page and / or image processing application.

The user terminal 200 can transmit the original image to the image processing apparatus 100. [ The user terminal 200 can select an image stored therein as an original image and transmit the selected image to the image processing apparatus 100. For example, the user terminal 200 can execute a photo album application or the like to select a previously stored image as an original image. Also, the user terminal 200 can receive an image from an external server and select the original image. For example, the user terminal 200 may access a social network server, a cloud server, or a content providing server to download images. Also, the user terminal 200 can capture an image using a camera provided therein and select the captured image as an original image. At this time, the user terminal 200 can execute a camera application to capture an image.

The user terminal 200 can transmit an acknowledgment signal and / or an invalid signal to the object cut received from the image processing apparatus 100. [ When the user terminal 200 transmits a disable signal to the image processing apparatus 100, the first user input information and the second user input information may be transmitted at the request of the image processing apparatus 100. The transmission of the first user input information and the second user input information may be repeated until the user terminal 200 transmits an acknowledgment signal to the image processing apparatus 100 for the object cut.

The user terminal 200 includes a desktop computer 201 (Fig. 1), a smart phone 202, a notebook computer 203, a tablet PC, a smart TV, a personal digital assistant an assistant, a laptop, a media player, a micro server, a global positioning system (GPS) device, an electronic book terminal, a digital broadcast terminal, a navigation device, a kiosk, an MP3 player, a digital camera, , But is not limited thereto. In addition, the user terminal 200 may be a wearable terminal having a communication function and a data processing function, such as a watch, a pair of glasses, a hair band, and a ring. The user terminal 200 is not limited to the above description, and a terminal capable of web browsing as described above can be borrowed without limitation.

Meanwhile, the communication network 300 connects the user terminal 200 with the image processing apparatus 100. That is, the communication network 400 may refer to a communication network that provides a connection path so that the user terminal 200 can access the image processing apparatus 100 and transmit / receive predetermined information. The communication network 300 may be a wired network such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, But the scope of the present invention is not limited thereto.

FIG. 2 is a view for schematically explaining the detailed configuration of the image processing apparatus 100 of the image processing system 1 of FIG. 2, the image processing apparatus 100 includes a communication unit 110, a storage medium 120, a program storage unit 130, a control unit 140, a database 150, and an artificial intelligence processing unit 160 .

The communication unit 110 may provide a communication interface required to provide a transmission / reception signal between the image processing apparatus 100 and the user terminal 200 in the form of packet data in cooperation with the communication network 300. Further, the communication unit 110 can receive a predetermined information request signal from the user terminal 200 and transmit the information processed by the artificial intelligence processing unit 160 to the user terminal 200 . Here, the communication network is a medium for connecting the image processing apparatus 100 and the user terminal 200, and is a medium for allowing the user terminal 200 to access the image processing apparatus 100, And a path for providing a connection path. The communication unit 110 may be a device including hardware and software necessary for transmitting / receiving signals such as a control signal or a data signal through a wired / wireless connection with other network devices.

The storage medium 120 performs a function of temporarily or permanently storing data processed by the control unit 140. Here, the storage medium 120 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. The storage medium 120 may include internal memory and / or external memory, and may be a volatile memory such as a DRAM, an SRAM, or an SDRAM, an OTPROM (one time programmable ROM), a PROM, an EPROM, an EEPROM, , NAND flash memory, or NOR flash memory, and the like. A flash drive such as a compact flash (CF) card, an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as a HDD. In this embodiment, the storage medium 120 may include one or more instructions that configure the neural network, and one or more instructions that control the neural network.

The program storage unit 130 may include an operation of obtaining image segmentation information obtained by dividing the original image received from the user terminal 200 into a plurality of regions having similar characteristics, an operation of extracting an object from the original image based on the learning result using the neural network An operation of generating a division image in which the object extracted from the original image is divided into a first region to a third region using the image division information, a correction image in which any one of the division images is clearly corrected, A task of outputting an object cut generated by calculating an original image and a corrected image, an operation of including an original image and a classified image in learning using a neural network, and a receiving operation of a user terminal 200) for requesting and receiving first user input information and second user input information, And a control software for performing an operation of generating an additional classification image in which a part of the classification image of the object cut is clearly corrected using the user input information, the second user input information, and the image division information.

The database 150 stores the original image received from the user terminal 200 and various images and / or information generated by the artificial intelligence processing of the image processing apparatus 100, for example, image division information on the original image, , The corrected image, and the object cut as training data for the neural network. In addition, the database 150 stores a series of process information (for example, a process of generating an additional object cut) based on the first user input information and the second user input information received from the user in response to the disable signal for the object cut, Additional classification image, additional correction image, additional object cut) can be stored as learning data for the neural network.

Further, the database 150 may further include a user database for storing user information. Here, the user database may store user information for a user who wants to use a service for extracting an object cut from an original image. Here, the user information includes basic information about the user such as the user's name, affiliation, personal information, gender, age, contact, e-mail, address, and authentication information (login) such as ID (or e-mail) and password Information about a connection country, a connection location, information on a device used for connection, information related to connection such as a connected network environment, and the like.

The artificial intelligence processing unit 160 extracts and provides an object cut from the original image based on the learning result using the neural network and may include the information and / or the image generated for the object cut extraction in the learning using the neural network . The intelligent processing unit 160 extracts and provides the additional object cut using the first user input information and the second user input information received from the user when receiving the impossible signal from the user for the extracted object cut, The information and / or the image generated for the neural network can be included in the learning using the neural network, and the object cut extraction process can be repeated until the confirmation signal is received from the user.

Artificial intelligence (AI) technology is a computer processing technology that implements human-level intelligence. Unlike conventional rule-based smart technology, it is a technology that machines learn, judge and become smart. Artificial intelligence technology has become more and more recognizable as users use it, and existing rule-based smart systems are increasingly being replaced by deep-run-based artificial intelligence systems.

Artificial intelligence technology can be composed of element technologies that utilize deep learning and machine learning. Machine learning is an algorithm technology that classifies / learns the characteristics of input data by itself. Element technology is a technology that simulates functions such as recognition and judgment of human brain using machine learning algorithms such as deep learning. Understanding, reasoning / prediction, knowledge representation, motion control, and the like.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology for recognizing, applying / processing human language / character and may include natural language processing, machine translation, dialogue system, query response, speech recognition / synthesis, and the like. Visual understanding is a technology for recognizing and processing objects as human vision, and may include object identification, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, and the like. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge / probability based reasoning, optimization prediction, preference base planning, and recommendation. Knowledge representation is a technology that automates the processing of human experience information into knowledge data, which can include knowledge building (data generation / classification) and knowledge management (data utilization). The motion control is a technique for controlling the autonomous travel of the vehicle and the motion of the robot, and may include motion control (navigation, collision, traveling), operation control (behavior control), and the like.

Generally, in order to extract an object cut from an original image, manual intervention of a user is essential. In this embodiment, an object cut is automatically extracted from an original image using a learning result using a neural network based on artificial intelligence It is possible to extract the object cuts conveniently without user intervention.

FIG. 3 is a diagram for explaining a detailed configuration of the artificial intelligence processing unit 160 according to an embodiment of the image processing apparatus 100 of FIG. Referring to FIG. 3, the AI processing unit 160 may include a first analyzing unit 161, a learning unit 162, and a processing unit 163.

The first analysis unit 161 can obtain the image segmentation information obtained by dividing the original image into a plurality of regions having similar characteristics. The first analysis unit 161 finds at least one region having a similar brightness, edge, color, or the like around the seed as a seed at an arbitrary position in the original image, If the regions adjacent thereto have the same characteristics, the regions are integrated into one region, and the regions having the same characteristics are gradually grown, and finally the entire original image is divided into a plurality of regions having similar characteristics. The first analysis unit 161 may store the acquired image segmentation information in the database 150. [

FIG. 4 is a diagram illustrating a detailed configuration of the first analysis unit 161 of the artificial intelligence processing unit 160 of FIG. 3. Referring to FIG. 4, the first analyzing unit 161 includes a setting unit 161-1, a calculating unit 161-2, a clustering unit 161-3, a first generating unit 161-4, And a generation unit 161-5.

The setting unit 161-1 can set the first parameter and the second parameter for acquiring the image segmentation information from the original image (Fig. 8A). Here, the first parameter may include the number of seeds, and the first parameter may be set and received from the user terminal 200, or may be set by calculating the size of the original image divided by the number of pixels in the area, It may be set to a random value for each operation. And the second parameter may include a repetition number for calculating the distance from each seed to each of all the pixels. If the distance calculation is continuously repeated without designating the number of repetitions, the throughput increases and the capacity shortage of the storage medium 120 occurs. Therefore, it may be required to set an appropriate number of repetitions. The second parameter may be received and set from the user terminal 200, or may be set to a default value.

The calculation unit 161-2 can calculate the distance from each seed to each of all the pixels and express the distance calculation result in Lab color. The number of repetitions of the distance calculation of the calculation unit 161-2 can be repeatedly performed by the set second parameter.

The clustering unit 161-3 clusters the distance calculation results of each seed to each of all the pixels repeatedly performed by the second parameter, and includes a pixel having similar Lab color (distance calculation result) in the original image in one area . In this way, the original image can be divided into a plurality of regions having similar Lab colors.

The first generating unit 161-4 can generate the image division index information in which an index is attached to each of a plurality of areas similar in Lab color. FIG. 8B shows an example in which the image segmentation index information image generated from the original image (FIG. 8A) is expressed in color. The first generation unit 161-4 may store the generated image division index information in the database 150. [

The second generation unit 161-5 generates connection information that links the average pixel value calculated from each of a plurality of similar Lab color regions and the image division index information of the four surrounding azimuths searched around one reference region And image division information including the image division index information generated by the first generation unit 161-4. FIG. 8C shows an example of an image segmentation information image generated using an original image (FIG. 8A) and an image segmentation index information image (FIG. 8B). The second generation unit 161-5 can store the generated image division information in the database 150. [

Returning to FIG. 3, the learning unit 162 can identify the object from the original image based on the learning result using the neural network. For this, the learning unit 162 may further include a neural network module (not shown). Here, the neural network may be a set of algorithms for identifying and / or determining objects in the original image by extracting and using various attributes in the original image, using the results of statistical machine learning. The neural network can identify objects in the original image by abstracting various attributes contained in the original image input to the neural network. In this case, abstracting the attributes in the source image may be to detect attributes from the source image and determine key attributes among the detected attributes.

For example, the learning unit 162 may input an original image and / or a classification image (also including an additional classification image) into a neural network, and classify the location of the object included in the original image and / Can be output from the network.

Specifically, the learning unit 162 detects predetermined image attributes in the original image and / or the classified image according to the learning result using the neural network, and detects the position and / or the position of the object in the original image based on the detected image attributes. Or the category of the object. Here, the image attribute may include a color, an edge, a polygon, a saturation, a brightness, etc. constituting an image, but the image attribute is not limited thereto.

Meanwhile, the learning unit 162 may learn the neural network to identify one or more objects from the original image and / or the classified image, in order to use the neural network. For example, the learning unit 162 repeatedly performs an operation of analyzing and / or evaluating the results of map learning and / or non-image learning (or autonomous learning or active learning) on object-specific image attributes in the neural network The neural network can be learned. The learning unit 162 can utilize the original image and the classified image as learning data in neural network learning for object identification. Here, the classification image may be the final classification image, and the final classification image may include a classification image and / or an additional classification image for the object cut in which a confirmation signal described later is received.

The learning unit 162 can generate a classification image in which the object is divided into the first to third regions by using the object identified using the neural network and the image division information acquired by the first analysis unit 161 have. Wherein the first region may include a foreground region of the identified objects and may be represented by a first value (e.g., white). Also, the second area may include a background area of the identified objects, and may be represented by a second value (e.g., black). In addition, the third area may include an unclear area, which is uncertain whether it is the first area or the second area of the identified objects, and may be represented by a third value (for example, gray).

The learning unit 162 can generate and output a corrected image that is obtained by clearly correcting one of the divided images classified into the first to third regions. Herein, one of the regions may include a third region, and when a part of the third region is included in the first region, the correction image may correct a portion of the third region to the first region, And a part of the third area is included in the second area and the other part of the third area is corrected to the second area. The learning unit 162 can generate a corrected image from the classified image using the correlation between the original image and the classified image.

In another embodiment, the learning unit 162 may generate segmented images by comparing image segmentation information of an object and an object identified from the original image through semantic segmentation based on a learning result using a neural network, It is also possible to generate and output a corrected image in which one of the images is clearly corrected.

In this embodiment, at least one of the first analysis unit 161 and the learning unit 162 may be manufactured in at least one hardware chip form and mounted on the electronic device. For example, at least one of the first analysis unit 161 and the learning unit 162 may be manufactured in the form of a dedicated hardware chip for artificial intelligence, or may be an existing general purpose processor (e.g., a CPU or an application processor) It can be built as part of a dedicated processor (eg GPU) and loaded onto various electronic arches.

The processing unit 163 performs an AND operation on the original image and the corrected image, and outputs the generated object cut to the user terminal 200 as a result of the operation. The processing unit 163 may store the original image and the classified image in the database 150 and include the same in the learning using the neural network. In addition, the processing unit 163 can store the generated object cut in the database 150. [

FIG. 5 is a diagram illustrating a detailed configuration of the artificial intelligence processing unit 160 according to another embodiment of the image processing apparatus 100 of FIG. 5, the artificial intelligence processing unit 160 may include a first analyzing unit 161, a learning unit 162, a processing unit 163, a receiving unit 164, and a second analyzing unit 165.

The first analysis unit 161 can obtain the image segmentation information obtained by dividing the original image into a plurality of regions having similar characteristics.

The learning unit 162 identifies the object from the original image based on the learning result using the neural network, generates the classification image for the object extracted from the original image using the image segmentation information, It is possible to generate and output a corrected image in which an area is clearly corrected.

Hereinafter, the operations of the first analyzing unit 161 and the learning unit 162 are the same as those of the above-described FIG. 3 and will not be described.

The processing unit 163 may output the object cut generated by the logical product of the original image and the corrected image to the user terminal 200 and store the original image, the corrected image, and the object cut in the database 150. [

The receiving unit 164 may receive an acknowledgment signal or an unavailable signal for the object cut output to the user terminal 200. [ The receiving unit 164 may receive the first user input information and the second user input information from the user terminal 200.

When the receiving unit 164 receives the confirmation signal from the user terminal 200, the processing unit 163 may store the original image and the classified image in the database 150 and include the same in the learning using the neural network. FIG. 8D shows an example of an object cut receiving an acknowledgment signal from the user terminal 200. FIG.

When the receiving unit 164 receives an invalid signal from the user terminal 200, the second analyzing unit 165 starts the operation. At the same time, the processing unit 163 transmits the first user input information to the user terminal 200, And may request the second user input information input.

The second analyzing unit 165 analyzes the classification image of the object cut extracted from the database 150 through the processing unit 163 and the first user input information and the second user input information received from the receiving unit 164, An additional discrimination image reinforcing a part of the object included in the discrimination image is generated using the image segmentation information received from the discrimination unit 161 and an additional correction image in which one of the additional discrimination images is clearly corrected is outputted .

FIG. 8E shows an example of an object cut that receives a disable signal from the user terminal 200. FIG. Here, the first user input information may include a user input, for example, a first drag 810 for specifying a foreground region from an object cut received from the user terminal 200 as shown in FIG. 8F . In addition, the second user input information, as shown in FIG. 8F, specifies a foreground region from the object cut that received the cancellation signal output to the user terminal 200, and then inputs a user input specifying the background region, for example, 820 < / RTI > Also, the first user input information and the second user input information may be represented by different colors.

FIG. 6 is a diagram for explaining a detailed configuration of the second analysis unit 165 of the artificial intelligence processing unit 160 of FIG. 6, the second analyzing unit 165 may include a third generating unit 165-1 and a fourth generating unit 165-2. The second analyzing unit 165 may include a learning unit 162).

The third generation unit 165-1 generates a classification image of the object cut extracted from the database 150 through the processing unit 163, first user input information and second user input information received from the receiving

unit

164, 1 analysis unit 161 to generate an additional classification image reinforcing a part of the object included in the classification image.

As shown in FIG. 8F, a part of the object region of the object cut that has received the impossible signal output to the user terminal 200 is not outputted. The third generation unit 165-1 may extract the foreground region of the object cut from the first user input information for the classification image of the object cut and extract the background region of the object cut from the second user input information. The third generation unit 165-1 includes the position and pixel value of the foreground region of the object cut (the object in which the output of one part is missing) and the average pixel value, connection information, and image segmentation index information of each of the divided regions It is possible to generate an additional classification image in which the output of one portion is reinforced. FIG. 8G shows an example of an additional classification image reinforcing the output of a part of the object cut that receives the impossible signal.

In addition, the additional classification image is divided into a first region to a third region, wherein the first region may include a foreground region of the object, and may be represented by a first value (for example, white). Also, the second area may include a background area of objects, and may be displayed as a second value (for example, black). In addition, the third region may include an unclear region that is unclear, whether it is a first region or a second region, and may be represented by a third value (for example, gray).

The fourth generation unit 165-2 may generate a corrected image that is obtained by clearly correcting any one of the additional classification images divided into the first to third regions, and output the generated corrected image to the processing unit 163. [ Herein, one of the regions may include a third region, and when a part of the third region is included in the first region, a portion of the third region is corrected to the first region, And if another part is included in the second area, the other part of the third area is corrected to the second area. The fourth generation unit 165-2 may generate a correction image from the additional classification image using the correlation between the original image and the additional classification image. FIG. 8H shows an example of the corrected image generated for the additional classification image (FIG. 8G).

In this embodiment, at least one of the first analyzing unit 161, the learning unit 162, and the second analyzing unit 165 may be manufactured in at least one hardware chip form and mounted on the electronic device. For example, at least one of the first analysis unit 161, the learning unit 162, and the second analysis unit 165 may be manufactured in the form of a dedicated hardware chip for artificial intelligence, : A CPU or an application processor) or a graphics processor (e.g., a GPU) and may be mounted on various electronic arches.

Returning to FIG. 5, the processing unit 163 performs an AND operation on the original image and the additional correction image, and outputs the generated additional object cut to the user terminal 200. FIG. Thereafter, when the receiving unit 164 receives the confirmation signal from the user terminal 200, the processing unit 163 may store the original image and the additional classification image in the database 150 and include the same in the learning using the neural network . Herein, a process of receiving first user input information and second user input information until an acknowledgment signal for an additional object cut is received, generating an additional classification image for a foreground region and outputting an additional correction image, And the original image and the additional classification image can be included in the learning result using the neural network.

FIG. 7 is a diagram for explaining a detailed configuration of the user terminal 200 of the image processing system 1 of FIG. Referring to FIG. 7, the user terminal 200 may include a communication unit 210, a memory 220, an input / output unit 230, a program storage unit 240, a control unit 250, and a display unit 260.

The communication unit 210 may be a device including hardware and software necessary to transmit / receive a signal such as a control signal or a data signal through a wired / wireless connection with another network device such as the image processing apparatus 100. For example, the communication unit 210 may include a local communication unit or a mobile communication unit. A short-range wireless communication unit includes a Bluetooth communication unit, a Bluetooth low energy (BLE) communication unit, a near field communication unit, a WLAN communication unit, a Zigbee communication unit, Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, an Ant + communication unit, and the like. The mobile communication unit transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data depending on a voice call signal, a video call signal, or a text / multimedia message transmission / reception.

The memory 220 may temporarily or permanently store data processed by the controller 250 or may temporarily or permanently store data transmitted to the user terminal 200. [ Here, the memory 220 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto.

The input / output unit 230 may include a touch recognition display controller or various other input / output controllers. As an example, the touch-aware display controller may provide an output interface and an input interface between the device and the user. The touch-sensitive display controller can transmit and receive electrical signals to and from the control unit 250. [ Additionally, the touch-aware display controller may display a visual output to the user, and the visual output may include text, graphics, images, video, and combinations thereof. The input / output unit 230 may be a display member such as an organic light emitting display (OLED) or a liquid crystal display (LCD) capable of touch recognition.

The program storage unit 240 may include an operation of selecting an original image and transmitting it to the image processing apparatus 100, an operation of receiving and displaying an object cut and / or an additional object cut from the image processing apparatus 100, an object cut and / An operation of transmitting an acknowledgment signal or an invalid signal to the additional object cut, a task of receiving first user input information and second user input information input to the object cut and / or the additional object cut and transmitting the same to the image processing apparatus 100 Can be mounted.

The control unit 250 may provide various functions such as driving control software installed in the program storage unit 240 as a kind of central processing unit and controlling the display unit 260 to display predetermined information. Here, the control unit 250 may include all kinds of devices capable of processing data, such as a processor. Herein, the term " processor " may refer to a data processing device embedded in hardware, for example, having a circuit physically structured to perform the functions represented by the code or instructions contained in the program. As an example of the data processing apparatus built in hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC) circuit, and a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

Under the control of the control unit 250, the display unit 260 displays various information received from the image processing apparatus 100, for example, image processing web page and / or image processing application related information provided by the image processing apparatus 100, The first user input information and the second user input information on the original image to be transmitted to the image processing apparatus 100, the object cut and / or the additional object cut received from the image processing apparatus 100, the object cut and / And the like can be displayed.

8A to 8H are views showing examples of images processed by the image processing apparatus. 8A shows an example of an original image, FIG. 8B shows an example of color representation of an image segmentation index information image generated from an original image (FIG. 8A), FIG. 8C shows an original image And an image segmentation information image generated using the image segmentation index information image (FIG. 8B). FIG. 8D shows an example of an object cut that receives an acknowledgment signal from the user terminal 200, and FIG. 8E shows an example of an object cut that receives a disable signal from the user terminal 200. FIG. FIG. 8F shows an example of inputting the first user input information 810 and the second user input information 820 to the object cut receiving the impossible signal. FIG. 8G shows an example of an additional classification image reinforcing the output of a part of the object cut that receives the impossible signal. FIG. 8H shows an example of the supplementary classification image (FIG. 8G) FIG.

9 is a flowchart illustrating an image processing method according to an embodiment of the present invention. In the following description, the description of the parts overlapping with those of FIGS. 1 to 8 will be omitted.

In step S910, the image processing apparatus 100 acquires image division information that is divided into a plurality of regions having similar characteristics to the original image. The image processing apparatus 100 may perform the following processing for acquiring image segmentation information. The image processing apparatus 100 can set the first parameter including the number of seeds and the second parameter including the number of iterations for the distance calculation to each of all the pixels in each seed. The image processing apparatus 100 can calculate the distance from each seed to each of all the pixels and express the distance calculation result in Lab color. The image processing apparatus 100 clusters the distance calculation results of each seed to each of all the pixels repeatedly performed by the second parameter, and includes a pixel having similar Lab color (distance calculation result) in the original image into one area . The image processing apparatus 100 can generate image division index information in which an index is added to each of a plurality of areas similar in Lab color. The image processing apparatus 100 is configured to obtain connection information in which the average pixel value calculated from each of a plurality of similar Lab color regions and the image division index information of the four surrounding azimuths searched around a certain reference region, It is possible to generate the image division information including the division index information.

In step S920, the image processing apparatus 100 identifies the object from the original image based on the learning result using the neural network, and divides the object extracted from the original image by using the image segmentation information into a first area including the foreground area, A second region including a region and a third region including an unascertained region, and generates a corrected image in which the third region of the divided image is clearly corrected.

In step S930, the image processing apparatus 100 outputs the object cut generated by performing an AND operation between the original image and the corrected image to the user terminal 200, and includes the original image and the classified image in the learning using the neural network.

10 is a flowchart illustrating an image processing method according to another embodiment of the present invention. In the following description, the description of the parts overlapping with those of FIGS. 1 to 9 will be omitted.

In step S1010, the image processing apparatus 100 acquires image division information that is divided into a plurality of regions having similar characteristics to the original image.

In step S1020, the image processing apparatus 100 identifies the object from the original image based on the learning result using the neural network, and extracts the object extracted from the original image using the image segmentation information as a first region including the foreground region, A second region including a region and a third region including an unascertained region, and generates a corrected image in which the third region of the divided image is clearly corrected.

In step S1030, the image processing apparatus 100 performs an AND operation on the original image and the corrected image, and outputs the generated object cut to the user terminal 200. [

In step S1040, the image processing apparatus 100 determines whether it has received an invalid signal for the object cut.

In step S1050, when the image processing apparatus 100 receives the confirmation signal for the object cut, the original image and the classification image are included in the learning using the neural network.

In step S1060, when the image processing apparatus 100 receives the invalid signal for the object cut, the first user input for specifying the foreground area from the object cut receiving the impossible signal output from the object cut to the user terminal 200 And second user input information for designating a background area from the object cut received from the user terminal 200.

In step S1070, the image processing apparatus 100 adds the object cut segment image, the first user input information, the second user input information, and the image segmentation information so as to reinforce a part of the object included in the segmented image And generates an additional corrected image in which one of the additional classified images is clearly corrected. The image processing apparatus 100 divides the position and the pixel value of the foreground region of the object cut (the object in which the output of one part is missing) and the image including the average pixel value, connection information, It is possible to retrieve from the division information and generate an additional classification image reinforcing the output of one part.

In step S1080, the image processing apparatus 100 performs an AND operation on the original image and the additional correction image, and outputs the additional object cut generated as a result of the operation to the user terminal 200. [ Here, steps S1040 to S1080 are repeatedly performed until an acknowledgment signal for the additional object cut is received.

11 is a flowchart illustrating an image processing method according to another embodiment of the present invention. In the following description, the description of the parts overlapping with those of FIGS. 1 to 10 will be omitted.

In step S1110, the user terminal 200 accesses the image processing web page provided by the image processing apparatus 100 or executes the image processing application provided by the image processing apparatus 100. [

In step S1120, the user terminal 200 selects an original image and transmits it to the image processing apparatus 100. [ The user terminal 200 can execute a photo album application or the like to select a pre-stored image as an original image. Also, the user terminal 200 can receive an image from an external server and select the original image. Also, the user terminal 200 can capture an image using a camera provided therein and select the captured image as an original image.

In step S1130, the image processing apparatus 100 that has received the original image acquires image division information divided into a plurality of regions having similar characteristics to the original image.

In step S1140, the image processing apparatus 100 identifies the object from the original image based on the learning result using the neural network.

In step 1150, the image processing apparatus 100 extracts an object extracted from the original image using the image segmentation information by using a first region including a foreground region, a second region including a background region, and a third region including an un- And generates a corrected image in which the third region of the divided image is clearly corrected.

In step S1060, the image processing apparatus 100 performs an AND operation between the original image and the corrected image.

In step 1070, the image processing apparatus 100 transmits the generated object cut to the user terminal 200.

In step 1080, the user terminal 200 transmits an acknowledgment signal for the object cut.

In step 1090, the image processing apparatus 100 includes the original image and the classified image in the learning using the neural network.

12 is a flowchart illustrating an image processing method according to another embodiment of the present invention. In the following description, the description of the parts that are the same as those in the description of Figs. 1 to 11 will be omitted.

In step S1211, the user terminal 200 accesses the image processing web page provided by the image processing apparatus 100 or executes the image processing application provided by the image processing apparatus 100. [

In step S1213, the user terminal 200 selects an original image and transmits it to the image processing apparatus 100. [

In step S1215, the image processing apparatus 100 that has received the original image acquires image division information (for example, super pixel map information) obtained by dividing the original image into a plurality of regions having similar characteristics.

In step S1217, the image processing apparatus 100 identifies the object from the original image based on the learning result using the neural network. Here, the image processing apparatus 100 displays a boundary box including the identified object, detects the contour of the object, compares the boundary box and the contour line, and adjusts the size of the boundary box so that the contour line is included in the boundary box , The image processing apparatus can further perform the operation of cutting the boundary box using the image segmentation information.

In step 1219, the image processing apparatus 100 extracts an object extracted from the original image using the image segmentation information by using a first region including a foreground region, a second region including a background region, and a third region including an un- (E.g., a tri-map image), and generates a corrected image (e.g., matting image) in which the third region of the divided image is clearly corrected. Here, the image processing apparatus 100 may generate a segmented image and a corrected image with respect to the image in the boundary box in which the cutting is performed.

In step S1221, the image processing apparatus 100 performs an AND operation on the original image and the corrected image to generate an object cut.

In step 1223, the image processing apparatus 100 transmits the generated object cut to the user terminal 200.

In step 1225, the user terminal 200 transmits a disable signal for the object cut.

In step S127, the image processing apparatus 100 requests the user terminal 200 to transmit the first user input information and the second user input information.

In step 1229, the user terminal 200 transmits the first user input information and the second user input information to the image processing apparatus 100.

In step 1231, the image processing apparatus 100 extracts the foreground region from the segmented image of the object cut that received the invalid signal using the first user input information, and extracts the foreground region using the second user input information The background region is extracted from the cut image.

In step 1233, the image processing apparatus 100 calculates the position and the pixel value of the foreground region (the object in which the output of one part is missing) of the object cut, the average pixel value, the connection information, An additional discrimination image reinforcing the output of one part is generated and an additional correction image in which the third discrimination image is clearly corrected is generated.

In step S1235, the image processing apparatus 100 performs an AND operation on the original image and the additional correction image to generate an additional object cut.

In step S1237, the image processing apparatus 100 transmits the generated additional object cut to the user terminal 200. [

In step S1239, the image processing apparatus 100 repeatedly performs steps S1227 to S1237 until an acknowledgment signal for the additional object cut is received from the user terminal 200, and when an acknowledgment signal is received from the user terminal 200 The original image and the additional classification image are included in the learning using the neural network.

The embodiments of the present invention described above can be embodied in the form of a computer program that can be executed on various components on a computer, and the computer program can be recorded on a computer-readable medium. At this time, the medium may be a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floptical disk, , A RAM, a flash memory, and the like, which are specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention or may be known and used by those skilled in the computer software field. Examples of computer programs may include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter or the like.

The use of the terms " above " and similar indication words in the specification of the present invention (particularly in the claims) may refer to both singular and plural. In addition, in the present invention, when a range is described, it includes the invention to which the individual values belonging to the above range are applied (unless there is contradiction thereto), and each individual value constituting the above range is described in the detailed description of the invention The same.

Unless there is explicitly stated or contrary to the description of the steps constituting the method according to the invention, the steps may be carried out in any suitable order. The present invention is not necessarily limited to the order of description of the above steps. The use of all examples or exemplary language (e.g., etc.) in this invention is for the purpose of describing the present invention only in detail and is not to be limited by the scope of the claims, It is not. It will also be appreciated by those skilled in the art that various modifications, combinations, and alterations may be made depending on design criteria and factors within the scope of the appended claims or equivalents thereof.

Accordingly, the spirit of the present invention should not be construed as being limited to the above-described embodiments, and all ranges that are equivalent to or equivalent to the claims of the present invention as well as the claims .

Embodiments of the present invention relate to an image processing apparatus and method, in which an object cut is automatically extracted from an original image by using a learning result using a neural network, so that an object cut can be conveniently extracted without user intervention The present invention can be applied to an image processing apparatus and method.

Claims

A first analyzing unit for obtaining image segmentation information obtained by dividing the original image into a plurality of regions having similar characteristics to each other;

A method for identifying an object from an original image based on a learning result using a neural network and segmenting the object extracted from the original image using the image segmentation information into a first region to a third region, And outputting a corrected image in which one of the divided images is clearly corrected; And

And a processing unit for outputting the object cut generated by calculating the original image and the corrected image, and including the original image and the classified image into learning using the neural network.
The method according to claim 1,

A receiving unit receiving first user input information and second user input information from the object cut in response to receipt of an invalid signal for the object cut; And

Wherein the image processing apparatus further comprises: an additional segment image generation unit that generates a segmentation image by reinforcing a part of the segmented image for the object cut based on the image segmentation information corresponding to the first user input information and the second user input information, And a second analyzing unit for outputting an additional corrected image in which an area is clearly corrected.
The image processing apparatus according to claim 2,

And outputting the additional object cut generated by calculating the original image and the further corrected image, and incorporating the original image and the additional classified image into the learning using the neural network.
The method of claim 3,

And repeats the operations of the receiving unit, the second analyzing unit, and the processing unit until an acknowledgment signal for the additional object cut is received.
3. The apparatus of claim 2,

And receives the first user input information with respect to the foreground region included in the object cut and receives the second user input information with respect to the background region included in the object cut.
Obtaining image segmentation information obtained by dividing the original image into a plurality of regions having similar characteristics to each other;

A method for identifying an object from an original image based on a learning result using a neural network and segmenting the object extracted from the original image using the image segmentation information into a first region to a third region, And outputting a corrected image in which one of the classified images is clearly corrected; And

And outputting an object cut generated by operating the original image and the corrected image, and including the original image and the classified image in learning using the neural network.
The method according to claim 6,

Receiving first user input information and second user input information from the object cut in response to receipt of an invalid signal for the object cut; And

Wherein the image processing apparatus further comprises: an additional segment image generation unit that generates a segmentation image by reinforcing a part of the segmented image for the object cut based on the image segmentation information corresponding to the first user input information and the second user input information, And outputting an additional corrected image in which an area is clearly corrected.
8. The method of claim 7,

And outputting the additional object cut generated by operating the original image and the further corrected image, and including the original image and the additional classified image in the learning using the neural network. Way.
9. The method of claim 8,

And repeating the operations of receiving, outputting, and embedding until an acknowledgment signal for the additional object cut is received.
8. The method of claim 7,

Receiving the first user input information for a foreground area included in the object cut; And

And receiving the second user input information for a background area included in the object cut.
A computer program stored in the computer-readable medium for executing the method of any one of claims 6 to 10 using a computer.