KR102642894B1 - Method, apparatus and computer programs for identifying illustration images - Google Patents

Abstract

A drawing image identification method according to an embodiment of the present invention includes the steps of extracting an analysis target area from an image, and using a texture classification model to determine that the analysis target area includes an object with the texture of an art tool and a background object, It includes the step of identifying an image as a drawing image, and the texture classification model is trained with a learning image consisting of a set of a plurality of drawing images containing at least one of a point, line, or plane drawn with the texture of an art tool and a region of a background object. It is characterized as a machine learning model.
According to the present invention, only images containing picture data can be extracted, classified, and utilized among a plurality of images.

Description

Methods, devices and computer programs for identifying pictorial images {METHOD, APPARATUS AND COMPUTER PROGRAMS FOR IDENTIFYING ILLUSTRATION IMAGES}

The present invention relates to a method, device, and computer program for identifying a pictorial image in an image. Specifically, the present invention relates to a method, device, and computer program for identifying a pictorial image in an image using a machine learning model.

Classifying specific objects in images is one of the most important techniques in the field of computer vision. Typically, Convolution Neural Networks (CNN) is one of the most representative algorithms in the field of deep learning and is used to extract features from images, and Image Segmentation technology identifies which object a group of pixels within an image is. It can also be used to find out if it is applicable.

As such, various methods have been proposed to identify objects included in images, and different approaches to increase object identification have been developed depending on the type of object.

For example, Korea Patent Publication No. 2017-0047500 detects a document area in an image for electronic documentation of a printed document. To this end, one or more edges are detected in the image and a vertex is determined based on the intersection of grouped lines. The document area is detected in this way.

Previously, technology for detecting document areas that are distinct from background images has been applied to various applications such as document recognition and business card recognition, but technology for identifying images containing pictures rather than document images containing text has been relatively developed. This was insufficient.

Considering that recent attempts have been made to use picture data, such as analyzing psychology based on works of art, such as Korean Patent No. 10-2415106, before extracting objects from the image, the image must be a picture image. There is a need to select picture data from multiple image databases by automatically determining whether it is recognized or not. However, as mentioned above, technology for identifying picture images containing picture data is not widely available and development is needed.

The problem to be solved by the present invention is to provide a method, device, and computer program that can identify a picture image containing picture data.

The purpose of the present invention is to classify picture images containing picture data from multiple images in order to utilize picture data.

The problem to be solved by the present invention is not limited to the above-described problem, and problems not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings. .

A drawing image identification method according to an embodiment of the present invention includes the steps of extracting an analysis target area from an image, and if it is determined that the analysis target area includes an object with the texture of an art tool using a texture classification model, the image It includes the step of identifying a picture image, wherein the texture classification model is a machine learning model learned with a set of a plurality of picture images including at least one of a point, line, or surface drawn with the texture of an art tool.

A drawing image identification device according to an embodiment of the present invention uses a region extraction and texture classification model to extract an analysis target area from an image, and when it is determined that the analysis target area contains the texture of an art tool, the image is drawn as a drawing. It includes a determination unit that identifies images, and the texture classification model is a machine learning model learned with a set of a plurality of picture images including at least one of a point, line, or surface drawn with the texture of an art tool.

The means for solving the problem of the present invention are not limited to the above-mentioned solution means, and the solution methods not mentioned will be clearly understood by those skilled in the art from this specification and the attached drawings. You will be able to.

According to the method, device, and computer program according to an embodiment of the present invention, only images containing picture data can be extracted, classified, and utilized among a plurality of images.

Additionally, according to the present invention, the entire process of utilizing picture data can be automated. In other words, according to the present invention, it is possible to automatically select images that require utilization and analysis of picture data among images previously stored in a database, thereby omitting the step of the user selecting and entering images to be analyzed within the application. This can be done, and as a result, it can provide users with an excellent user experience.

The effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by those skilled in the art from this specification and the attached drawings.

1 is a block diagram of a picture image identification device according to an embodiment of the present invention.
Figure 2 is a block diagram of a picture image identification device according to another embodiment of the present invention.

Figure 4 is a schematic diagram showing an aspect of detecting a paper area using a machine learning model according to an embodiment of the present invention.
Figure 3 is a schematic diagram showing an aspect of acquiring texture information included in an analysis target area using a texture classification model according to an embodiment of the present invention.
Figure 5 is a flowchart showing a method for identifying a picture image according to an embodiment of the present invention.
Figure 6 is a flowchart for explaining a method for extracting a subject area according to an embodiment of the present invention.
Figure 7 is a flowchart explaining a paper area extraction method according to another embodiment of the present invention.
Figure 8 is a flowchart for explaining a method for identifying a picture image according to an embodiment of the present invention.
Figure 9 is a flow chart to explain a method for identifying a picture image using RGB depth values according to an embodiment of the present invention.

The above-described objects, features and advantages of the present invention will become more apparent through the following detailed description in conjunction with the accompanying drawings. However, since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail below.

Like reference numerals throughout the specification in principle refer to the same elements. In addition, components with the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, numbers (eg, first, second, etc.) used in the description of this specification are merely identifiers to distinguish one component from another component.

In addition, the suffix “part” or “module” for the components used in the following embodiments is given or used interchangeably only considering the ease of writing the specification, and does not have a distinct meaning or role in itself.

In the following examples, singular terms include plural terms unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean the presence of features or components described in the specification, and do not preclude the possibility of adding one or more other features or components.

In the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present invention is not necessarily limited to what is shown.

In cases where an embodiment can be implemented differently, the order of specific processes may be performed differently from the order described. For example, two processes described in succession may be performed substantially simultaneously, or may proceed in an order opposite to that in which they are described.

In the following embodiments, when components are connected, this includes not only the case where the components are directly connected, but also the case where the components are indirectly connected by intervening between the components.

For example, in this specification, when components, etc. are said to be electrically connected, this includes not only cases where the components are directly electrically connected, but also cases where components, etc. are interposed and indirectly electrically connected.

Meanwhile, in this specification, “picture image” means an image containing picture data. “Picture data” may include two-dimensional or three-dimensional data consisting of points, lines, or surfaces created using art tools. Additionally, picture data may include picture data generated using an application of an electronic device.

Typically, “painting” refers to a painting in which a shape is expressed on a flat surface using colored tools, but in this specification, “painting” refers to sculptures, crafts, etc. created using various materials such as colored paper, wood, and plastic. It can be understood as encompassing all works of art such as architecture, design, prints, and drawings. In other words, “picture data” in this specification can be interpreted to mean visual data corresponding to a work of art, and may be displayed through a two-dimensional or three-dimensional image.

Hereinafter, the picture image identification device, method, and computer program of the present invention will be described with reference to FIGS. 1 to 7.

Figure 1 is a block diagram of a picture image identification device 100 according to an embodiment of the present application.

The picture image identification device 100 according to an embodiment of the present invention may be a processor of an electronic device including a pre-processing unit 110 (can be omitted), an area extraction unit 130, and a determination unit 150. The picture image identification device 100 may be a processor operating in a server or a processor operating in a terminal. If the picture image identification device 100 is a processor that operates on a server, the server can acquire an image from the terminal through a communication module and transmit it to the picture image identification device 100 to identify whether the image is a picture image. . If the picture image identification device 100 is a processor operating in a terminal, the terminal may transmit an image stored in the terminal device or an image acquired using a camera module included in the terminal device to the picture image identification device 100, The picture image identification device 100 can identify whether an image is a picture image.

The pre-processing unit 110 may be optionally included when the picture image identification device 100 identifies a picture image.

The preprocessor 110 may apply the image to a text extraction model to extract one or more character areas containing text from the image. The pre-processing unit 110 can remove areas recognized as letters from the image and can additionally remove noise.

In another embodiment, the preprocessor 110 may generate an image from which the text area and noise have been removed as a preprocessed image and transmit it to the area extractor 130. When the pre-processing unit 110 performs a task, the area extracting unit 130 may extract the analysis target area from the pre-processing image rather than the image.

The region extraction unit 130 may extract an analysis target region from the image. Here, the image may be a photographic image or an image generated by an electronic device. The image may be an image stored in a database, or may be acquired in real time from an imaging device (or module) such as a camera.

The analysis target area is the area of interest to be analyzed, and may be a subject area or an area judged on paper.

The area extraction unit 130 may extract the analysis target area including the subject area or the paper area and transmit it to the determination unit 150.

Below, an example where the region of interest to be analyzed by the region extraction unit 130 is a subject region and an embodiment where the region is a paper region will be described in more detail.

When the analysis target area is a subject area, the area extractor 130 removes the background from the image and detects the subject area. According to one embodiment, the area extractor 130 may remove the background by applying the image to a subject/background area classification model and cropping the subject area. Here, the subject/background area classification model may be a machine learning model learned using a large data set in which area characteristics are labeled on a pixel basis. For example, a subject/background area classification model may be a deep learning model learned from a large data set in which each pixel is labeled as subject or background. DUTS-TE, DUT-OMRON, HKU-IS, ECSSD, etc. can be used as large data sets, and individually collected image data can be used.

Meanwhile, a unet series deep learning model can be used as a subject/background area classification model, but is not necessarily limited to this.

When a subject area is detected in an image, the area extractor 130 may resize the subject area, set it as an analysis target area, and apply the resized subject area to a texture classification model. In one embodiment, the area extractor 130 may resize the subject area so that the width and height are the same. Resizing the subject area can be performed when the texture classification model receives a preset image size.

When the analysis target area is a paper area, the area extractor 130 may detect the paper area in the image, crop the detected paper area, and set it as the analysis target area. At this time, the paper area refers to an area recognized as paper, such as a sketchbook, drawing paper, A4 paper, etc.

According to one embodiment, when detecting the paper area, the area extractor 130 may detect the paper area using a paper area detection model, an edge detector, and a sketchbook area detection model. An image is included in each machine learning model. It is also possible to apply and crop the paper area according to the detection result of a single model, and it is also possible to cross-verify the paper area detection result obtained by applying the image to multiple models, as shown in FIG. 3. According to this embodiment, the paper area detection accuracy increases, and as a result, the picture image identification accuracy can also increase.

In the above-described embodiment, the paper area detection model may be a machine learning model such as YOLO learned with a plurality of paper images. The paper area detection model is a model learned using a learning data set labeled using a bounding box. When an image is applied, the bounding box coordinates corresponding to the paper area can be output. The edge detector can detect edges using the histogram difference between adjacent pixels. The edge detector detects edges using an image processing library such as canny edge detector. It is a detector that obtains histogram values between pixels and defines them as edges when the difference between adjacent pixels and histogram values is large. Additionally, the sketchbook area detection model may be a model learned using training data labeling the sketchbook area in pixel units. Another example of detecting a paper area using an edge detector is as follows. The area extractor 130 can detect the edges of all objects included in the image using an edge detector. The area extractor 130 can analyze the shape of the edges and extract all edges (outlines) that have a rectangular shape. And, among the edges having a square shape, an area corresponding to the largest square can be set as the paper area. Here, an area set as a paper area can be identified as a drawing area.

When the area extraction unit 130 detects and crops the paper area in the image, it transmits this to the determination unit 150 so that the determination unit 150 can determine whether picture data is included in the paper area.

The determination unit 150 may determine whether an object having the texture of an art tool is included in the analysis target area using a texture classification model. If, as a result of the determination, it is determined that the analysis target area includes an object having the texture of an art tool, the determination unit 150 may classify the image as a drawing image. At this time, the art tool may be at least one of crayons, paint, colored paper, pencil, ballpoint pen, marker pen, pastel, paint, clay, stamp, or marker, and the paint may be of any color, such as acrylic paint, watercolor paint, or oil paint. It can contain all substances that contain .

The determination unit 150 uses a texture classification model to determine whether an object (picture data) with a texture of an art tool is included. The texture classification model is a machine learning model learned with a set of multiple picture images containing at least one of points, lines, or faces drawn with the texture of an art tool, and may be a CNN-based multi-classification model. A plurality of picture image sets used to train a texture classification model include, in addition to a set of picture images containing objects drawn with the texture of art tools, a region of the object having at least one texture among plastic, skin, flooring, flooring, or wood. A learning image consisting of may be further included. The above-described objects are frequently included in picture images and can be included in learning images to increase the accuracy of picture image classification. That is, the texture classification model may be a model learned using as learning data an image of a region cropped from a photo of an object with a specific texture, etc., and may classify the texture of one or more objects included in the input image. The judgment unit 150 includes both a texture classification model learned from a set of picture images containing only objects with the texture of art tools, objects with the texture of art tools, and background objects (skin, flooring, flooring, wood, etc.) Texture classification models learned on a set of picture images can be used together, or they can be used individually.

If it is confirmed that at least one of the art tools (crayons, paints, colored paper, pencils, ballpoint pens, marker pens, etc.) exists with a high probability as a result of the classification of the texture classification model, the judgment unit 150 determines that the drawing data is in the corresponding analysis target area. It can be judged as included. And the image containing the analysis target area can be judged to be a drawing image, that is, a drawing image containing drawing data. That is, as shown in Figure 4, the texture classification model is a model that obtains texture information included in the analysis target area when the analysis target area is input. The output value is the probability that an object with the texture of each art tool is included. can be provided. The determination unit 150 uses the output texture information to determine whether picture data is included. If it is determined that the analysis target area contains picture data, the analysis target image including the analysis target area is judged to be a picture image. It becomes possible.

Hereinafter, a picture image identification device 200 according to another embodiment of the present invention will be described with reference to FIG. 2. According to an embodiment shown in FIG. 2, the picture image identification device 200 according to an embodiment of the present invention includes a preprocessor (210/omitted), a pixel analysis unit 230, and a determination unit 250. It may be a processor of an electronic device including a processor. The picture image identification device 200 may be a processor operating in a server or a processor operating in a terminal. If the picture image identification device 200 is a processor that operates on a server, the server can acquire an image from the terminal through a communication module and transmit it to the picture image identification device 200 to identify whether the image is a picture image. . If the picture image identification device 200 is a processor operating in a terminal, the terminal may transmit an image stored in the terminal device or an image acquired using a camera module included in the terminal device to the picture image identification device 200, The picture image identification device 200 can identify whether an image is a picture image.

The pre-processing unit 210 can perform the same tasks as the pre-processing unit 110 described in FIG. 1, and has the same omitted characteristics. Therefore, the description of the pre-processing unit 210 replaces the description of the pre-processing unit 110.

The pixel analysis unit 230 may extract characteristic information of pixels constituting the image from the image. At this time, the characteristic information may be either the RGB value of the pixel or the RGB depth value (RGB-DEPTH) of the pixel. The pixel analysis unit 230 may transmit the RGB value of the pixel or the RGB depth value of the pixel constituting the image to the determination unit 250. A method of extracting the RGB depth value of a pixel according to one embodiment is as follows. First, the pixel analysis unit 230 applies the image to a monocular depth estimation model learned using a data set in which depth is expressed in RGB, such as the RGB-d Object Dataset, and sets the depth of the image to RGB. It can be expressed as a value. The RGB-d object dataset is a dataset that can learn how deep an object is located in a 2D image using RGB values, and the monocular depth estimation model is a monocular depth estimation model, and pix2pix or unet-based models can be used. Additionally, the pixel analysis unit 230 can obtain the RGB depth value of each pixel by extracting the depth value of the RGB value for each pixel from the image displayed as the RGB value.

The determination unit 250 may identify the image as a drawing image using pixel characteristic information (pixel RGB value, pixel RGB depth value).

For example, if the characteristic information extracted by the pixel analysis unit 230 is an RGB value, the determination unit 250 may detect a pixel group in which the difference in RGB values of adjacent pixels is greater than or equal to a preset threshold. At this time, the difference between RGB values can be obtained by calculating the Euclidean distance. And, if it is determined that there are more than a preset number of pixel groups with RGB value differences greater than the threshold, the image can be identified as a drawing image.

For example, the RGB values of pixels at positions (2,3), (3,4), (4,5) are [220,255,80], (2,4),(3,5),(4,6) ) Assume that the RGB value of the pixel at the location is [255,255,255], and the threshold value that serves as a criterion for judging the difference in RGB values between adjacent pixels is 150. The determination unit 250 determines adjacent pixels, that is, {(2,3) and (2,4)}, {(3,4) and (3,5)}, and {(4,5) and (4,6). } can be calculated, and in this case, the difference in RGB values between each adjacent pixel is calculated to be about 178.4, so it can be determined that it is a group of pixels that all have a difference greater than the threshold value. A pixel group with an RGB value difference greater than a threshold means an unnatural gradation. If the image contains an object with an unnatural gradation of points, lines, or surfaces, the determination unit 250 converts the image into a drawing image. You can judge.

As another example when the characteristic information extracted by the pixel analysis unit 230 is an RGB value, the determination unit 250 clusters the extracted RGB values, and if the number of clustered clusters is less than a preset number, the image is converted into a picture image. can be identified. By clustering RGB values for each pixel in a manner such as k-means clustering, colors in a similar range can be clustered. In the case of drawing images, creation is generally done within the colors provided by art tools, so the number of color groups included in the drawing image is likely to be smaller than the number of color groups included in the photographic image. Using this, a standard number for identifying an image as a drawing image can be set, and if it has fewer color clusters than the standard number, the determination unit 250 can determine that the image is a drawing image. For example, when the standard number is 8, the judgment unit 250 classifies the image as a picture image if the number of color clusters extracted and clustered from the image is 5, and if the number of color clusters is 10, it is classified as a photo image. It is a classification method.

The standard number may be set differently depending on user characteristics. If the picture image identification device 100 analyzes user A's picture images for a certain period of time and determines that user A's picture uses about 6 color groups on average, the determination unit 250 sets the standard number to 7~ It can be set to 8. If the picture image uploaded by user B uses about 15 color groups on average, the picture image identification device 100 sets the standard number to 15 when identifying the picture image for the images in user B's database. You can set it to 16.

When the characteristic information extracted by the pixel analysis unit 230 is an RGB depth value, the determination unit 250 may generate a cropped image by cropping an area in the image where the RGB depth value falls within a preset range. For example, if the depth values are within a similar range, this means that the pixels are located on the same line, and there is a high possibility that the pixels represent a flat object. Accordingly, the determination unit 250 can detect the paper area by cropping an area that is likely to be flat paper or colored paper. The paper area (cropped image) detected in this way can be resized to have the same width and height and then applied to the texture classification model. In other words, in this embodiment, the cropped image can be said to become the analysis target area (region of interest) shown in FIG. 3, and in the subsequent process, the determination unit 150 of FIG. 1 identifies the picture image using the texture classification model. It is similar to doing The characteristics of the texture classification model have been described in Figure 1, so they are omitted.

Hereinafter, a method for identifying a picture image according to an embodiment of the present invention will be described with reference to FIGS. 5 to 7.

Figure 5 is a flowchart showing a method for identifying a picture image according to an embodiment of the present invention. Referring to FIG. 5, the drawing image identification method according to an embodiment of the present invention includes an image acquisition step (S50), an analysis target area extraction step (S100), and a determination of whether the analysis target area includes an object with the texture of an art tool. It may include a step (S200) and a step of identifying the image as a picture image (S300).

In the image acquisition and preprocessing step (S50), the electronic device can retrieve an image stored in a database or acquire an image in real time from an imaging device (or module) such as a camera. If the electronic device is a server, the server can obtain the image by receiving the image from the terminal using a network. Meanwhile, in step 50, the electronic device may selectively preprocess the image. In other words, step 50 can be understood as being omitted. When preprocessing the image in step 50, the electronic device may apply the image to a text extraction model to extract one or more character areas containing text from the image. In step 50, the electronic device may remove areas recognized as letters from the image and may additionally remove noise. If step 50 is included, the electronic device may extract the analysis target area in step 100 from the image from which the text area and noise have been removed, rather than the original image.

In the step of extracting the analysis target area from the image (S100), the electronic device may extract the analysis target area from a photographic image or an image generated by the electronic device.

The analysis target area is the area of interest to be analyzed, and may be a subject area or an area judged on paper. The operation of extracting the analysis target area is to increase the accuracy and speed of picture image identification. If the acquired image is a photographic image, the pictorial data is likely to be contained within the paper area. Therefore, according to an embodiment of the present invention, the texture of an object included in the paper area can be identified more quickly and accurately by removing the background area that may be noise.

Hereinafter, with reference to FIGS. 6 and 7 , a method in which an electronic device extracts a subject area or a paper area as an analysis target area will be described in detail.

Figure 6 is a flowchart for explaining a method for extracting a subject area according to an embodiment of the present invention. Referring to FIG. 6, in the analysis target area extraction step (S100), the electronic device may remove the background from the image and detect the subject area (S110). Next, the electronic device can resize the subject area and set it as the analysis target area (S130).

In step 110, the electronic device may apply the image to a subject/background area classification model to remove the background by cropping the subject area. Here, the subject/background area classification model may be a machine learning model learned using a large data set in which area characteristics are labeled on a pixel basis. For example, a subject/background area classification model may be a deep learning model learned from a large data set in which each pixel is labeled as subject or background. DUTS-TE, DUT-OMRON, HKU-IS, ECSSD, etc. can be used as large data sets, and individually collected image data can be used.

Meanwhile, a unet series deep learning model can be used as a subject/background area classification model, but is not necessarily limited to this.

In step 130, the electronic device can resize the subject area so that the width and height are the same, and the resized subject area can be set as an analysis target area and applied to the texture classification model.

Figure 7 is a flowchart explaining a method for extracting an analysis target region according to another embodiment of the present invention. Referring to FIG. 7, in step 100, the electronic device may detect a paper area in the image (S130), crop the detected paper area, and set it as an analysis target area (S150). At this time, the paper area refers to an area recognized as paper, such as a sketchbook, drawing paper, A4 paper, etc.

According to one embodiment, the electronic device may detect the paper area using a paper area detection model, an edge detector, and a sketchbook area detection model in step 130. An image is applied to each machine learning model and the detection result of a single model is obtained. It is also possible to crop the paper area according to , and as shown in FIG. 3, the paper area detection result obtained by applying the image to a plurality of models can be cross-verified. Using this cross-validation method increases the accuracy of paper area detection, which can ultimately increase the accuracy of picture image identification.

The paper area detection model may be a machine learning model such as YOLO learned with multiple paper images. The paper area detection model is a model learned using a learning data set labeled using a bounding box. When an image is applied, the bounding box coordinates corresponding to the paper area can be output. The edge detector can detect edges using the histogram difference between adjacent pixels. The edge detector detects edges using an image processing library such as canny edge detector. It is a detector that obtains histogram values between pixels and defines them as edges when the difference between adjacent pixels and histogram values is large. Additionally, the sketchbook area detection model may be a model learned using training data labeling the sketchbook area in pixel units.

Another example of detecting a paper area using an edge detector is as follows. In step 130, the electronic device may detect the edges of all objects included in the image using an edge detector. Next, the electronic device can analyze the shape of the edges and extract all edges (outlines) that have a square shape. And, among the edges having a square shape, an area corresponding to the largest square can be set as the paper area. Here, an area set as a paper area can be identified as a picture, and the electronic device can also designate the area as a picture area.

Referring again to FIG. 5 , in step 200, the electronic device may determine whether an object having the texture of an art tool is included in the analysis target area. If, as a result of the determination in step 200, it is determined that the analysis target area includes an object having the texture of an art tool, the electronic device may classify and identify the image as a drawing image (S300). At this time, the art tool may be at least one of crayons, paint, colored paper, pencil, ballpoint pen, marker pen, pastel, paint, clay, stamp, or marker, and the paint may be of any color, such as acrylic paint, watercolor paint, or oil paint. It can contain all substances that contain . In step 200, a texture classification model is used to determine whether an object (picture data) having a texture of an art tool is included. The texture classification model is a machine learning model learned with a set of multiple picture images containing at least one of points, lines, or faces drawn with the texture of an art tool, and may be a CNN-based multi-classification model. A plurality of picture image sets used to train a texture classification model include, in addition to a set of picture images containing objects drawn with the texture of art tools, a region of the object having at least one texture among plastic, skin, flooring, flooring, or wood. A learning image consisting of may further be included. The above-described objects are frequently included in picture images and can be used to increase the accuracy of picture image classification.

In step 200, the electronic device includes both a texture classification model learned from a set of picture images containing only objects with the texture of art tools, objects with the texture of art tools, and background objects (skin, flooring, flooring, wood, etc.) Texture classification models learned with a set of picture images can be used together, or they can be used individually.

If at least one of the art tools (crayons, paints, colored paper, pencils, ballpoint pens, marker pens, etc.) is confirmed to exist with a high probability as a result of the classification of the texture classification model in step 200, the electronic device generates picture data in the analysis target area. It can be judged as included. And the image containing the analysis target area can be judged to be a drawing image, that is, a drawing image containing drawing data.

In step 300, the electronic device may identify or classify the image as a drawing image based on the result of step 200 that the analysis target area includes an object with the texture of an art tool.

Below, with reference to FIG. 8, a method for identifying a picture image using pixel characteristic information will be described. Figure 8 is a flowchart for explaining a method for identifying a picture image according to an embodiment of the present invention. Referring to FIG. 8, the picture image identification method according to an embodiment of the present invention may include an image acquisition step (S5), a pixel characteristic information detection step (S10), and a picture image identification step (S30).

Since the image acquisition step (S5) has the same characteristics as the image acquisition step (S50) described with reference to FIG. 5, refer to the content of step 50.

In the step of detecting pixel characteristic information (S10), the electronic device may extract characteristic information of pixels constituting the image from the image. At this time, the characteristic information may be either the RGB value of the pixel or the RGB depth value (RGB-DEPTH) of the pixel. In step 10, the electronic device may display the depth of the image as an RGB value by applying the image to a monocular depth estimation model learned using a data set in which depth is expressed in RGB, such as the RGB-d object dataset. At this time, the monocular depth estimation model is a monocular depth estimation model, and a pix2pix or unet-based model may be used. And the electronic device can obtain the RGB depth value of each pixel by extracting the depth value of the RGB value for each pixel from the image displayed as RGB value.

In the step (S30) of identifying a picture image using pixel characteristic information, the electronic device may perform different operations depending on the pixel characteristic information. For example, if the extracted characteristic information is an RGB value, in step 30, the electronic device detects a pixel group in which the difference in RGB values of adjacent pixels is more than a preset threshold, and the pixel group with the difference in RGB values more than the threshold is preset. If it is determined that there are more than one number, the image can be identified as a picture image. At this time, the difference between RGB values can be obtained by calculating the Euclidean distance.

As another example of step 30, if the extracted characteristic information is an RGB value, the electronic device clusters the extracted RGB values, and if the number of clustered clusters is less than a preset number (standard number), the electronic device may identify the image as a picture image. . Clustering algorithms such as k-means and hierarchical may be used, but are not limited to these. When setting a reference number that serves as an identification standard for a picture image, the electronic device can use the characteristics of picture images pre-stored for each user. In other words, color cluster statistics within the picture image can be calculated for each user, and the calculation results can be used to set the standard number.

Figure 9 is a flow chart to explain a method for identifying a picture image using RGB depth values according to an embodiment of the present invention.

According to an embodiment of using RGB depth values as pixel characteristic information, the electronic device generates a cropped image by cropping an area where the RGB depth value falls within a preset range (S31), and applies the cropped image to the texture classification model. Thus, the picture image can be identified (S33). In step 31, the electronic device can set a region so that pixels with depth values within a similar range can be included in one region. Since pixels with depth values in the same range mean pixels representing flat objects, the electronic device can recognize the cropped image as a paper area, resize it, and apply it to the texture classification model. In one embodiment, the cropped image may be resized so that its width and height have the same length. In step 33, the electronic device obtains texture information by applying the resized cropped image to a texture classification model, and can identify the picture image based on the texture information. The characteristics of the texture classification model are the same as those described in FIGS. 1 and 4.

The preprocessing units 110 and 210, the area extraction unit 130, the pixel analysis unit 230 and the determination units 150 and 250 that constitute the picture image identification device 100 and 200 according to an embodiment of the present invention. They can be selected and combined as needed. For example, after extracting pixel characteristic information from the pixel analysis unit 230, the paper area can be extracted from the area extraction unit 130 using this, and the determination unit 250 sets the paper area as the area of interest and texture By applying it to the classification model, texture information contained in the extracted paper area can be obtained. Therefore, the present invention is not limited to the embodiments described herein, and the goal of identifying picture images can be achieved through selective combination of each module.

The picture image identification method described above with reference to FIGS. 4 to 9 may be implemented through a computer program stored in a medium to execute any one of the methods using a computer.

According to the present invention, the entire process of utilizing picture data can be automated. In other words, according to the present invention, it is possible to automatically select images that require utilization and analysis of picture data among images previously stored in a database, thereby omitting the step of the user selecting and entering images to be analyzed within the application. This can be done, and as a result, it can provide users with an excellent user experience.

The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified and implemented in other embodiments by a person with ordinary knowledge in the field to which the embodiments belong. Therefore, contents related to such combinations and modifications should be construed as being included in the scope of the present invention.

In addition, although the above description focuses on the embodiment, this is only an example and does not limit the present invention, and those skilled in the art will be able to understand the above without departing from the essential characteristics of the present embodiment. You will see that various modifications and applications not illustrated are possible. In other words, each component specifically shown in the embodiment can be modified and implemented. And these variations and differences in application should be construed as being included in the scope of the present invention as defined in the appended claims.

100, 200: Picture image identification device
110, 210: Preprocessing unit
130: Area extraction unit
230: Pixel analysis unit
150, 250: Judgment unit

Claims (19)

A method for an electronic device to identify a pictorial image containing pictorial data in one or more images, comprising:
Extracting an analysis target area from an image;
If it is determined that the analysis target area includes an object with the texture of an art tool and a background object using a texture classification model, identifying the image as a drawing image;
The texture classification model is a machine learning model learned from a learning image consisting of a set of a plurality of picture images including at least one of a point, line, or surface drawn with the texture of an art tool and a region of a background object. A method for identifying picture images. According to paragraph 1,
The art tool is a drawing image identification method including at least one of crayons, paint, colored paper, pencil, ballpoint pen, marker pen, pastel, clay, stamp, or marker. According to paragraph 1,
A method for identifying a picture image, wherein the background object is an object having at least one texture selected from plastic, skin, flooring, flooring, or wood. According to paragraph 1,
The analysis target area extraction step is,
removing the background from the image and detecting the subject area;
A picture image identification method comprising the step of resizing the subject area and setting it as an analysis target area.

According to paragraph 4,
The subject area detection step is,
A step of cropping the subject area by applying the image to a subject/background area classification model,
The subject/background area classification model is a picture image identification method that is a machine learning model learned using a large data set in which area characteristics are labeled in pixel units.
The method of claim 5, wherein the machine learning model is a unet series model.
According to paragraph 1,
The analysis target area extraction step is,
detecting a paper area in the image;
A pictorial image identification method including the step of cropping the detected paper area and setting it as the analysis target area.
In clause 7,
The paper area detection step is,
One of the following: a paper area detection model learned from a set of multiple paper images, an edge detector that detects edges using histogram differences between adjacent pixels, or a sketchbook area detection model learned using training data labeling sketchbook areas in pixel units. A pictorial image identification method comprising the step of cross-validating the paper area detection result by applying the image to the above.
According to clause 1,
The electronic device is a server,
A method of identifying a picture image, wherein the one or more images are received from a terminal.
According to paragraph 1,
Extracting one or more character areas containing text by applying the image to a text extraction model;
Further comprising generating a preprocessed image by removing the text area from the image,
The analysis target area extraction step is
Comprising the step of extracting the analysis target area from the preprocessed image,
A picture image identification method in which the text extraction model is a model learned with a plurality of text data.

A method for an electronic device to identify a pictorial image containing pictorial data in one or more images, comprising:
Extracting RGB depth values (RGB-DEPTH) of pixels constituting the image;
If it is determined that the RGB depth value (RGB-DEPTH) contains the texture of an art tool by applying a texture classification model to an area within a similar range, identifying the image as a drawing image;
The texture classification model is a picture image identification method that is a machine learning model learned from a set of a plurality of picture images containing at least one of a point, line, or surface drawn with the texture of an art tool. According to clause 11,
The picture image identification step is,
generating a crop image by cropping an area in the image where the RGB depth values fall within a similar range;
resizing the cropped image;
A drawing image identification method comprising the step of applying the resized cropped image to a texture classification model and, if it is determined to contain the texture of an art tool, identifying the image as a drawing image. According to clause 11,
The step of extracting the RGB depth value of the pixel is:
Displaying the depth of the image as an RGB value using a monocular depth estimation model learned with the RGB-d object dataset;
A picture image identification method comprising extracting the depth value of the RGB value for each pixel from the image displayed with the RGB value. According to clause 13,
The monocular depth estimation model is a monocular depth estimation model, a picture image identification method characterized in that it is a pix2pix or unet based model. A device for identifying a picture image containing picture data in one or more images, comprising:
an area extraction unit that extracts an analysis target area from the image;
If it is determined that the analysis target area includes an object with the texture of an art tool and a background object using a texture classification model, it includes a determination unit that identifies the image as a drawing image,
The texture classification model is a machine learning model learned from a learning image consisting of a set of a plurality of picture images including at least one of a point, line, or surface drawn with the texture of an art tool and a region of a background object. A picture image identification device. A device for identifying a picture image containing picture data in one or more images, comprising:
A pixel analysis unit that extracts the RGB depth value (RGB-DEPTH) constituting the image;
If it is determined that the RGB depth value (RGB-DEPTH) contains a texture of an art tool by applying an area within a similar range to a texture classification model, it includes a determination unit that identifies the image as a drawing image,
The texture classification model is a picture image identification device that is a machine learning model learned from a set of a plurality of picture images containing at least one of a point, line, or surface drawn with the texture of an art tool. A computer program stored in a medium to execute any one of the methods of claims 1 to 14 using a computer.
delete delete