JP6900016B2 - Image area extraction processing method and image area extraction processing program - Google Patents

Description

The present invention relates to an image area extraction processing method and an image area extraction processing program.

Conventionally, when coloring a digital image with a computer, the user selects an arbitrary color from the color palette and colors it directly, or creates a closed area with a line drawing and fills the entire closed area. It was taken.

However, in a closed area with trace lines that are premised on coloring as in animation production, there are many tone patterns (screen tones) and detailed writing depictions, and there are many small areas, so it is necessary to manually cut out and color. There was a problem that it took time and effort.

Further, in recent image production, the writing taste such as so-called pen touch is regarded as important, and many line breaks can be seen in the middle. Therefore, the coloring worker visually confirms the person, the background, etc., grasps the contour line, performs an input operation for specifying the contour, such as appropriately finely cutting out the area of the digital image, and colors the contour. Is going. Therefore, there is a problem that the coloring work on the digital image processing takes time and the coloring result varies from person to person.

Therefore, it is a coloring processing device that obtains a colored line drawing by coloring a region defined by a drawing line in an uncolored line drawing, and is a feature amount extraction means for extracting the feature amount of the line drawing by image analysis of the uncolored line drawing. A plurality of types of the plurality of closed regions are provided according to a region separating means for separating the drawing surface of the line art into a plurality of closed regions based on the feature quantity and a predetermined rule defined in relation to the feature quantity. An object recognition means for classifying and recognizing an object, a line break correction means for obtaining a closed curve by correcting a line break of a drawing line surrounding a closed area recognized as an object, and a partial area surrounded by the closed curve. The cutout area corresponding to the closed area recognized as the object based on the closed area cutting means for obtaining the cutout area by cutting out from the drawing surface and the coloring information predetermined for each object. A coloring processing apparatus or the like characterized by comprising a coloring means for coloring is proposed (for example, Patent Document 1).

Further, as an example of the technology for performing automatic coloring using a neural network, CLIP STUDIO PAINT (Non-Patent Document 1), which is the technology of the applicant, and the website of Paints Chainer (Non-Patent Document 2) operated by a person other than the applicant. The site can be mentioned. This technique is a coloring processing technique for obtaining a colored image by coloring a region defined by a drawing line in an uncolored line drawing.

However, the above technique (Patent Document 1) utilizes an intermediate gradation region obtained by reducing or averaging the tone pattern region. Therefore, the coloring effect can hardly be obtained with the image of only the line drawing in which the tone pattern is not used.

In addition, CLIP STUDIO PAINT and PaintsChainer only offer a function to automatically color uncolored areas for automatic coloring of line drawings.

JP-A-2009-134410

CLIP STUDIO PAINT [Access date: June 25, 2019], Internet <https://www.clip-studio.com/clip_site/> PaintsChainer [Access date: June 26, 2019], Internet <https://paintschainer.preferred.tech/index_ja.html>

The disclosed technique aims to make it easier to use the extracted region by more appropriately extracting the region from the line drawing including the unclosed line.

The disclosed technology uses an image conversion model that outputs a colored image from an input line drawing, which is machine-learned using a line drawing and a colored line drawing as teacher data, to convert the target line drawing into a colored line drawing. Image area extraction having a step of converting and a step of extracting an image area colored in a predetermined color (one or more colors, one color range or a plurality of color ranges) from the colored line drawing. Provides a processing method.
The disclosed technology also provides a program that causes a computer to execute the above method and a medium on which the program is recorded.

According to the disclosed technique, by more appropriately extracting a region from a line drawing including an unclosed line, the extracted region can be easily used.

FIG. 1A is a diagram showing an example of an unclosed nail region. FIG. 1B is a diagram showing an example in which an unclosed nail region is extracted using one embodiment. FIG. 2 shows a block diagram of one embodiment. FIG. 3 shows a hardware configuration in which a program or method of one embodiment is executed. FIG. 4A shows an example in which coloring is performed using AI. FIG. 4B is a diagram showing an example in which a colored region is selected. FIG. 5 (A) is a diagram showing an example in which coloring is performed using an AI different from that of FIG. 4 (A). FIG. 5B is a diagram showing an example in which the colored region is selected in FIG. 5A. FIG. 6 is a diagram showing an example after coloring is performed using the same AI as in FIG. 5A and then edge-retaining smoothing treatment is performed. FIG. 6B is a diagram showing an example in which the colored region is selected in FIG. 6A. FIG. 7 (A) is a diagram showing an example in which a region is selected by designating a threshold value different from that of FIG. 6 (B) in the image of FIG. 6 (A). FIG. 7B is a diagram showing an example in which a region is selected by designating a threshold value different from that in FIG. 7A. FIG. 8 (A) is a diagram showing an example in which the nail portion is closed by adding a line drawing to the tip portion of the nail portion using the region selected in FIG. 7 (B). FIG. 8B is a diagram showing an example in which the nail portion of FIG. 7A is colored with a different color. FIG. 9A is an example in which the coloring of the nail portion of the closed line drawing is deleted. FIG. 9B is a diagram showing an example in which another image is attached to the claw portion of the closed line drawing. FIG. 10 (A) is a diagram showing an example of extracting an image of a nail including an unclosed line drawing obtained by FIG. 6 (A). FIG. 10B is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 8B. FIG. 10 (C) is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 9 (A). FIG. 10 (D) is a diagram showing an example in which a line drawing is obtained by removing a colored portion from the image obtained in FIG. 9 (A). FIG. 11 is a diagram showing an example of learning the coloring function by AI. FIG. 12 is a flowchart showing an example of image processing of one embodiment. FIG. 13 is a diagram showing an example in which a region of a predetermined color is extracted by correcting the coloring as necessary after coloring by AI. In FIG. 13, since it is not necessary in this example, an example of extracting a predetermined color region without modifying the coloring is shown. FIG. 14 is a diagram showing an example in which a predetermined color region is extracted by correcting the coloring as necessary. As an example of color correction, an example of extracting a predetermined color region after performing edge-preserving smoothing processing is shown. FIG. 15 is a diagram showing an example of processing an image of the extracted region.

Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. The following embodiments are not mutually exclusive. As long as there is no contradiction, some of the embodiments may be replaced or combined with some of the other embodiments. In addition, each step of the disclosed flow can be replaced as long as there is no contradiction. Also, each flow can be executed at the same time. Some of the embodiments can be implemented either in programs or in hardware. In addition, a part of the program can utilize the functions of the operating system. The program can be recorded on a portable recording medium.

The drawings use monochrome images. Therefore, it should be noted that the shades of color are expressed in gray scale for convenience. In an actual embodiment, an image composed of color pixels can be used. When a color is specified or a threshold value for a color is specified, color saturation, lightness, hue, or a combination thereof can be used as the threshold value.

FIG. 1A is a diagram showing an example of an unclosed nail region. FIG. 1B is a diagram showing an example of extracting a region of a nail that is not closed.
As shown in FIG. 1A, a line drawing of a nail 1018 and a finger 800 is drawn on the canvas 100. The finger 800 forms a closed region with the line drawing of the finger 801 and the line drawing of the finger 802 and the frame line of the canvas 100.

On the other hand, the line drawing 1010 of the claw 1018 is open at the end 1012 of the line drawing and the end 1014 of the line drawing. Therefore, the claw 1018 does not form a closed region and is connected to the background 1019. For this reason, it is not easy to select or extract a nail 1018 having an open line drawing that is not a closed region. FIG. 1B shows a state in which the line drawing 1010a of the nail is extracted. The line drawing 1010a of the finger is open at the end 1012a of the line drawing and the end 1014a of the line drawing. Therefore, the extraction of the nail line drawing 1010a composed of the open curve cannot be extracted by using a general algorithm for selecting the closed curve.
In the embodiment described below, even the line drawing 1010a of the nail composed of such an open curve can be easily extracted and processed.

FIG. 2 shows a block diagram of the image area extraction processing device 200 according to the embodiment. The image area extraction processing device 200 includes a coloring unit 202, an edge-holding type smoothing processing unit 204, an area extraction unit 206, an image processing unit 208, and a display unit 210.

The original line drawing is input to the coloring unit 202. In the coloring section, AI is used to color the original line drawing. The method of learning the AI of the coloring part will be described later.

The image colored by the coloring unit 202 is given to the region extraction unit 206 by SW1. Alternatively, the image colored by the coloring unit 202 by SW1 is given to the region extraction unit 206 after being corrected by the edge holding type smoothing processing unit 204.
The edge-holding type smoothing processing unit 204 has a function of smoothing the image of the colored part while holding the line drawing part having an edge in the colored line drawing.

Edge-preserving smoothing is a spatial filtering processing technique. In a general smoothing process such as "blurring", a "blurring" process is performed to smooth the image as a whole including the line drawing portion having an edge. On the other hand, edge-preserving smoothing is a process of smoothing only a small luminance gradient while maintaining a large luminance gradient (edge) in an image. In the present embodiment, the shape of the line drawing portion having a large luminance gradient is maintained, and the colored portion including the shade of color composed of the small luminance gradient is smoothed.
Such an edge-retaining smoothing technique can be realized by, for example, the following techniques known to those skilled in the art.

For example, a "bilateral filter" is a filter capable of performing the most basic edge-preserving smoothing process. The basic processing of this filter is a filter in which the closer the color is, the greater the weight is, depending on both the color difference between the pixels and the position difference. Instead of determining the weight only by the distance between pixels like "blurring" using the Gaussian function, the difference in the brightness of the pixels is also seen, and where there is a large change, the weight is reduced to leave an edge. Filtering is done.
Specific examples of processing on the image to which the edge-holding type smoothing process has been performed will be described with reference to FIGS. 5 and 6.

The area extraction unit 206 can receive, for example, a color designation from the operator and a designation of the location of the image. In this case, the region extraction unit 206 identifies a region of a designated color around the designated image region. To specify a color, a color can be specified by a combination of hue, lightness, or saturation, which are three elements of color. Color designations can include grayscale designations of shades of gray.

Further, the threshold value can be set to a range having upper and lower threshold values having a certain width. In addition, if there is an instruction to expand the specified area based on the operator's instruction, the value of either or both of the upper and lower threshold values can be changed so that the width of the range defined by the upper and lower threshold values can be changed. You may. An example of changing the area of the specified image by changing the threshold value will be described with reference to FIG. 7.

The image processing unit 208 can perform various processing on the extracted image area. For example, it is possible to color the extracted region, remove the color, remove the region other than the extracted image, replace the extracted region with another image, and the like.

In FIG. 2, SW1 allows the output image from the coloring unit 202 to be directly given to the area extraction unit 206 (the contact 221 and the contact 220 of SW1 are connected), or the edge holding type smoothing processing unit. Two routes can be selected, either given to the region extraction unit 206 via 204 (contact 221 and contact 222 of SW1 are connected).

FIG. 3 shows a hardware configuration 300 in which the program or method of one embodiment is executed. In the embodiment, the line drawing input by the client terminal may be sent to the server device via the network and executed by the server device. Alternatively, the client terminal may perform the processing of the embodiment. Therefore, the hardware configuration 300 shown in FIG. 3 applies to the configuration of either the client terminal or the server device.

The hardware configuration 300 includes a ROM 303, a RAM 305, an input unit 307, a communication unit 309, a display unit 311 and an external storage control unit 313. The external storage control unit 313 can read and write data or a program to the memory 315.
The program that implements the embodiment may be stored in memory 315, ROM 303 or RAM 305.

FIG. 4A shows an example in which coloring is performed using AI. FIG. 4B is a diagram showing an example in which a colored region is selected.
In FIG. 4A, the finger 802 and the nail 1041 are colored. As will be described later, this coloring is generated by coloring the input line drawing by the coloring unit 202 using AI learned using, for example, GAN (Generative Adversarial Network), which is one of the deep learning algorithms. This is an example of a colored image. As already described, in FIG. 4, it is represented in gray scale, but the actual colored image is an image having color. In the colored image of FIG. 4A, the peripheral positions of the line drawing end 1042 and the line drawing end 1044 at the tip of the nail line drawing 1010 are also colored to resemble the shape of the nail. Therefore, the nail line drawing 1010 is an example in which the nail line drawing is appropriately colored by AI even though it is not a closed line drawing. This example is an ideal example. In this case, if the operator specifies a part of the area of the colored nail 1041 and the colored color, the color is substantially the same as the specified color (or one of the colored nails 1041). By specifying the position of the part by the operator, the area of the colored color around the part of the position) is specified. Here, substantially the same color means a color corresponding to a designated color and a color defined by a plurality of predetermined threshold values. As already mentioned, the threshold value is a threshold value corresponding to color saturation, lightness or hue, or a combination thereof (including gray scale). As the threshold value, a predetermined color range can be specified by setting a plurality of threshold values.

FIG. 4B is a diagram showing an example in which a colored region is selected (specified). The area of the selected (specified) claw 1041a is shown by hatching. In the line drawing 1010a of the nail composed of the open line drawing, the end 1042a of the line drawing and the end 1044a of the line drawing still exist.

In this way, by applying a coloring process to the line drawing using AI, even if it is a nail line drawing 1010a composed of an open line drawing, the nail portion is appropriately selected (specified). be able to. By doing so, it is possible to easily extract the selected portion of the nail 1041a and edit the image.

FIG. 5 (A) is a diagram showing an example in which coloring is performed using an AI different from that of FIG. 4 (A). FIG. 5B is a diagram showing an example in which the colored region is selected in FIG. 5A.

In FIG. 5A, a different AI algorithm is applied to color the inside of the nail line drawing 1010. In this case, the brightness of the coloring changes extremely in the vicinity of the area 1056, and the area 1055 is colored, but the area 1057 is white, which is the same as the background color. It is supposed to be uncolored.

FIG. 5B is a diagram showing an example in which the colored region is selected in FIG. 5A. In such a case, when the operator specifies the position of a certain part of the area 1055 and causes the area extraction unit 206 to select an area having substantially the same color as the specified position, the inside of the nail line drawing 1010a is formed. Region 1055a is in the selected (specified) state. However, the region 1055a is in a state of being selected up to the boundary 1056a, and the portion of the region 1057a is not selected (specified).

When the coloring unit 202 using AI is used, even if the AI is the same, FIGS. 5 (A) and 5 (B) are due to slight differences in line drawing patterns, differences in parameter settings, differences in the degree of learning, and the like. ) Can occur.
In order to cope with the situation as shown in FIG. 5, it is useful to use the following method.

FIG. 6 is a diagram showing an example after coloring is performed using the same AI as in FIG. 5 (A) and then edge-retaining smoothing processing is performed. FIG. 6B is a diagram showing an example in which the colored region is selected in FIG. 6A.

In FIG. 6 (A), after coloring using the same AI as in FIG. 5 (A), the edge-retaining smoothing process is applied to the colored regions 1055, 1056, and 1057 of the nail in FIG. It is a figure which shows the example of the region 1065 after the application.

As shown in FIG. 6 (A), the image of the nail line drawing 1010 is held in the nail region 1065 subjected to the edge-holding type smoothing treatment, and the colored region is colored up to the region 1067. It can be seen that it is in a state of being applied.
FIG. 6B is a diagram showing an example in which a nail region is selected by designating a position of a part of the colored regions 1065 and 1067 in FIG. 6A.

In FIG. 6B, since the portion of the region 1067a is colored, it can be seen that the selected (specified) portion indicated by the diagonal line covers the entire region of the nail including the 1065a and 1067a. ..

In this way, as shown in FIG. 6A, by performing the edge-holding type smoothing process, it is possible to smooth the coloring of the nail portion region without smoothing the line drawing including the edges. it can. By doing so, even when the coloring in the coloring portion 202 is biased to a part of the region, the edge-retaining smoothing process is performed to obtain an appropriate nail region. Can be selected (specified).

FIG. 7 (A) is a diagram showing an example in which a region is selected by designating a threshold value different from that of FIG. 6 (B) in the image of FIG. 6 (A). FIG. 7B is a diagram showing an example in which a colored region is selected by designating a threshold value different from that in FIG. 7A.

FIG. 7A shows an example in which the colored region 1077b at the tip of the nail is not selected because the color threshold is not appropriate. The selected region 1075b is selected up to the boundary 1076b at the tip of the nail.

FIG. 7B is a diagram showing an example in which a region is selected by designating a threshold value different from that in FIG. 7A. In this case, since a threshold value different from that in FIG. 7A is specified, the region up to 1077c is selected. Therefore, the boundary region 1076c is aligned with the tip portion of the nail, and is an appropriate boundary region.

By adjusting the threshold value of the color to be selected in this way, the region of the nail portion can be appropriately selected (specified). When the threshold value is specified, it is desirable that the operator can confirm the colored image or the colored image subjected to the edge-preserving smoothing process on the display screen. By doing so, the operator can determine whether the coloring process itself is appropriate or inappropriate, whether it is important to adjust the threshold value, or whether it is important to specify the smoothing parameters in the edge-preserving smoothing process. You can check.

In addition, it is desirable to set default values for the parameters and threshold values of the edge retention type smoothing process in advance in order to reduce the work of the operator. Alternatively, a plurality of pairs of these values may be prepared in advance so that the operator can select them. It is also desirable to allow the operator to fine-tune such values when such a selection is made.

FIG. 8 (A) is a diagram showing an example in which the nail portion is closed by adding a line drawing to the tip portion of the nail portion using the region selected in FIG. 7 (B). FIG. 8B is a diagram showing an example in which the nail portion of FIG. 7A is colored with a different color.

In FIG. 8 (A), for example, as in FIG. 74 (B), FIG. 6 (B), or FIG. 7 (C), when the claw portion is appropriately selected (specified), the end of the line is formed. An example is shown in which 1042 and the end of the line 1044 are added with a curve 1083d along the boundary of the selected region to make the nail line drawing 1080d a closed region.
By doing so, the line drawing 1010 of the nail that did not form the closed region can be corrected to the line drawing 1080d of the nail that forms the closed region.

By doing so, it is possible to easily obtain a line drawing that forms an appropriate closed region by appropriately connecting the ends of the lines with a curve with respect to the line drawing that does not form the closed region.

FIG. 8B is a diagram showing an example in which the nail portion of FIG. 7A is colored with a different color. With respect to the line drawing 1080e of the nail formed in the closed region formed in FIG. 8A, the nail 1085e colored with another color can be easily obtained.

FIG. 9A is an example in which the coloring of the nail portion of the closed line drawing is deleted. FIG. 9B is a diagram showing an example in which another image is attached to the claw portion of the closed line drawing.
FIG. 9A shows an example in which the color of the nail region 1095f inside the line drawing 1090f of the closed nail is deleted.
FIG. 9B shows an example in which another image is attached to the inner nail region 1095 g of the closed nail line drawing 1090 g. The image to be pasted is not particularly limited, but it is desirable to paste the image on a new layer in consideration of changing the pasted image to another image.

FIG. 10 (A) is a diagram showing an example of extracting an image of a nail including an unclosed line drawing obtained by FIG. 6 (A). FIG. 10B is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 8A. FIG. 10 (C) is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 9 (A). FIG. 10 (D) is a diagram showing an example in which a line drawing is obtained by removing a colored portion from the image obtained in FIG. 9 (A).

FIG. 10A shows an example in which the nail portion shown in FIG. 6A is extracted together with the nail line drawing 1100h. When extracting the nail region including the open line drawing 1100h, it is sufficient to extract a region slightly enlarged from the selected region.

FIG. 10B is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 8B. In this case, since the nail line drawing 1100j is closed, the inside of the nail and the closed nail line drawing may be selected and extracted.

FIG. 10 (C) is a diagram showing an example of extracting an image of a nail including a closed line drawing obtained by FIG. 9 (A). This process is also the same as the process shown in FIG. 10 (B).

FIG. 10 (D) is a diagram showing an example in which a line drawing is obtained by removing a colored portion from the image obtained in FIG. 9 (A). In this case, by sequentially deleting the colored images from the image shown in FIG. 9A, line drawings 800 m and 1100 m composed of closed line drawings can be obtained.

FIG. 11 is a diagram showing an example of learning the coloring function by AI. This AI is trained using GAN (Generative Adversarial Network). The line drawing 402 of the learning image is input to the generator 404. The generator 404 performs learning so as to generate a colored image 406 that is as similar as possible to the learning colored image 408. The colored image 406 generated by the generator 404 or the learning colored image 408 is input to the discriminator 410, and the determination of whether it is genuine (Real) or fake (Fake) is output.

The learning of the generator 404 and the learning of the discriminator 410 are usually executed alternately. By advancing the learning, the generator 404 will generate a colored image that closely resembles the colored image for learning, and the discriminator 410 will determine that the colored image generated by the generator 404 is genuine. Learning is advanced. The discriminator 410 performs learning so that the colored image generated by the generator 404 and the colored image for learning can be appropriately distinguished.
By using the generator 404 learned as described above for the coloring unit 202 of FIG. 2, the line drawing is colored.

FIG. 12 is a flowchart showing an example of image processing of one embodiment. Each step will be described below.
[Step S1202] The line drawing is received. The process proceeds to step S1204.
[Step S1204] The received line drawing is colored. The process proceeds to step S1207.
[Step S1207] If necessary, the colored line drawing is subjected to coloring correction (for example, edge-retaining smoothing process). Colored areas are selected (specified) and extracted using thresholds specified by the specified color (hue, saturation or lightness, or a combination thereof). The process proceeds to step S1210.
[Step S1210] Processing is added to the image of the extracted region. The process proceeds to step S1212.
[Step S1212] The intermediate image and the processed image of the processing are displayed (output). End the process.
By the above processing, an appropriate area is selected (specified) and extracted for the line drawing including the line that does not necessarily include the closed area. Appropriate processing can be easily applied to the extracted region.

FIG. 13 is a diagram showing an example in which a region of a predetermined color is extracted by correcting the coloring as necessary after coloring by AI. Note that FIG. 13 shows an example of extracting a predetermined color region without modifying the coloring because it is not necessary in this example. This process shows an example of the subroutine of step S1207 of FIG. 12 (correcting the coloring as necessary and extracting a region of a predetermined color). Each step will be described below.

[Step S1302] From the operator, for example, a position in the area to be extracted is specified. By this instruction, it is specified which area to process. The process proceeds to step S1304.

[Step S1304] A region having a color similar to the color around the designated position is specified. Similar colors can be specified by hue, saturation or lightness or a combination thereof. Grayscale without color may be specified. By designating two thresholds, a color region between the two thresholds is specified. Alternatively, one threshold value and the allowable amount of the upper and lower values may be specified. Alternatively, there may be a plurality of threshold values, and upper and lower threshold values such as hue, saturation, lightness, and tone may be set respectively.
The process proceeds to step S1306.
[Step S1306] The selected region is identified and extracted. The process proceeds to step S1308.

[Step S1308] For example, when an instruction to change the threshold value is received (YES), the region is specified again, so the process returns to step S1304. If the instruction to change the threshold value or the like has not been received (NO), the process returns.
By the above processing, the threshold value is changed by the operator, so that an appropriate area is selected.

FIG. 14 is a diagram showing an example in which a predetermined color region is extracted by correcting the coloring as necessary. An example of performing edge-preserving smoothing processing as a correction of coloring is shown. Then, a region of a predetermined color is extracted. When the edge-retaining smoothing process is used, in the coloring process, there are cases where the coloring is uneven over the entire desired region, or where the coloring is biased in a part of the region. This process shows another example of the subroutine of step S1207 of FIG. 12 (correcting the coloring as necessary and extracting a region of a predetermined color). Hereinafter, each step will be described.

[Step S1402] It is determined whether or not there is an instruction for edge-holding type smoothing. If this determination is YES, the process proceeds to step S1404. If this determination is NO, the process returns.

[Step S1404] An edge-preserving smoothing process is executed around the designated area. The process proceeds to step S1406.

[Step S1406] A region corresponding to the threshold range is specified and extracted. The region specified is preferably a continuous region. The process proceeds to step S1408.

[Step S1408]
It is determined whether or not a parameter change instruction related to the edge-preserving smoothing process has been received. If the determination is YES, the process proceeds to step S1404. If the determination is NO, the process returns.
By the above processing, an appropriate edge-preserving type smoothing process can be performed. It is desirable that this process be performed in combination with the threshold setting of FIG. That is, the processes of FIGS. 13 and 14 are not exclusive and may be executed in combination as necessary.

FIG. 15 is a diagram showing an example of processing an image of the extracted region.
The processes of FIGS. 15A to 15E show a subroutine example of step S1210 of FIG. Each process of FIG. 15 can be executed independently, but if there is no contradiction, a plurality of processes may be executed in combination. Alternatively, different processing may be applied to each of the plurality of regions existing in one image.
Each process will be described below.

FIG. 15A shows a process of coloring the extracted region.
[Step S1502] The extracted area is colored. Coloring can be done with a color different from the already colored color. Alternatively, various filtering processes may be applied to the extracted region.

FIG. 15B shows a process of removing a predetermined color from the extracted region.
[Step S1504] A predetermined color is removed from the extracted region. Prior to color removal, a process may be performed in which the endpoints of the open line art are connected to form a closed line art.

FIG. 15C shows a process of removing an image of a region not included in the extracted region.
[Step S1506] The image of the region not included in the extracted region is removed. By this process, it is possible to obtain an image in which only the image of the region of the extracted portion is left.

FIG. 15D shows a process of replacing the image of the extracted region with another image.
[Step S1508] A process of replacing the image of the extracted region with another image is performed. In consideration of the case where another image is replaced with another image, the image may be placed on a layer different from the colored image.

FIG. 15E shows a process of generating a line drawing by surrounding the extracted region with a closed curve.
[Step S1510] The extracted region may not be surrounded by a closed curve. In such a case, a closed curve may be formed by performing a process of adding a curve so that the end points of the curve are connected along the extracted region. Alternatively, a closed curve surrounding the extracted region is created so that the divided curves existing around the closed region are smoothly connected to form the closed curve while referring to the created closed curve. , The divided curves may be joined to form a closed curve.
By the above processing, an appropriate region can be easily extracted from the line drawing. Then, desired image processing can be applied to the extracted region.
It should be noted that the above image processing is an example and is not limited to these.
The AI used in the above embodiments is an AI trained using GAN, but the learning method of AI is not limited to GAN.

200 Image area extraction processing device 202 Coloring unit 204 Edge retention type smoothing processing unit 206 Area extraction unit 208 Image processing unit 210 Display unit

Claims (10)

An image area extraction processing method for extracting an image area approximately surrounded by the unclosed lines from a line drawing including unclosed lines.
Using an image conversion model that outputs a colored image from the input line drawing, which is machine-learned using the line drawing and the colored line drawing as teacher data, the line drawing including the unclosed line is converted into a colored line drawing. Steps to convert and
From among the color already line drawing or corrected the color already line drawing, extracting an image region color is applied in a shape surrounded by the approximated by the unclosed line,
Image area extraction processing method having. Between the conversion step and the extraction step,
A step of making corrections to retain and smooth the edges of the colored line art,
Further have,
The image area extraction processing method according to claim 1. The step of extracting the image area is
A step of identifying the image region by comparing the magnitude, saturation or brightness value of the colored line art or the corrected colored line art with a predetermined threshold value.
The image area extraction processing method according to claim 1 or 2, which comprises. Steps of coloring the extracted image area,
The image area extraction processing method according to any one of claims 1 to 3, further comprising. A step of removing a predetermined color from the extracted image area,
The image area extraction processing method according to any one of claims 1 to 3, further comprising. A step of removing a line drawing of a region not included in the extracted image area or the colored line drawing from the line drawing including the unclosed line , the colored line drawing or the corrected colored line drawing.
The image area extraction processing method according to any one of claims 1 to 3, further comprising. A step of replacing the extracted image area with a predetermined image,
The image area extraction processing method according to any one of claims 1 to 3, further comprising. A step of generating a line drawing by surrounding the extracted image area with a closed curve,
The image area extraction processing method according to any one of claims 1 to 3, further comprising. Steps to display the colored line art,
The image area extraction processing method according to any one of claims 1 to 8, further comprising. A program that causes a computer to execute the method according to any one of claims 1 to 9.

Images