TW202123167A - Method for processing pictures - Google Patents

Abstract

A method for processing pictures includes: recognizing a category of a picture based on a picture classification neural model; if the picture belongs to a type with a simple background, detecting an object and a non-object in the picture based on an object detection neural model; removing the non-object from the picture, so as to form a missing part as a result of the area overlapped by the object and the non-object; and filling the missing part to repair the object of the picture based on a picture repair neural model.

Description

Image processing method

The invention relates to a picture editing technology, in particular to a method of automatically editing pictures using artificial intelligence.

Online sales channels, such as auction networks, shopping networks, etc., will present pictures of the goods being sold, so that users can see photos of actual products. However, distributors often require suppliers to provide clear pictures to avoid displaying unnecessary information (such as text, patterns) on the pictures. Therefore, the supplier needs to spend extra time and manpower to edit and edit the pictures that meet the needs, which is very inconvenient.

In view of this, an embodiment of the present invention proposes a picture processing method, including: classifying a neural model according to the picture to identify the type of the picture; if the picture is of a type with a simple background, detecting the neural model according to the object, detecting the subject in the picture, and Non-subject; remove the non-subject in the picture so that the area where the subject and the non-subject overlap each other forms a missing part; and repair the missing part according to the picture to restore the main body by repairing the neural model.

To sum up, according to the embodiments of the present invention, non-subjects in the picture can be automatically detected and removed, and the main body can be repaired, so that a picture that meets the requirements can be obtained in a matching quantity.

Referring to FIG. 1, it is a flowchart of a picture processing method according to an embodiment of the present invention. The image processing method can be executed by an electronic device. 2 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 221, a memory 222, a non-transitory computer-readable recording medium 223, a peripheral interface 224, and a bus 225 for communicating the above components with each other. The processor 221 includes, but is not limited to, a central processing unit (CPU) 2213 and a neural network processor (NPU) 2215. The memory 222 includes, but is not limited to, volatile memory (such as random access memory (RAM)) 2224 and non-volatile memory (such as read-only memory (ROM)) 2226. The non-transitory computer-readable recording medium 223 may be, for example, a hard disk, a solid state disk, or the like. The peripheral interface 224 may include, for example, an input/output interface, a graphics interface, a communication interface (such as a network interface), and so on. The bus 225 includes, but is not limited to, one or more combinations of system bus, memory bus, and peripheral bus.

The electronic device may be composed of one or more computing devices. In some embodiments, the electronic device can support cloud computing services for other connected devices to connect and access. Cloud computing services include, but are not limited to, for example, infrastructure as a service, platform as a service, software as a service, storage as a service, desktop As a service (desktop as a service), data as a service (data as a service), security as a service (security as a service), and API (application programming interface) as a service (API as a service).

Refer to Figure 1 and Figure 3 together. FIG. 3 is a schematic diagram of a picture processing architecture according to an embodiment of the present invention. In step S110, the picture 400 to be processed is input into the picture classification neural model 310 to identify the type of the picture 400 according to the picture classification neural model 310. And in step S120, it is determined whether the picture belongs to a type that can be processed. If the identified picture type is a type that can be processed, the picture 400 is input into the object detection neural model 320 to detect the subject and non-subject in the picture 400 (step S130). Here, the type that can be processed is the type with a simple background in the picture 400. On the contrary, if it is a type that cannot be processed, here it is a type with repeated objects or a type with a complex background, the process ends. The simple background means that the background is a gradient color, uniform color, black and white, transparent, etc. The repeated object refers to that the picture 400 has a repeated main body. The complex background refers to photos or drawings with irregularities, such as landscapes, contexts, and humanities. The subject is a commodity, but the present invention is not limited to this. For example, the subject may be creatures such as characters and animals, or non-living creatures such as commodities and buildings. The non-subject may be text or/and a pattern. The text may be, for example, descriptive text (such as advertising copy, product description, trademark, watermark), and the pattern may be, for example, a promotional pattern, frame, watermark, trademark, etc.

Please refer to FIGS. 4 and 5. FIG. 4 is a schematic diagram of an original picture 400 according to an embodiment of the present invention, and FIG. 5 is a schematic diagram of a picture 400 with non-subject 420 removed according to an embodiment of the present invention. As shown in FIG. 4, the picture 400 includes a main body 410 and a non-main body 420. Here, the main body 410 takes a liquid crystal screen product as an example, and the non-main body 420 is the advertisement text (including the blue circular base map) located at the upper left of the main body 410 and the trademark located at the lower right of the main body 410. In step S140, the non-main body 420 in the picture 400 is removed, so that the area where the main body 410 and the non-main body 420 overlap each other forms a missing portion 430 (as shown in FIG. 5). In some embodiments, the area outside the main body 410 may also be removed (backing).

1 and 3, in step S150, the picture 400 with the missing part 430 is input to the image repair neural model 330 to repair the neural model 330 according to the picture, and the missing part 430 is repaired to restore the main body 410. Referring to FIG. 6, a schematic diagram showing a repaired picture 400 is shown.

Referring to FIG. 7, it is a schematic diagram of a neural model 310 for image classification according to an embodiment of the present invention. The image classification neural model 310 uses a multi-task learning method, and includes a feature extraction neural model 312 and a complex sub-neural network model 314. After the picture 400 is input to the feature extraction neural model 312, the feature extraction neural model 312 can obtain the complex feature vector of the picture 400. The feature extraction neural model 312 may, for example, use the EfficientNet open source model of Google, but the present invention is not limited to this. Here, three complex sub-neural network models 314 are used as examples to identify different types of pictures. Therefore, the parameters of the sub-neural network models 314 are not shared with each other. For example, the first sub-neural network model 314 is used to recognize whether there are patterns in the picture 400, the second sub-neural network model 314 is used to recognize whether the picture 400 has duplicate objects, and the third sub-neural network model 314 is used To identify whether the picture 400 has a complicated background. The sub-neural network model 314 may be implemented by using an inverted residual block (inverted residual block) in MobileNetV2.

Referring to FIG. 8, it is a schematic diagram of an object detection neural model 320 according to an embodiment of the present invention, which uses the RetinaNet architecture. The object detection neural model 320 first uses the Residual Network (ResNet) 321 to extract the feature map (Feature Map) of the image 400, and the extracted feature map uses the Feature Pyramid Networks (FPN) 323 to extract the feature map. Predict the features of each layer. Each prediction is input to a sub-neural network model 325 respectively. Each sub-neural network model 325 includes a class subnet (Class Subnet) 3251 and a box subnet (Box Subnet) 3252. The category subnet 3251 is used to obtain the object category, and the frame subnet 3252 is used to obtain the object location.

In one embodiment, the training method of the image classification neural model 310 and the object detection neural model 320 is described as follows. Referring to FIG. 9, it is a schematic diagram of generating training pictures according to an embodiment of the present invention to illustrate how to generate training pictures. First, provide multiple data sets. This data set includes the first data set 510 containing complex background pictures, the second data set 520 containing simple background pictures, the third data set 530 containing main pictures, and the fourth data containing non-main pictures Set 540. In some embodiments, the subject picture includes the subject and a simple background (such as a white background). Second, randomly select a first picture 601 from the first data set 510 or the second data set 520, a second picture 602 from the third data set 530, and a third picture from the fourth data set 540 603. Third, the first picture 601, the second picture 602, and the third picture 603 are synthesized into a training picture 700. By repeating the above steps, multiple training pictures 700 can be randomly generated. Based on these training pictures 700, the picture classification neural model 310 and the object detection neural model 320 can be trained.

In some embodiments, the training picture 700 is generated by itself. While being generated, it can be known which of the first data set 510 or the second data set 520 is the first picture 601 selected, and a first mark can be generated accordingly, that is, the mark is complicated or simple. In addition, the positions and sizes of the second picture 602 and the third picture 603 in the training picture 700 can also be known, so that the second mark for marking the subject 410 and the third mark for marking the non-subject 420 can be generated. The marking method may, for example, use geometric figures such as boxes or circle the main body 410 and the non-main body 420 according to the outline of the object. Therefore, during training, the output of the image classification neural model 310 and the object detection neural model 320 can be verified according to the first label, the second label, and the third label, so as to update the image classification neural model 310 and the object detection neural model 320 Parameters. That is to say, when the image classification neural model 310 and the object detection neural model 320 are trained, it is performed based on the training image 700 and the first, second, and third labels.

Referring to FIG. 10, it is a schematic diagram of the image repair neural model 330 according to an embodiment of the present invention, which uses the EdgeConnect architecture. The image repair neural model 330 includes an edge generator 332 and a repair generator 334. First, a grayscale, an edge, and a mask are generated according to the picture 400 with the missing part 430, and input to the edge generator 332 to generate a predicted edge map. Then, the predicted edge map and the picture 400 with the missing part 430 are input to the inpainting generator 334 for picture inpainting, and the inpainted picture 400 is obtained at the output end. Here, the edge generator 332 is composed of a generator 3321 and a discriminator 3322. The repair generator 334 is also composed of a generator 3341 and a discriminator 3342. Each generator 3321, 3341 includes an encoder (Encoder), a dilated convolution (​Dilated Convolution​), a residual block (Residual Block), and a decoder (Decoder). Each discriminator 3322, 3342 includes a complex convolutional layer.

In an embodiment, the training method of the image repair neural model 330 is described as follows. First, the edge generator 332 is trained. First, one or more of the aforementioned training pictures 700 are randomly masked to form a plurality of training data. These training data are then input to the edge generator 332 to obtain the detection result output by the edge generator 332. On the other hand, an edge detection algorithm (such as Canny edge detection) is used to calculate the edges of the training data to verify the detection result of the edge generator 332. In some embodiments, the training image 700 is a grayscale image, or the training image 700 is converted into a grayscale image in advance. Second, the repair generator 334 is trained. Specifically, the edges and training data calculated by the aforementioned edge detection algorithm for the training picture 700 are input to the repair generator 334 to obtain the detection result of the repair generator 334 and verify the accuracy of the repair. Third, freeze the parameter update of the edge generator 332, and use the training data to train the edge generator 332 and the repair generator 334.

To sum up, according to the embodiment of the present invention, the non-subject 420 in the picture 400 can be automatically detected and removed, and the main body 410 can be repaired, and the picture 400 that meets the requirements can be obtained in a matching quantity.

221: Processor 222: Memory 223: Non-transient computer readable recording media 224: Surrounding interface 225: Bus Bar 2213: Central Processing Unit 2215: Neural Network Processor 2224: Volatile memory 2226: Non-volatile memory 310: Image classification neural model 312: Feature extraction neural model 314: Sub-neural network model 320: Object detection neural model 321: Residual Network 323: Feature Pyramid Network 325: Sub-neural network model 3251: Category subnet 3252: Frame network 330: Picture repair neural model 332: Edge generator 3321: Generator 3322: Discriminator 334: Repair generator 3341: Generator 3342: Discriminator 400: Picture 410: Subject 420: Non-subject 430: Missing Department 510: The first data set 520: The second data set 530: The third data collection 540: Fourth Information Collection 601: The first picture 602: The second picture 603: The third picture 700: Training pictures S110, S120, S130, S140, S150: Steps

[Fig. 1] is a flowchart of a picture processing method according to an embodiment of the present invention. [Fig. 2] is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. [Figure 3] is a schematic diagram of a picture processing architecture according to an embodiment of the present invention. [Figure 4] is a schematic diagram of an original picture according to an embodiment of the present invention. [Fig. 5] is a schematic diagram of removing non-subject pictures according to an embodiment of the present invention. [Figure 6] is a schematic diagram of a repaired picture according to an embodiment of the present invention. [Fig. 7] is a schematic diagram of a neural model for image classification according to an embodiment of the present invention. [Fig. 8] is a schematic diagram of an object detection neural model according to an embodiment of the present invention. [Fig. 9] is a schematic diagram of generating training pictures according to an embodiment of the present invention. [Figure 10] is a schematic diagram of an image repair neural model according to an embodiment of the present invention.

S110, S120, S130, S140, S150: steps

Claims (10)

An image processing method, including: Identify the type of a picture according to a picture classification neural model; If the picture is of a type with a simple background, detect a subject and a non-subject in the picture according to an object detection neural model; Removing the non-subject in the picture so that the area where the main body and the non-subject overlap each other forms a missing part; and Repair the neural model according to a picture, and repair the missing part to restore the main body. The image processing method according to claim 1, wherein the step of identifying the type of the image according to the image classification neural model includes: According to a feature extraction neural model, the complex feature vector of the picture is obtained; and The feature vectors are input to the complex neural network model, each of the neural network models has parameters that are not shared with each other, so as to recognize different image types respectively. The image processing method according to claim 2, wherein the type of the image also includes a type with repeated objects that does not belong to the type with a simple background and a type with a complex background. The picture processing method according to claim 3, wherein if the result of the step of identifying the type of a picture is the type with the repeated object or the type with the complex background, the picture is not processed. The image processing method according to claim 1, wherein the image repair neural model includes an edge generator and a repair generator. The image processing method described in claim 5 further includes the step of training the image repair neural model, including: Input the complex training data formed by randomly occluding an area with a training picture to the edge generator, and verify the detection result of the edge generator according to an edge detection algorithm; Use the edge detection algorithm to train the repair generator on the edge calculated for the training picture and the training data; and The parameter update of the edge generator is frozen, and the training data is used to train the edge generator and the repair generator. The image processing method according to claim 1, wherein the non-subject is a text or a pattern. The image processing method according to claim 1, wherein the subject is a commodity. The image processing method described in claim 1, further including: Provide multiple data sets, including a first data set containing complex background pictures, a second data set containing simple background pictures, a third data set containing main pictures, and a A fourth data set of the main picture; Randomly select a first picture from the first data set or the second data set, a second picture from the third data set, and a third picture from the fourth data set; Synthesize the first picture, the second picture, and the third picture into a training picture; and Training the image classification neural model and the object detection neural model according to the training image. The image processing method described in claim 9 further includes: According to which of the first picture is selected from the first data set or the second data set, a first mark is generated; and Generate a second mark and a third mark respectively according to the position and size of the second picture and the third picture in the training picture; Wherein, the step of training the picture classification neural model and the object detection neural model according to the training picture is also trained according to the first mark, the second mark, and the third mark.

Images