WO2023239299A1 - 图像处理方法、装置、电子设备及存储介质 - Google Patents
图像处理方法、装置、电子设备及存储介质 Download PDFInfo
- Publication number
- WO2023239299A1 WO2023239299A1 PCT/SG2023/050392 SG2023050392W WO2023239299A1 WO 2023239299 A1 WO2023239299 A1 WO 2023239299A1 SG 2023050392 W SG2023050392 W SG 2023050392W WO 2023239299 A1 WO2023239299 A1 WO 2023239299A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- preset
- model
- preset object
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- the present disclosure provides an image processing method, device, electronic device and storage medium to remove target objects in images in real time and reduce the cost of removing element objects in images.
- An embodiment of the present disclosure provides an image processing method.
- the method includes: obtaining an image to be processed, wherein the image to be processed is an image with a preset object, and some pixels of the preset object are located in the image to be processed. Within the main body outline area in the image, another part of the pixel points of the preset object is located outside the main body outline area; input the image to be processed into the preset object removal processing model to obtain a target image, where, the target image Remove images for objects corresponding to the images with preset objects;
- the preset object removal processing model is a model obtained by training a set based on pre-established image samples without preset objects, wherein, the no preset objects
- Each non-preset object image sample pair in the object image sample pair set includes an original image with a preset object, and the pixel points and the preset object's pixels located outside the body outline area in the original image respectively.
- An embodiment of the present disclosure also provides an image processing device.
- the device includes: an image acquisition module configured to acquire an image to be processed, wherein the image to be processed is an image with a preset object, and the preset object is Some pixel points are located within the subject outline area in the image to be processed, and another part of the pixel points of the preset object are located outside the subject outline area; an image processing module configured to input the image to be processed to the preset object Remove the processing model and obtain the target image, where, the target image Remove images for preset objects corresponding to the images with preset objects;
- the preset object removal processing model is a model obtained by training a set based on pre-established image samples without preset objects, wherein, the preset object removal processing model is Each non-preset object image sample pair in the set of preset object image sample pairs includes an original image with a preset object, and identifies the
- the preset object-removed image is obtained by processing the pixel points of the preset object outside the main body outline area and the pixel points of the preset object located in the main body outline area in the original image.
- An embodiment of the present disclosure also provides an electronic device, including: at least one processor; a storage device configured to store at least one program, and when the at least one program is executed by the at least one processor, the at least one process The processor implements the image processing method described in any one of the embodiments of the present disclosure.
- Embodiments of the present disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in any embodiment of the present disclosure.
- Figure 1 is a schematic flow chart of an image processing method provided by an embodiment of the present disclosure
- Figure 2 is a schematic diagram of an image to be processed provided by an embodiment of the present disclosure
- Figure 3 is an image processing method provided by an embodiment of the present disclosure.
- Figure 4 is a schematic flowchart of an image processing method provided by an embodiment of the present disclosure;
- Figure 5 is a schematic diagram of an image background repair provided by an embodiment of the present disclosure;
- Figure 6 is a schematic flowchart of an image processing method provided by an embodiment of the present disclosure.
- the user when responding to the user's active request, send a prompt message to the user to clearly prompt the user that the operation requested will require obtaining and using the user's Personal information. Therefore, the user can autonomously choose according to the prompt information whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that perform the operations of the technical solution of the present disclosure.
- the method of sending prompt information to the user in response to receiving the user's active request, can be, for example, a pop-up window, and the prompt information can be presented in the form of text in the pop-up window.
- FIG. 1 is a schematic flow chart of an image processing method provided by an embodiment of the present disclosure.
- the embodiment of the present disclosure is suitable for converting images to be processed into
- the method can be executed by an image processing device, which can be implemented in the form of software and/or hardware, and can optionally be implemented by an electronic device.
- the electronic device can be a mobile terminal or a personal computer.
- Computer Personal Computer, PC
- the image processing method includes:
- the image to be processed is an image containing an image special effects processing object, and can be an image obtained by downloading, shooting, or uploading.
- the image to be processed is an image with a preset object.
- the preset object is the image special effects processing object in the image feature processing process. It is the target object that needs to be removed, and some pixels of the preset object are located in the to-be-processed image. Within the subject outline area in the processed image, another part of the pixels is located outside the subject outline area in the image to be processed.
- the pixels of the preset object located in different areas can be processed separately according to the pixel information characteristics of different parts.
- the subject may be a foreground object containing partial pixels of the preset object or a partial area object of the foreground in the image to be processed, and the subject outline is a line formed by edge pixels of the corresponding foreground or partial area of the foreground.
- a unified pixel value can be used to replace the pixel value of the preset object, such as making the preset object pure white or pure black, or using the average pixel of the image to be processed. The value replaces the pixel value of the preset object.
- the pixel points of the preset object within the subject outline range appear as the pixel features of the subject itself in the image to be processed.
- the pixel points of the preset object outside the subject outline range are removed, they appear as the pixel features of the subject image to be processed. Pixel characteristics of the background other than the subject.
- the preset object can be any object. When the relationship between the preset object in the image to be processed and the foreground subject object in the image satisfies the requirement that some of the pixel points of the preset object are located in the subject outline area of the image to be processed, and the other part of the pixel points are located in When the subject outline area in the image to be processed is outside, the image processing method of this embodiment can be used to perform image special effects processing.
- an image to be processed is shown.
- the subject is a dining table
- the preset object is an object placed on the dining table
- the object is two parts separated by a dotted line.
- part of the pixel points of the object are within the outline range of the dining table
- the other part of the pixel points are outside the outline range of the dining table.
- the image special effect processing effect of removing objects is that the pixels of the corresponding objects within the outline of the main body of the dining table are processed into pixels that are consistent with the main body of the dining table, and the pixels of the corresponding objects outside the outline of the main body of the dining table are processed into pixels that are consistent with the main body of the dining table in the image to be processed. pixels consistent with the background.
- the preset object removal processing model can realize the special effect of removing preset objects from the image to be processed. If the image to be processed containing the preset object is input into the preset object removal processing model, the corresponding output result can be obtained, that is, it does not contain the preset object.
- the preset object removal processing model may be based on a pre-established set of image sample pairs without preset objects. The model obtained by training.
- Each image sample pair without a preset object includes an original image with a preset object, and a pair of pixels of the preset object located outside the subject outline area in the original image and pixels located in the original image.
- the preset object removal image obtained by processing the pixel points of the preset object within the subject outline area.
- the preset object removal processing model can learn the mapping relationship between the original image with the preset object and the corresponding preset object removal image to achieve the preset object removal effect.
- the training process of the preset object removal processing model may include the following steps: Step-1: Identify the main body contour area showing the preset object in the original image with the preset object. In this step, the corresponding subject contour area can be identified and extracted in the original image through interactive image segmentation technology.
- Step 2 Process the pixels of the preset object located in the main body outline area in the original image to be the same as the pixels of the preset object in the main body outline area of the original image. pixels with consistent pixel information, and process pixels of the preset object located outside the main body outline area in the original image into non-preset pixels outside the main body outline area in the original image. If the pixel information of the pixels of the object is consistent, the preset object removal image is obtained.
- the consistency of pixel information can be understood as the same pixel characteristics, or the visual effect of the pixel information is the same.
- the pixels of the preset object are processed, that is, the pixels of the preset object in one area are processed first, and then the pixels of the preset object in another area are processed.
- the pixels of preset objects in different areas can also be processed according to corresponding pixel processing strategies at the same time.
- the average pixel value of the area can be used to replace the pixel value of the preset object, or interpolation can be used to calculate the preset based on the pixel information of the area.
- the updated pixel value of the object can be used to replace the pixel value of the preset object.
- Step 3 Train an initial image removal model based on the original image and the preset object removal image to obtain the preset object removal processing model.
- the original image can be used as the model input, and the preset object The object removal image is the expected output of the model.
- the training process can be completed, thereby obtaining the preset object removal processing model, which is used for During the removal process of the preset object.
- the technical solution of the embodiment of the present disclosure is that when an image to be processed is acquired, and the image to be processed is an image with a preset object, the pixel points of the preset object are located in the subject outline area of the image to be processed, and the other part of the pixel points are When located outside the body contour area; the image to be processed can be input to the preset object removal processing model to obtain the target image after the preset object is removed, wherein the preset object removal processing model is based on the preset-free A model obtained by training a set of object image sample pairs, where each pair of image samples without preset objects includes the original image with the preset object, and the preset object pixels located outside the subject outline area and the preset object pixels located in the subject.
- FIG. 3 is a schematic flow chart of another image processing method provided by an embodiment of the present disclosure. In the process of implementing the method flow, it describes the training process of the preset object removal processing model when the target object to be removed is hair. , especially the construction process of model training sample pairs.
- the method may be executed by an image processing device, which may be implemented in the form of software and/or hardware, or optionally, by an electronic device, which may be a mobile terminal, a PC, or a server.
- an image processing method includes the following steps.
- S210> Construct a pair of image samples without preset time for training the preset object removal processing model.
- an original image with a preset object is constructed, and a sample consisting of an image without the preset object after hair removal corresponding to the original image with the preset object is constructed.
- the pixels of the preset object outside the head area correspond to the background part of the original image after removal.
- Making the pixels of the preset object consistent with the pixel information of the non-preset object pixels outside the subject outline area in the original image after processing is to make the pixels of the preset object become part of the background of the original image after processing , which can achieve the effect of seamlessly removing preset objects.
- Deep learning can be used to superimpose the binary image of the head area with the original image, and input the image superposition result into the image background repair model to obtain a preliminary object removal image that removes preset objects located outside the head area.
- Superimposing the binary image of the head area with the original image can temporarily mask the pixel information in the head area of the original image, and avoid being affected by the pixel information in the head area during the image processing process of the image background repair model.
- Step 3 Based on Step 2, process the pixel points of the preset object located in the main body outline area in the original image to be the same as the non-preset objects in the main body outline area in the original image. pixels whose pixel information is consistent. After removing the hair in the head area, the pixel position corresponding to the head and hair corresponds to the scalp part.
- the preliminary object removal image obtained in step 2 can be input into the facial skin repair model to obtain the pixels located in the head area. Complete object removal images of preset object removal.
- the facial skin repair model is an image processing model trained based on the image of a bald head that does not contain hair and the head area corresponding to the bald head image superimposed on the hairstyle mask. This model can restore the bald head image based on the pixel information in the head area of the bald head image.
- the hair mask covers the area of facial skin, so that a complete head area image without hair can be obtained.
- S240 Input the image to be processed into the preset object removal processing model to obtain the target image.
- the technical solution of the embodiment of the present disclosure is to gradually process on the basis of the original image to obtain the corresponding sample image without preset objects and then construct a model training sample pair to train the preset object removal processing model, and then after obtaining the image to be processed , input the image to be processed into the preset object removal processing model to obtain the target image, which solves the unnatural problem in related technologies of changing the image attributes by adding special effects locally to the image, and realizes the real-time removal of targets in the image. image, which reduces the cost of target object removal in the image.
- FIG. 4 is a schematic flow chart of another image processing method provided by an embodiment of the present disclosure.
- the process of implementing the method flow it describes the training process of the preset object removal processing model when the target object to be removed is hair.
- the method can be executed by an image processing device, which can be implemented in the form of software and/or hardware, and optionally, can be implemented by an electronic device.
- the electronic device can be a mobile terminal, a PC or a server.
- a head region prediction model and use the head region prediction model to identify the main body outline region of the preset object in the original image with the preset object.
- the head area can be understood as the real head area after removing the hair (default object). This area can be represented by a black and white binary image.
- a model training sample pair is constructed.
- the model training sample pair includes the original image with the preset object, and the binary image of the head region corresponding to the original image.
- a three-dimensional skull model can be established. The display angle of the three-dimensional skull model can be adjusted arbitrarily, and the skull outline can also be adjusted.
- multiple original sample images with preset objects of known skull structures are rendered.
- a three-dimensional skull model with a corresponding display angle is matched; then the three-dimensional skull model is projected into a plane to obtain a binary image of the skull region that matches the original sample image.
- the outline of the binary image of the skull area can also be adjusted according to the facial outline of the human object in the original sample image.
- the original sample image can be used as the model input image, and the corresponding head region binary image can be used as the model's expected output image for neural network model training to obtain a head region prediction model.
- the trained head region prediction model can be used to predict the subject outline area (i.e., head area) on the original image with preset objects.
- the image background repair model trains the image background repair model, and process the pixel points of the preset object outside the subject outline area in the original image based on the image background repair model to obtain a primary object removal image. After obtaining the original sample image correspondence After removing the hair from the skull area, it becomes hairless. State, the hair can be divided into hair inside the skull area and hair outside the skull area. The hair outside the skull needs to be processed into the background, and the hair inside the skull area needs to be processed into facial skin.
- the image background repair model is a model used to process hair outside the head area.
- the head region binary images in the preset head region binary image set and the background images in the preset background image set are randomly combined to obtain multiple combinations, and each combination is The combined binary image of the head region is superimposed on the background image to obtain the first superimposed sample image, as shown in the left image in Figure 5.
- the head area of a head area binary image is represented by black.
- the first superimposed sample image is obtained by randomly superimposing the head area binary image with a background image. This processing is equivalent to The pixel information inside the head area is covered, and the pixel information inside the head area will not be extracted during the model learning image features.
- preset object pixel marking is performed outside the corresponding head area in the first superimposed sample image to obtain a first superimposed sample marked image, such as the middle image in Figure 5, where the gray area represents the preset object pixel mark. It can be understood that the area marked by the preset object pixel can be set with reference to different hairstyles.
- the neural network model is trained based on the first superimposed sample image and the first superimposed sample marked image to obtain an image background repair model.
- the first overlay sample marked image can be input into the initial image background repair model to obtain the first model generated image; the first model generated image can be combined with any other image in the preset background image set except the background image in the first overlay sample image.
- a background image is input into the first discriminator; the initial image background repair model is updated based on the output result of the first discriminator and the comparison result between the first model generated image and the first superimposed sample image to obtain an image background repair model.
- the background repair effect shown in the right picture in Figure 5 can be obtained by processing the first superimposed sample marked image.
- adding a second discriminator can make the result of image background repair closer to the real background image, and the effect is more natural, so that the discriminator cannot distinguish the original image.
- the background image is also a patched background image.
- a certain acceptable error will be allowed between the first model-generated image and the first superimposed sample image, and the two will not be forced to be exactly the same to avoid over-fitting of the model.
- the code image can be set with reference to various hairstyles. Then, a preset number of head region anchor points are collected in the head region of the second superimposed sample image according to the preset calibration point collection strategy, and a second superimposed sample image marked with the preset number of head region anchor points is obtained.
- the head region anchor point can be used as auxiliary reference information for the scraped neural network model, so that the neural network model can distinguish between the head region and outside the head region, and extract features within the area corresponding to the head region anchor point.
- the facial skin can be obtained by performing neural network model training based on the second superimposed sample image marked with a preset number of head region anchor points and the corresponding sample image that does not contain the preset object (the inverted right image in Figure 6). Patch the model.
- the strategy for collecting anchor points in the skull area can be: for the interior of the skull area of the second superimposed sample image, perform anchor point sampling according to the contour of the facial features; for the edge of the skull area contour of the second superimposed sample image, perform sampling at the contour edge according to the preset sampling interval Anchor point sampling*
- Anchor point sampling results please refer to the black point on the left in Figure 6.
- the second superimposed sample image marked with the preset number of head region anchor points can be input into the initial facial skin repair model to obtain a second model generated image; generate the second model
- the image and any sample image that does not contain the preset object except the original sample image that does not contain the preset object corresponding to the second superimposed sample image are input into the second discriminator; based on the output result of the second discriminator and The comparison result between the second model-generated image and the original sample image that does not contain the preset object corresponding to the second superimposed sample image is used to update the initial facial skin repair model to obtain a facial skin repair model.
- a second discriminator can make the result of facial skin repair in the image closer to the real image of a person without a preset object, making the effect more natural and making the facial skin repair result more natural.
- the discriminator cannot distinguish whether it is the original human object image without preset objects or the image without preset objects after facial skin patching.
- the technical solution of the embodiment of the present disclosure is to separately train the head region prediction model, the background repair model and the facial skin repair model.
- the corresponding non-preset object sample is obtained through step-by-step processing through the scraped and trained model.
- the image is then used to construct a model training sample pair to train the preset object removal processing model.
- the to be processed image is input to the preset object removal processing model to obtain the target image, which solves the problem of image-based processing in related technologies.
- FIG. 7 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure.
- the device is suitable for removing preset objects from images to be processed. It can be implemented in the form of software and/' or hardware.
- the image processing device can be configured on an electronic device, and the electronic device can be a mobile terminal, a PC, a server, etc.
- the image processing device includes: an image acquisition module 410 and an image processing module 420.
- An image acquisition module is configured to acquire an image to be processed, wherein the image to be processed is an image with a preset object, Some of the pixel points of the preset object are located within the main body outline area in the image to be processed, and another part of the pixel points of the preset object are located outside the main body outline area; the image processing module is configured to process the to-be-processed image.
- the image is input to the preset object removal processing model to obtain the target image, wherein the target image is the preset object removal image corresponding to the image with the preset object; the preset object removal processing model is based on.
- the technical solution provided by the embodiment of the present disclosure is to obtain an image to be processed, wherein the image to be processed is an image with a preset object, and some pixels of the preset object are located in the subject outline area in the image to be processed.
- the pixel points is located outside the main body outline area; input the image to be processed into the preset object removal processing model to obtain a target image, where the target image corresponds to the image with the preset object
- the preset object removal image obtained by processing the pixels of the preset object solves the problem in related technologies of changing the image attributes by adding special effects locally to the image, and achieves the real-time removal of target objects in the image. Reduce the cost of target object removal in images.
- the image processing device further includes a model training sample construction module, configured to: identify the body outline area showing the preset object in the original image with the preset object Domain; Process the pixels of the preset object located in the main body outline area in the original image into pixels that are not the preset object pixels in the main body outline area of the original image. pixels with consistent information, and process the pixels of the preset object located outside the main body outline area in the original image as non-preset pixels outside the main body outline area in the original image.
- the pixels with consistent pixel information of the object's pixels are used to obtain a preset object-removed image; the original image and the preset object-removed image are composed of the preset object-free image sample pair.
- the model-training sample construction module is configured to: input the original image with the preset object into the skull area prediction model , obtain a binary image of the head region showing the preset object; superimpose the binary image of the head region with the original image to obtain an image overlay result, and input the image overlay result into the image background repair model, Obtain a preliminary object removal image in which the preset objects located outside the head area in the original image are removed; input the preliminary object removal image into the facial skin repair model to obtain the original image
- the preset object-removed complete object-removed image located in the head area; the original image and the complete object-removed image form the preset-object-free image sample pair.
- the image processing device further includes a first auxiliary model training module configured to train the head region prediction model.
- the training process includes the following steps: Obtain the image with the preset time image. any sample image, and match the corresponding three-dimensional skull model for any sample image; perform planar projection on the three-dimensional skull model to obtain a binary image of the head region that matches any sample image; match the any sample image A sample image is used as the model input image, and the binary image of the head region matching any of the sample images is used as the expected output image of the model for neural network model training to obtain the head region prediction model.
- the image processing device further includes a second auxiliary model training module configured to train the image background repair model.
- the training process includes the following steps: Add the preset head region binary image set to The binary image of the head region is randomly combined with the background image in the preset background image set to obtain multiple combinations, and the binary image of the head region in each combination is superimposed on the background image in each group to obtain A first superimposed sample image; Mark the preset object pixels outside the corresponding head area in the first superimposed sample image to obtain a first superimposed sample marked image; Based on the first superimposed sample image and the first superimposed sample The labeled image is trained on a neural network model to obtain the image background repair model.
- the second auxiliary model training module is configured to: input the first superimposed sample marked image into the initial image background repair model to obtain the first model generated image; Any background image in the first model generated image and the preset background image set except the background image in the first superimposed sample image is input into the first discriminator; based on the first discriminator; The initial image background repair model is updated based on the output result of the discriminator and the comparison result between the first model generated image and the first superimposed sample image to obtain the image background repair model.
- the image processing device further includes a third auxiliary model training module, configured to train the facial skin repair model.
- the training process includes the following steps: any collected images that do not contain Superimpose the preset object mask image in the head area of the sample image of the preset object to obtain a second superimposed sample image; collect a preset number of head areas in the head area of the second superimposed sample image according to the preset calibration point collection strategy anchor points to obtain a second superimposed sample image marked with the preset number of head region anchor points; based on the second superimposed sample image marked with the preset number of head region anchor points and the corresponding image that does not contain the preset Assume that the sample image of the object is trained on the neural network model to obtain the facial skin repair model.
- the third auxiliary model scraping module is configured to: input the second superimposed sample image marked with the preset number of head region anchor points into the initial facial skin. Obtain a second model-generated image from the repair model; compare the second model-generated image with any image except the collected sample image that does not contain a preset object corresponding to the second superimposed sample image. The sample image that does not contain the preset object is input into the second discriminator; based on the output result of the second discriminator and the second model, an image and the collected image corresponding to the second superimposed sample image are generated. Based on the comparison result of any sample image that does not contain the preset object, the initial facial skin repair model is updated to obtain the facial skin repair model.
- the third auxiliary model training module may also be configured to: perform anchor point sampling according to facial features contours inside the skull area of the second superimposed sample image; Anchor point sampling is performed on the contour edge of the head region of the sample image according to a preset sampling interval.
- the image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and has functional modules corresponding to the execution method.
- FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Referring now to FIG. 8 , a schematic structural diagram of an electronic device (such as the terminal device or server in FIG. 8 ) 500 suitable for implementing embodiments of the present disclosure is shown.
- Terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablet computers (Portable Android Device, PAD), and portable multimedia players. (Portable Media Player, PMP).
- Mobile terminals such as vehicle-mounted terminals (such as vehicle-mounted navigation terminals) and mobile terminals such as Digital TV (Television, TV).
- Fixed terminals such as desktop computers.
- the electronic device 500 shown in FIG. 8 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present disclosure.
- the electronic device 500 may include a processing device (such as a central processing unit, a graphics processor, etc.) 50L, which may process data according to a program stored in a read-only memory (Read-Only Memory, ROM) 502 or from a storage device 508
- ROM Read-Only Memory
- RAM Random Access Memory
- the processing device 50E ROM 502 and RAM 503 are connected to each other via a bus 504 .
- An input/output (hipirt/output I/O) interface 505 is also connected to bus 504.
- Input devices S06 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD) , an output device 507 such as a speaker, a vibrator, etc.; a storage device 508 including a magnetic tape, a hard disk, etc.; and a communication device 509.
- Communication device 509 may allow electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data.
- FIG. 8 illustrates electronic device 500 with various devices, it should be understood that implementation or availability of all illustrated devices is not required. More or fewer means may alternatively be implemented or provided.
- embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including program code for performing the method illustrated in the flowchart. .
- the computer program may be downloaded and installed from the network via communication device 509, or from storage device 508, or from ROM 502.
- the processing device 501 When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are performed.
- the names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.
- Embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided by the above embodiments is implemented.
- the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
- the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof.
- Examples of computer-readable storage media may include, but are not limited to For: Electrical connections with one or more wires, portable computer disks, hard drives, RAM, ROM, Erasable Programmable Read-Only Memory (EPROM) or flash memory, fiber optics, portable compact disks only Read-only memory (Compact Disc Read-Only Memoiy, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
- a computer-readable storage medium may be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device.
- the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried.
- This propagated data signal can take many forms, including but not limited to electromagnetic signals > optical signals or any suitable combination of the above.
- a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
- Program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
- the client and server can communicate using any currently known or future developed network protocol such as HyperText Transfer Protocol (HTTP), and can communicate with digital data in any form or medium. Communication (e.g., communication network) interconnection.
- HTTP HyperText Transfer Protocol
- Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any current network for knowledge or future research and development.
- the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
- the computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an image to be processed, wherein the image to be processed is an image with a preset object.
- the preset object removal processing model is based on a pre-established image without a preset object A model obtained by training a set of sample pairs, wherein each pair of image sample pairs without a preset object in the set of image sample pairs without a preset object includes an original image with a preset object, and is respectively aligned with the body contour
- the preset object removal image is obtained by processing the preset object pixel points outside the area and the preset object pixel points located within the body contour area.
- each block in the flowchart or block diagram may represent a module, segment, or portion of code that contains one or more items that implement the specified Executable instructions for logical functions.
- the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown one after another may actually execute substantially in parallel, or they may execute in a sequential order, depending on the functionality involved.
- each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of dedicated hardware and computer instructions.
- the units involved in the embodiments of the present disclosure may be implemented in software or hardware. In one case, the name of the unit does not constitute a limitation on the unit itself.
- the first acquisition unit can also be described as "the unit that acquires at least two Internet Protocol addresses.”
- the functions described above herein may be performed, at least in part, by one or more hardware logic components.
- exemplary types of hardware logic components include: field programmable gate array (Field Programmable Gate Array, FPGA), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), application specific standard product (Application Specific Standaid Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), etc.
- the machine-readable medium may be a tangible medium, which may A program contained or stored for use by or in conjunction with an instruction execution system, apparatus, or device.
- the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
- Machine-readable medium May include but are not limited to electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems" devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media would include electrical connections based on one or more wires. Laptop computer disks. Hard drives. RAM. ROM. EPROM or flash memory, fiber optics. CD-ROM, optical storage devices, magnetic storage. storage device, or any suitable combination of the above.
- Example 1 provides an image processing method, which method includes: obtaining an image to be processed, wherein the image to be processed is an image with a preset object, and the preset object is Assume that some pixel points of the object are located within the main body outline area in the image to be processed, and another part of the pixel points of the preset object are located outside the main body outline area; input the image to be processed into the preset object removal processing model , to obtain a target image, wherein the target image is an object-removed image corresponding to the image with a preset object; the preset object removal processing model is based on a set of pre-established image samples without preset objects.
- each non-preset object image sample pair in the set of preset object-free image sample pairs includes an original image with a preset object, and each pair of the original image is located in the body contour.
- the preset object-removed image is obtained by processing the pixel points of the preset object outside the area and the pixel points of the preset object located in the body contour area in the original image.
- Example 2 provides an image processing method, which further includes: In some optional implementations, the non-preset object image sample pairs no preset objects in the set.
- the construction process of object image sample pairs includes: identifying the main body outline area in the original image with the preset object that exhibits the preset object; converting the preset object located in the main body outline area in the original image
- the pixels in the original image are processed into pixels consistent with the pixel information of the pixels of the non-preset object in the main body outline area in the original image, and the pixels in the original image located outside the main body outline area are processed
- the pixel points of the preset object are processed into pixel points that are consistent with the pixel information of the pixel points of the non-preset object outside the main body outline area in the original image, and a preset object removed image is obtained;
- the original image and the preset object-removed image form the preset-object-free image sample pair.
- Example 3 provides an image processing method, including:
- the construction process of the image sample pairs without preset objects in the set of image sample pairs without preset objects includes: inputting the original image with the preset objects into the head area prediction model to obtain the displayed
- the binary image of the head region of the preset object is superimposed on the binary image of the head region and the original image to obtain an image overlay result, and the image overlay result is input into the image background repair model to obtain the original image.
- Example 4 provides an image processing method, which also includes:
- the training process of the head region prediction model includes: Obtain any sample image with the preset object, and match the corresponding three-dimensional skull model for any sample image; perform planar projection on the three-dimensional skull model to obtain a skull region matching the any sample image.
- Example 5 provides an image processing method, which also includes:
- the training process of the image background repair model includes: Randomly combine the head region binary images in the preset head region binary image set with the background images in the preset background image set to obtain multiple groups, and superimpose the head region binary images in each combination on the background image.
- a superimposed sample labeled image is used for neural network training to obtain the image background repair model.
- Example 6 provides an image processing method, further comprising: In some optional implementations, based on the first superimposed sample image and the third Performing neural network model training on a superimposed sample labeled image to obtain the image background repair model includes: inputting the first superimposed sample labeled image into the initial image background repair model to obtain a first model generated image; Any background image in the model generated image and the preset background image set except the background image in the first superimposed sample image is input into the first discriminator; based on the output result of the first discriminator and The comparison result between the first model generated image and the first superimposed sample image updates the initial image background repair model to obtain the image background repair model.
- Example 7 provides an image processing method, further comprising: the training process of the facial skin repair model, including: any collected image that does not contain a preset object Superimpose the preset object mask image in the head area of the sample image to obtain a second superimposed sample image; collect a preset number of head area anchor points in the head area of the second superimposed sample image according to the preset calibration point collection strategy to obtain A second overlay sample image marked with the preset number of head region anchor points; based on the second overlay sample image marked with the preset number of head region anchor points and the corresponding image that does not include the preset pair
- the sample images are trained on the neural network model to obtain the iftr facial skin repair model.
- Example 8 provides an image processing method, further comprising:
- the preset is based on the mark Perform neural network model training on second superimposed sample images of a number of skull region anchor points and corresponding sample images that do not contain preset objects to obtain the facial skin repair model, including: The second superimposed sample image of the regional anchor point is input into the initial facial skin repair model to obtain the second model generate an image; compare the second model generated image with any sample image that does not include a preset object except for any of the collected sample images that do not include a preset object corresponding to the second superimposed sample image.
- Example 9 provides an image processing method, further comprising: In an optional implementation, the second acquisition strategy is performed according to the preset calibration point acquisition strategy.
- Collecting a preset number of head area anchor points in the head area of the superimposed sample image includes: performing anchor point sampling according to the outline of the facial features inside the head area of the second superimposed sample image; Anchor point sampling is performed on the contour edge of the region according to a preset sampling interval.
- Example 10 provides an image processing device, which includes: an image acquisition module configured to acquire an image to be processed, wherein the image to be processed has a preset object The image, part of the pixel points of the preset object is located within the main body outline area in the image to be processed, and another part of the pixel points of the preset object is located outside the main body outline area; the image processing module is set to Input the image to be processed into the preset object removal processing model to obtain a target image, wherein the target image is a preset object removal image corresponding to the image with the preset object; the preset object removal processing The model is a model obtained by training based on a pre-established set of image sample pairs without preset objects, wherein each pair of image sample pairs without preset objects in the set of image sample pairs without preset objects includes an image sample pair with a preset object.
- Example 111 provides an image processing device, further including:
- the image processing device further includes a model training sample construction module, It is set to: identify the main body outline area showing the preset object in the original image with the preset object; process the pixel points of the preset object located in the main body outline area in the original image to be consistent with the The pixels in the original image that are not the pixels of the preset object within the main body outline area have consistent pixel information, and the pixels of the preset object that are outside the main body outline area in the original image are Process pixel points that are consistent with pixel information of pixel points that are not the preset object outside the body outline area in the original image to obtain a preset object-removed image; combine the original image and
- Example 12 provides an image processing device, further including:
- the model training sample construction module is set to: input the original image with the preset object into the head region prediction model to obtain a binary image of the head region showing the preset object;
- the binary image of the head region is superimposed with the original image to obtain an image overlay result, and the image overlay result is input to the image background repair model to obtain the pre-set image located outside the head region in the original image.
- the primary object removal image of elephant killing removal is input into the facial skin repair model to obtain the complete removal of the preset object located in the head area in the original image.
- Example 13 provides an image processing device, further including:
- the image processing device further includes a first auxiliary model training
- the module is configured to train the head region prediction model.
- the training process includes the following steps: Obtain any sample image with the preset object, and match the corresponding three-dimensional head model for any sample image;
- the head model performs planar projection to obtain a binary image of the head area that matches the any sample image; use the any sample image as a model input image, and use the two-value image of the head area that matches the any sample image.
- Example 14 provides an image processing device, further comprising: In an optional implementation, the image processing device further includes a second auxiliary model.
- the training module is configured to train the image background repair model.
- the training process includes the following steps: Randomly combine the head region binary image in the preset head region binary image set with the background image in the preset background image set to obtain a multi-dimensional combinations, and superimpose the binary image of the head region in each combination on the background image in each combination to obtain a first superimposed sample image; perform the operation outside the corresponding head region in the first superimposed sample image Preset object pixel marks to obtain a first superimposed sample marked image; perform neural network model training based on the first superimposed sample image and the first superimposed sample marked image to obtain the image background repair model.
- Example 15 provides an image processing device, further comprising:
- the second auxiliary model training module is configured to: Input the first superimposed sample mark image into the initial image background repair model to obtain a first model generated image; divide the first superimposed sample image from the first model generated image and the preset background image set Any background image other than the background image is input into the first discriminator; based on the output result of the first discriminator and the comparison result of the first model generated image and the first superimposed sample image
- the initial image background repair model is updated to obtain the image background repair model.
- Example 16 provides an image processing device, which also includes:
- the image processing device further includes a third auxiliary model training
- the training module is configured to train the facial skin repair model.
- the training process includes the following steps: superimpose the preset object mask image in the skull area of any collected sample image that does not contain the preset object to obtain a second superimposed sample.
- the facial skin repair model is obtained by performing neural network model training based on the second superimposed sample image marked with the preset number of head region anchor points and the corresponding sample image not containing the preset object.
- the third auxiliary model training module is configured To: input the second superimposed sample image marked with the preset number of head region anchor points into the initial facial skin repair model to obtain a second model generated image; divide the second model generated image with Any sample image that does not contain a preset object other than any of the collected sample images that does not contain a preset object corresponding to the second superimposed sample image is input into the second discriminator; based on the third The output result of the second discriminator and the second model generated (image and the comparison result of any of the collected sample images that do not contain the preset object when compared with the second superimposed sample image, update The initial facial skin repair model is used to obtain the facial skin repair model.
- Example 18 provides an image processing device, further comprising:
- the third auxiliary model training module may also be configured to: perform anchor point sampling according to the outline of the five palaces inside the skull area of the second superimposed sample image; Anchor point sampling is performed on the contour edge of the head region according to preset sampling intervals.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
Description
Claims
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US18/873,688 US20250371671A1 (en) | 2022-06-10 | 2023-06-01 | Method and apparatus for image processing, electronic device and storage medium |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210657612.2 | 2022-06-10 | ||
| CN202210657612.2A CN115937009A (zh) | 2022-06-10 | 2022-06-10 | 图像处理方法、装置、电子设备及存储介质 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2023239299A1 true WO2023239299A1 (zh) | 2023-12-14 |
Family
ID=86654759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/SG2023/050392 Ceased WO2023239299A1 (zh) | 2022-06-10 | 2023-06-01 | 图像处理方法、装置、电子设备及存储介质 |
Country Status (3)
| Country | Link |
|---|---|
| US (1) | US20250371671A1 (zh) |
| CN (1) | CN115937009A (zh) |
| WO (1) | WO2023239299A1 (zh) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116934714A (zh) * | 2023-07-20 | 2023-10-24 | 中科慧远视觉技术(洛阳)有限公司 | 图像处理方法、装置、设备及存储介质 |
| CN117315406B (zh) * | 2023-11-28 | 2024-02-13 | 吉咖智能机器人有限公司 | 一种样本图像处理方法、装置及设备 |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008832A (zh) * | 2019-02-27 | 2019-07-12 | 西安电子科技大学 | 基于深度学习人物图像自动分割方法、信息数据处理终端 |
| CN112215784A (zh) * | 2020-12-03 | 2021-01-12 | 江西博微新技术有限公司 | 图像去污方法、装置、可读存储介质及计算机设备 |
| CN112734633A (zh) * | 2021-01-07 | 2021-04-30 | 京东方科技集团股份有限公司 | 虚拟发型的替换方法、电子设备及存储介质 |
| CN113963409A (zh) * | 2021-10-25 | 2022-01-21 | 百果园技术(新加坡)有限公司 | 一种人脸属性编辑模型的训练以及人脸属性编辑方法 |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113344776B (zh) * | 2021-06-30 | 2023-06-27 | 北京字跳网络技术有限公司 | 图像处理方法、模型训练方法、装置、电子设备及介质 |
-
2022
- 2022-06-10 CN CN202210657612.2A patent/CN115937009A/zh active Pending
-
2023
- 2023-06-01 WO PCT/SG2023/050392 patent/WO2023239299A1/zh not_active Ceased
- 2023-06-01 US US18/873,688 patent/US20250371671A1/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110008832A (zh) * | 2019-02-27 | 2019-07-12 | 西安电子科技大学 | 基于深度学习人物图像自动分割方法、信息数据处理终端 |
| CN112215784A (zh) * | 2020-12-03 | 2021-01-12 | 江西博微新技术有限公司 | 图像去污方法、装置、可读存储介质及计算机设备 |
| CN112734633A (zh) * | 2021-01-07 | 2021-04-30 | 京东方科技集团股份有限公司 | 虚拟发型的替换方法、电子设备及存储介质 |
| CN113963409A (zh) * | 2021-10-25 | 2022-01-21 | 百果园技术(新加坡)有限公司 | 一种人脸属性编辑模型的训练以及人脸属性编辑方法 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115937009A (zh) | 2023-04-07 |
| US20250371671A1 (en) | 2025-12-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110058685B (zh) | 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质 | |
| CN113344776B (zh) | 图像处理方法、模型训练方法、装置、电子设备及介质 | |
| WO2022166872A1 (zh) | 一种特效展示方法、装置、设备及介质 | |
| CN114092678A (zh) | 图像处理方法、装置、电子设备及存储介质 | |
| CN113688928B (zh) | 图像匹配方法、装置、电子设备和计算机可读介质 | |
| WO2023239299A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
| CN114049417B (zh) | 虚拟角色图像的生成方法、装置、可读介质及电子设备 | |
| WO2023051244A1 (zh) | 图像生成方法、装置、设备及存储介质 | |
| CN115937356B (zh) | 图像处理方法、装置、设备及介质 | |
| WO2022132032A1 (zh) | 人像图像处理方法及装置 | |
| US20240386640A1 (en) | Method, apparatus, device and storage medium for generating character style profile image | |
| CN114863214A (zh) | 图像生成模型训练、图像生成方法、装置、介质及设备 | |
| WO2023143129A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
| US12526460B2 (en) | Video generation method and apparatus, and device and storage medium | |
| CN114418835B (zh) | 图像处理方法、装置、设备及介质 | |
| WO2023140787A2 (zh) | 视频的处理方法、装置、电子设备、存储介质和程序产品 | |
| WO2023143222A1 (zh) | 图像处理方法、装置、设备及存储介质 | |
| CN116596748A (zh) | 图像风格化处理方法、装置、设备、存储介质和程序产品 | |
| CN114841984B (zh) | 图像处理方法、装置、电子设备及存储介质 | |
| CN115661493A (zh) | 一种对象位姿的确定方法及装置、设备及存储介质 | |
| WO2020034981A1 (zh) | 编码信息的生成方法和识别方法 | |
| CN115311178A (zh) | 图像拼接方法、装置、设备及介质 | |
| WO2023239300A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
| CN115937338B (zh) | 图像处理方法、装置、设备及介质 | |
| CN118537206A (zh) | 一种模型构建方法、图像生成方法、装置、设备、介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23820194 Country of ref document: EP Kind code of ref document: A1 |
|
| WWE | Wipo information: entry into national phase |
Ref document number: 18873688 Country of ref document: US |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 23820194 Country of ref document: EP Kind code of ref document: A1 |
|
| WWP | Wipo information: published in national office |
Ref document number: 18873688 Country of ref document: US |

