WO2023273414A1 - 图像处理方法、装置、设备以及存储介质 - Google Patents

图像处理方法、装置、设备以及存储介质 Download PDF

Info

Publication number
WO2023273414A1
WO2023273414A1 PCT/CN2022/080677 CN2022080677W WO2023273414A1 WO 2023273414 A1 WO2023273414 A1 WO 2023273414A1 CN 2022080677 W CN2022080677 W CN 2022080677W WO 2023273414 A1 WO2023273414 A1 WO 2023273414A1
Authority
WO
WIPO (PCT)
Prior art keywords
rendering
space
real image
rendering material
physical object
Prior art date
Application number
PCT/CN2022/080677
Other languages
English (en)
French (fr)
Inventor
陶然
杨瑞健
赵代平
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2023273414A1 publication Critical patent/WO2023273414A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G06T15/205—Image-based rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation

Definitions

  • the present application relates to the field of computer technology, and in particular to an image processing method, device, equipment and storage medium.
  • Method 1 Culling by occluding planes.
  • the occlusion plane can be set in advance, and all the parts behind the occlusion plane in the rendering material are removed as the occluded parts.
  • Such a "one size fits all" culling method cannot simulate the real occlusion scene well, and the rendering effect is not good.
  • Method 2 Eliminate through depth testing.
  • special hardware is required, and the rejection cannot be realized for equipment without this special hardware, and the universality is poor.
  • the present application discloses an image processing method.
  • the method may include: acquiring a target area and a rendering material in a real image, the target area including a physical object having an occlusion relationship with the rendering material in the same space; projecting the rendering material onto the real image, and Determining the projection part in the target area, determining the part of the projection part behind the preset occlusion plane as the part blocked by the physical object in the rendering material;
  • the part blocked by the physical object is subjected to occlusion and culling processing, and the rendering material after occlusion and culling processing is used to perform rendering processing on the real image to obtain a rendered image.
  • the rendering material identifies a plurality of first key points
  • the real image includes a target rendering object to be rendered
  • the target rendering object identifies a key point that is identical to the plurality of first key points.
  • a corresponding plurality of second key points; the projecting the rendering material to the real image includes: obtaining the position mapping relationship between the plurality of first key points and the plurality of second key points ; According to the position mapping relationship, map the rendering material to a first space associated with the target rendering object, the first space includes a space obtained by performing three-dimensional modeling based on the real image; map to the The rendering material in the first space is projected onto the real image.
  • the method further includes: receiving configuration information for the occlusion plane, the configuration information including at least depth information and orientation information of the occlusion plane in the second space associated with the rendering material; based on The configuration information determines the preset occlusion plane.
  • the first space and the second space are different spaces
  • the method further includes: using each vertex included in the projected part as the current vertex; according to the origin of the coordinate system corresponding to the first space obtained by performing three-dimensional modeling based on the real image and the For the current vertex, determine a straight line passing through the preset occlusion plane mapped to the first space, and determine an intersection point where the line intersects the preset occlusion plane; in response to the first intersection point from the origin to the intersection point A distance is smaller than a second distance from the origin to the current vertex, determining that the position of the current vertex is behind the preset occlusion plane; in response to the first distance being greater than the second distance, determining that the current The position of the vertex is in front of the preset occlusion plane.
  • the occlusion and culling processing of the part of the rendering material that is occluded by the physical object includes at least one of the following methods: deleting the part of the rendering material that is occluded by the physical object; Transparency of the part blocked by the real object in the rendering material; modifying the pixel blending mode of the part blocked by the real object in the rendering material and the background part in the real image.
  • the acquiring the target area in the real image includes: using a segmentation network generated based on a neural network to segment the real image to obtain the target area; the method further includes: acquiring the target area including A training sample set of a plurality of training samples, the training samples include labeling information of the target area; the target area includes an area preset according to business requirements; based on the training sample set, the segmentation network is trained.
  • the rendering material includes a three-dimensional virtual human head; or, the target area includes a foreground area in the real image; or, the physical object includes a person's body; or, the target included in the real image
  • the rendering object is a real human head.
  • the present application also proposes an image processing device, the device may include: an acquisition module, configured to acquire a real image, a target area in the real image, and a rendering material, the target area includes A physical object that generates an occlusion relationship; a determination module, configured to project the rendering material onto the real image, and determine the projection part within the target area, and place the part of the projection part behind the preset occlusion plane The part is determined as the part of the rendering material that is blocked by the physical object; the rendering module is configured to perform occlusion and culling processing on the part of the rendering material that is occluded by the physical object, and use the rendering after occlusion and culling processing The material renders the real image to obtain a rendered image.
  • an acquisition module configured to acquire a real image, a target area in the real image, and a rendering material, the target area includes A physical object that generates an occlusion relationship
  • a determination module configured to project the rendering material onto the real image, and determine the projection part within the target area, and
  • the rendering material identifies a plurality of first key points
  • the real image includes a target rendering object to be rendered
  • the target rendering object identifies a key point that is identical to the plurality of first key points.
  • a corresponding plurality of second key points; the determination module is specifically configured to: obtain the position mapping relationship between the plurality of first key points and the plurality of second key points; according to the position mapping relationship , mapping the rendering material to a first space associated with the target rendering object, the first space including a space obtained by performing three-dimensional modeling based on the real image; mapping the rendering material to the first space
  • the rendered material is projected onto the reality image.
  • the apparatus further includes: a configuration module, configured to receive configuration information for the occlusion plane, where the configuration information includes at least depth information of the occlusion plane in a second space associated with the rendering material and orientation information; based on the configuration information, determine the preset occlusion plane.
  • a configuration module configured to receive configuration information for the occlusion plane, where the configuration information includes at least depth information of the occlusion plane in a second space associated with the rendering material and orientation information; based on the configuration information, determine the preset occlusion plane.
  • the first space and the second space are different spaces
  • the determining module is specifically configured to: map the preset occlusion plane to the first space; A part of the part whose position is behind the preset occlusion plane mapped to the first space is determined as an occluded part of the rendering material.
  • the device further includes: using each vertex included in the projection part as the current vertex; according to the origin of the coordinate system corresponding to the first space obtained by performing three-dimensional modeling based on the real image and the For the current vertex, determine a straight line passing through the preset occlusion plane mapped to the first space, and determine an intersection point where the line intersects the preset occlusion plane; in response to the first intersection point from the origin to the intersection point A distance is smaller than a second distance from the origin to the current vertex, determining that the position of the current vertex is behind the preset occlusion plane; in response to the first distance being greater than the second distance, determining that the current The position of the vertex is in front of the preset occlusion plane.
  • the occlusion and culling processing of the part of the rendering material that is occluded by the physical object includes at least one of the following methods: deleting the part of the rendering material that is occluded by the physical object; adjusting The transparency of the part of the rendering material that is blocked by the physical object; modifying the pixel blending mode between the part of the rendering material that is blocked by the physical object and the background part of the real image.
  • the acquisition module is specifically configured to: use a segmentation network generated based on a neural network to segment the real image to obtain the target area; the device further includes: a training module, configured to Obtain a training sample set including a plurality of training samples, the training sample includes label information of a target area; the target area includes a pre-set area according to business requirements; based on the training sample set, the segmentation network is trained .
  • the rendering material includes a three-dimensional virtual human head; or, the target area includes a foreground area in the real image; or, the physical object includes a person's body; or, the target included in the real image
  • the rendering object is a real human head.
  • the present application also proposes an electronic device, the device comprising: a processor; a memory for storing processor-executable instructions; wherein, the processor executes the executable instructions to implement the electronic device as shown in any of the foregoing embodiments image processing method.
  • the present application also proposes a computer-readable storage medium, where the storage medium stores a computer program, and the computer program is used to cause a processor to execute the image processing method as shown in any one of the foregoing embodiments.
  • the target area of the physical object that has an occlusion relationship with the rendering material in the same space in the real image can be obtained; then the rendering material is projected onto the real image, and determined to be within the target area Afterwards, the part of the projected part whose position is behind the preset occlusion plane is determined as the part occluded by the physical object in the rendering material, and subsequent occlusion culling processing and image rendering processing are performed.
  • the part behind the preset occlusion plane in the target area containing the physical object that may have an occlusion relationship with the rendering material can be determined as the part occluded by the physical object, so that Frame the part of the rendering material that may be occluded by the target area, so that the part of the rendering material that cannot be occluded with the physical object will not be determined as the occluded part.
  • it can be accurately determined. Render the occluded part of the material to improve the occlusion rendering effect.
  • Fig. 1 is a flow chart of an image processing method shown in the present application
  • Fig. 2 is the flowchart of a kind of segmentation network training method shown in the present application
  • FIG. 3 is a schematic flowchart of a rendering material space mapping method shown in the present application.
  • FIG. 4 is a schematic flowchart of a method for configuring a preset occlusion plane shown in the present application
  • FIG. 5 is a schematic flowchart of a method for determining an obscured part shown in the present application
  • FIG. 6A is a schematic diagram of a virtual human head shown in this application.
  • FIG. 6B is a schematic diagram of a live image shown in this application.
  • FIG. 6C is a schematic diagram of a character foreground area shown in this application.
  • FIG. 6D is a schematic diagram of a rendered image shown in the present application.
  • FIG. 7 is a flowchart of an image rendering method shown in the present application.
  • FIG. 8 is a schematic structural diagram of an image processing device shown in the present application.
  • FIG. 9 is a schematic diagram of a hardware structure of an electronic device shown in the present application.
  • the present application proposes an image processing method.
  • the method can acquire a target area in a real image that includes a physical object that has an occlusion relationship with the rendering material in the same space; then project the rendering material onto the real image, and determine a projected part within the target area; then Determining the part of the projection part that is behind the preset occlusion plane as the part of the rendering material that is occluded by the physical object, and performing subsequent occlusion culling processing and image rendering processing.
  • the part behind the preset occlusion plane in the target area containing the physical object that may have an occlusion relationship with the rendering material can be determined as the part occluded by the physical object, so that Frame the part of the rendering material that may be occluded by the target area, so that the part of the rendering material that cannot be occluded with the physical object will not be determined as the occluded part.
  • it can be accurately determined. Render the occluded part of the material to improve the occlusion rendering effect.
  • FIG. 1 is a flow chart of an image processing method shown in this application.
  • the method may include:
  • the method can be applied to electronic equipment.
  • the electronic device may implement the method by carrying a software device corresponding to the image processing method.
  • the type of the electronic device may be a notebook computer, a computer, a server, a mobile phone, a PAD terminal and the like.
  • the present application does not specifically limit the specific type of the electronic device.
  • the electronic device may be a device on the client side or on the server side.
  • the server may be a server or a cloud provided by a server, a server cluster or a distributed server cluster.
  • an electronic device (hereinafter referred to as device) is taken as an example for description.
  • the user can transmit the real image to the device through the client program provided by the device.
  • the real-world images may include real-world images captured for the real world.
  • the real image may include images collected for objects such as people, vehicles, and houses.
  • the real image may also be an image collected by the user through the image collection hardware mounted on the device.
  • the device may be a mobile phone terminal
  • the image acquisition hardware may be a camera mounted on the mobile phone terminal. Users can collect real images through the camera.
  • the device may execute S102.
  • the rendering materials described in this application specifically refer to materials for rendering real images.
  • the rendering material may be two-dimensional or three-dimensional material.
  • the following uses a 3D material as an example for description.
  • the rendering material may be some virtual props.
  • the rendering material in a scene where a virtual human head is used to perform image rendering on a human face in a character image, the rendering material may be a virtual human head.
  • the rendering material in a scene where a virtual animal head is used to perform image rendering on a human face in a character image, the rendering material may be a virtual animal head.
  • the rendering material can render the target rendering object in the real image.
  • the target rendering object may refer to an object to be rendered in a real image. During the rendering process, the rendering material will replace the target rendering object and appear in the rendered image after rendering.
  • the target rendering object may be a real human head.
  • Virtual human heads or virtual animal heads can replace real human heads in rendered images.
  • the target area described in the present application may refer to an area including a physical object in a real image.
  • the physical object refers to an object that has an occlusion relationship with the rendering material in the same space.
  • the hair of the virtual human head may be obscured by the body of the person in the real image.
  • the foreground area including the person's body (physical object) may be determined as the target area.
  • the part of the hair part of the virtual human head projected into the target area can be determined, and the part behind the preset occlusion plane in this part can be determined as the part occluded by the body of the character.
  • the part projected in the target area containing the real object and behind the preset occlusion plane is determined as the part blocked by the real object, and the part of the rendered material that may be blocked can be framed by the target area, so that it is impossible for the rendered material to be occluded with the real object
  • the part of the object where the occlusion relationship occurs will not be determined as the occluded part. Therefore, on the one hand, the efficiency of occluded part confirmation is improved.
  • the occluded part in the rendered material can be accurately determined, thereby improving the occlusion rendering effect.
  • the real image when performing S102, may be segmented using a segmentation network generated based on a neural network to obtain the target area.
  • the segmentation network may be a pixel-level segmentation network generated based on a neural network or deep learning.
  • the output result of the pixel-level segmentation network may include binary classification for each pixel included in the input image to distinguish whether the pixel is in the target area.
  • the present application does not specifically limit the network structure of the segmented network.
  • the segmentation network may be trained using multiple training samples including label information of the target region.
  • FIG. 2 is a flowchart of a segmentation network training method shown in the present application.
  • S202 may be executed first to acquire a training sample set including a plurality of training samples.
  • the training samples included in the training sample set may include label information of the target area.
  • the target area includes an area preset according to business requirements.
  • the target area in the image can be determined according to business requirements. Then mark the target area on some of the acquired images to obtain training samples.
  • an area including a body object in the person image may be determined as a target area.
  • the target area may be a foreground area of a person image.
  • the foreground region can usually cover a complete person, that is, the entire body object.
  • each pixel in the acquired person image can be marked to determine whether each pixel is in the target area, and the mark of the person image is completed to obtain a training sample.
  • S204 may be executed to train the segmentation network based on the training sample set. After the training is completed, the segmentation network can be used to segment the real image to obtain the target area.
  • the target area can be flexibly determined according to business requirements, so that the segmentation network trained by using these training samples can obtain target areas that meet business requirements from real images, thereby improving rendering flexibility.
  • the rendering materials described in this application are maintained in the second space.
  • the second space may include a three-dimensional space in which the material is generated, represented by a second coordinate system.
  • the device may store the position information of each vertex included in the rendered material in the second coordinate system, so as to store the material.
  • performing three-dimensional modeling on the real image can obtain the first space.
  • the first space can be understood as the imaging three-dimensional space of the real image.
  • a monocular camera or a binocular camera method may be used to perform three-dimensional modeling on real images based on device parameters of image acquisition hardware to obtain the first space.
  • the first space is represented by a first coordinate system.
  • first space and the second space may be the same space or different spaces. If the first space and the second space are the same space, S104 may be directly executed. If the first space and the second space are different spaces, when performing S104, the rendering material may first be mapped into the first space.
  • the rendering material may be identified with multiple first key points.
  • the multiple first key points may be multiple key points on the outline of the virtual prop.
  • the plurality of first key points may be key vertices at preset positions such as the top of the head, ears, nose tip, and face. Wherein, the position corresponding to the first key point in the second space is the first position.
  • the real image includes target rendering objects to be rendered.
  • the rendering material will replace the target rendering object and appear in the rendered image.
  • the target rendering object is identified with a plurality of second key points corresponding one-to-one to the plurality of first key points.
  • the plurality of first key points and the plurality of second key points may be points corresponding to positions on the contour.
  • the plurality of second key points can be the top of the head of the real person in the real image, the ears, the tip of the nose, The key vertices of the corresponding position such as the face.
  • the corresponding position of the second key point in the real image is the second position.
  • FIG. 3 is a schematic flowchart of a rendering material space mapping method shown in the present application.
  • S302 may be executed to obtain the position mapping relationship between the plurality of first key points and the plurality of second key points.
  • a mapping relationship solving algorithm can be used, using the first position of multiple first key points in the second space, the second position of multiple second key points in the first space, and the image of the real image collected Information such as hardware parameters is collected to obtain a mapping relationship that maps vertices in the second space to the first space.
  • the algorithm for solving the mapping relationship may include a PNP (Perspective-N-Point, multi-point perspective imaging) algorithm.
  • the position mapping relationship may include a translation amount and a rotation amount of the same pixel when it is mapped from the second space to the first space. This application does not specifically limit the algorithm for solving the mapping relationship.
  • S304 may be executed to map the rendering material to the first space associated with the target rendering object according to the position mapping relationship.
  • the relationship can be used to map each vertex included in the rendering material to the first space. Therefore, on the one hand, the material and the real object in the real image can be placed in the same three-dimensional space, which is convenient for occlusion judgment, image rendering and other processing. On the other hand, the rendering material can be closer to the real position and orientation of the target rendering object. Detail state to improve the rendering effect.
  • S306 may be executed to project the rendering material mapped into the first space onto the real image.
  • the three-dimensional coordinate position of the rendering material in the first space may be transformed into a two-dimensional coordinate position to realize projection onto the real image.
  • S308 is executed to determine the projected part within the target area.
  • vertices whose projection positions in the real image are within the target area may be determined, and then the three-dimensional parts corresponding to these vertices in the first space may be determined as the projected parts.
  • the projection part may be understood as a part in the rendering material that may have an occlusion relationship with physical objects in the target area.
  • S104 may be continued to determine a part of the projection part that is behind a preset occlusion plane as a part of the rendering material that is blocked by the physical object.
  • the preset occlusion plane may be a plane preset according to business requirements.
  • the part of the rendered material behind the preset occlusion plane may be occluded by physical objects in reality.
  • different occlusion planes may be set.
  • the body plane can be used as the preset occlusion plane.
  • the body plane may refer to the front surface of the body.
  • a preset occlusion plane can be configured.
  • FIG. 4 is a schematic flowchart of a method for configuring a preset occlusion plane shown in the present application.
  • S402 may be executed first to receive configuration information for the occlusion plane.
  • the user for example, a technician
  • the device may receive the configuration information.
  • the configuration information may at least include depth and orientation information of the occlusion plane in the second space associated with the rendering material.
  • the depth and orientation information can be used to indicate the position information of the plane.
  • the depth may indicate the distance from the origin of the second coordinate system corresponding to the second space to the preset occlusion plane.
  • the orientation may indicate an angle of a normal vector of the preset occlusion plane.
  • the front surface of the body can be set as the preset occlusion plane, and the user can set the depth of the front surface of the body in the second space and Pack the orientation information into the configuration information, and pass the configuration information to the device through the interface provided by the device.
  • the device may execute S404 to determine the preset occlusion plane based on the configuration information.
  • the device can use the on-board image processing software to complete the configuration of the occlusion plane.
  • the first space and the second space are different spaces.
  • the preset occlusion plane in the second space may be mapped to the first space, so that the occlusion plane It is placed in the same three-dimensional space as the rendering material for occlusion judgment.
  • FIG. 5 is a schematic flowchart of a method for determining an obscured part shown in the present application.
  • S502 may be executed, and each vertex included in the projection part is used as a current vertex. In this step, each vertex can be used as the current vertex according to a preset order.
  • S504 can be executed, according to the origin of the coordinate system corresponding to the first space obtained by performing three-dimensional modeling based on the real image and the current vertex, determine the distance passing through the preset occlusion plane mapped to the first space a straight line, and determine an intersection point where the straight line intersects the preset occlusion plane.
  • a straight line may be determined by using the origin and the current vertex, and the intersection point of the straight line and the occlusion plane may be determined as the intersection point.
  • the first distance from the origin to the intersection can be compared with the second distance from the origin to the current vertex, and S506 and S508 are executed, in response to the first distance from the origin to the intersection If it is less than a second distance from the origin to the current vertex, it is determined that the position of the current vertex is behind the preset occlusion plane. In response to the first distance being greater than the second distance, it is determined that the position of the current vertex is in front of the preset occlusion plane.
  • the position of the current vertex is located at the preset Before the occlusion plane, when the first distance is greater than the second distance, the position of the current vertex is behind the preset occlusion plane.
  • S106 may be executed.
  • At least one of the following methods may be used to perform occlusion culling processing:
  • the deleting the part of the rendering material that is blocked by the physical object may include deleting the vertices of the part of the rendering material that are blocked by the physical object and the pixels corresponding to the vertices by running a pre-written occlusion removal program, so as not to render Display the part in the image to achieve the effect of removing the part.
  • the adjusting the transparency of the part of the rendering material that is blocked by the physical object may include adjusting the pixel value of the vertex of the occluded part of the rendering material by running a pre-written occlusion removal program, so that the transparency of the occluded part Large enough that the part will not show up in the rendered image to achieve a culling effect on the part.
  • the modifying the pixel blending mode of the part of the rendered material occluded by the physical object and the background part of the real image may include modifying the pixel blending mode by running a pre-written occlusion removal program.
  • the real image when performing S106, may be rendered by using the rendering material in a rasterized rendering manner.
  • any of the aforementioned occlusion removal processing methods can be used to remove the part of the rendering material that is blocked by the physical object, so that the rendered image can show a rendering effect that matches the occlusion relationship in the real scene.
  • the target area of the physical object that has an occlusion relationship with the rendering material in the same space in the real image can be obtained; then the rendering material is projected onto the real image, and determined to be in the target area Then determine the part of the projection part that is behind the preset occlusion plane as the part of the rendering material that is occluded by the physical object, and perform subsequent occlusion culling processing and image rendering processing.
  • the part behind the preset occlusion plane in the target area containing the physical object that may have an occlusion relationship with the rendering material can be determined as the part occluded by the physical object, so that Frame the part of the rendering material that may be occluded by the target area, so that the part of the rendering material that cannot be occluded with the physical object will not be determined as the occluded part.
  • it can be accurately determined. Render the occluded part of the material to improve the occlusion rendering effect.
  • Embodiments are described below in conjunction with live broadcast scenarios.
  • the virtual head provided in the virtual prop library can be used to render the collected live images in real time.
  • the live broadcast client used in the live broadcast process can be carried in the mobile terminal.
  • the mobile terminal can be equipped with an ordinary camera (the camera is not required to have a depth test function) for real-time collection of live images.
  • the virtual item library can be installed locally on the mobile terminal (hereinafter referred to as the terminal), or in a server corresponding to the live broadcast client (hereinafter referred to as the client).
  • the developer configures an occlusion plane for the virtual human head in advance in the virtual prop library, and configures the category of the part of the virtual human head that can be occluded as hair.
  • FIG. 6A is a schematic diagram of a virtual human head shown in this application. Assume that the virtual human head selected by the user is as shown in FIG. 6A .
  • FIG. 7 is a flowchart of an image rendering method shown in this application.
  • the terminal may execute S71 to receive live images captured by the camera in real time.
  • FIG. 6B is a schematic diagram of a live image shown in this application. Assume that the collected live image is as shown in FIG. 6B .
  • S72 can be executed, using the pre-trained character foreground region segmentation network to obtain the character foreground region of the live image (that is, the target region in the foregoing embodiment).
  • This area may include the body part of the character, which may block the hair of the virtual human head.
  • the foreground area of the character on the one hand, the scope of occlusion judgment can be narrowed, and the efficiency of occluded part confirmation can be improved.
  • it can be obtained by The foreground area of the character frames the part of the virtual head that may be occluded, so that the part of the virtual head outside the foreground area of the character will not be determined as the occluded part. Compared with the "one size fits all" approach, the rendering can be accurately determined. The occluded part of the material, thereby improving the occlusion rendering effect.
  • FIG. 6C is a schematic diagram of a foreground area of a person shown in the present application. Assume that the foreground area of the character obtained after S72 is shown in Figure 6C
  • S73 can be executed to collect the two-dimensional coordinates of multiple second key points on the real human head in the live image, and use the parameters of the camera and the three-dimensional coordinates of the multiple corresponding first key points on the virtual human head to convert the user-selected
  • the virtual human head and the occlusion plane corresponding to the virtual human head are mapped in the imaging space formed by the camera (ie, the aforementioned first space). Therefore, on the one hand, the material can be placed in the same three-dimensional space as the human body object (physical object) in the real image, which is convenient for occlusion judgment, image rendering and other processing; on the other hand, the rendering material can be closer to the real human head (object) Rendering object) the real position, orientation and other details to improve the rendering effect.
  • S74 may be executed to project the head of the virtual person onto the live image, and determine the projected part in the foreground area of the person that may be blocked by the body.
  • S75 can be executed to determine the hair part of the virtual human head that is blocked by the body according to the positional relationship between the projected part and the occlusion plane.
  • the part of the virtual human head that is in the foreground area of the character and behind the occlusion plane can be determined as the hair part occluded by the body, so that the accurate occluded part that conforms to the real scene can be determined, and the rendering effect can be improved. reality.
  • S76 may be executed to remove the hair part blocked by the body during the rasterized rendering of the live image to obtain a more realistic rendered image.
  • FIG. 6D is a schematic diagram of a rendered image shown in this application. After the steps of S71-S76, the rendered image as shown in FIG. 6D can be obtained. In this way, a rendering effect that matches the real occlusion relationship between the hair and the body can be obtained.
  • the present application proposes an image processing device 80 .
  • FIG. 8 is a schematic structural diagram of an image processing device shown in this application.
  • the device 80 may include: an acquisition module 81, configured to acquire a target area and a rendering material in a real image, the target area including a physical object that has an occlusion relationship with the rendering material in the same space; A determining module 82, configured to project the rendering material onto the real image, and determine a projected part within the target area, and determine a part of the projected part behind a preset occlusion plane as the Render the part of the rendering material that is blocked by the physical object; the rendering module 83 is configured to perform occlusion and culling processing on the part of the rendering material that is occluded by the physical object, and use the occlusion and culling of the rendered material to render the actual The image is rendered and processed to obtain a rendered image.
  • an acquisition module 81 configured to acquire a target area and a rendering material in a real image, the target area including a physical object that has an occlusion relationship with the rendering material in the same space
  • a determining module 82 configured to project the rendering material onto the real image
  • the rendering material identifies a plurality of first key points
  • the real image includes a target rendering object to be rendered
  • the target rendering object identifies a key point that is identical to the plurality of first key points.
  • a corresponding plurality of second key points; the determination module 82 is specifically configured to: obtain the position mapping relationship between the plurality of first key points and the plurality of second key points; according to the position mapping relationship, mapping the rendering material to a first space associated with the target rendering object, where the first space includes a space obtained by performing three-dimensional modeling based on the real image; mapping all the objects mapped to the first space
  • the rendering material is projected onto the real image.
  • the apparatus 80 further includes: a configuration module, configured to receive configuration information for the occlusion plane, the configuration information at least including the depth of the occlusion plane in the second space associated with the rendering material information and orientation information; based on the configuration information, determine the preset occlusion plane.
  • a configuration module configured to receive configuration information for the occlusion plane, the configuration information at least including the depth of the occlusion plane in the second space associated with the rendering material information and orientation information; based on the configuration information, determine the preset occlusion plane.
  • the first space and the second space are different spaces
  • the determining module 82 is specifically configured to: map the preset occlusion plane to the first space; A part of the projection part whose position is behind the preset occlusion plane mapped to the first space is determined as the occluded part of the rendering material.
  • the device 80 further includes: taking each vertex included in the projection part as the current vertex; the current vertex, determine a straight line passing through the preset occlusion plane mapped to the first space, and determine an intersection point where the line intersects the preset occlusion plane; in response to the intersection point from the origin to the intersection point The first distance is smaller than the second distance from the origin to the current vertex, determining that the position of the current vertex is behind the preset occlusion plane; in response to the first distance being greater than the second distance, determining the The position of the current vertex is in front of the preset occlusion plane.
  • the occlusion and culling processing of the part of the rendering material that is occluded by the physical object includes at least one of the following methods: deleting the part of the rendering material that is occluded by the physical object; adjusting The transparency of the part of the rendering material that is blocked by the physical object; modifying the pixel blending mode between the part of the rendering material that is blocked by the physical object and the background part of the real image.
  • the acquisition module 81 is specifically configured to: use a segmentation network generated based on a neural network to segment the real image to obtain the target area; the device 80 also includes: a training module, It is used to obtain a training sample set including a plurality of training samples, the training sample includes label information of a target area; the target area includes an area preset according to business requirements; based on the training sample set, the segmentation network to train.
  • the rendering material includes a three-dimensional virtual human head; or, the target area includes a foreground area in the real image; or, the physical object includes a person's body; or, the target included in the real image
  • the rendering object is a real human head.
  • Embodiments of the image processing apparatus shown in this application can be applied to electronic equipment.
  • the present application discloses an electronic device, which may include: a processor, and a memory for storing instructions executable by the processor.
  • the processor is configured to call executable instructions stored in the memory to implement the image processing method shown in any one of the foregoing embodiments.
  • FIG. 9 is a schematic diagram of a hardware structure of an electronic device shown in this application.
  • the electronic device may include a processor 901 for executing instructions, a network interface 902 for network connection, a memory 903 for storing operating data for the processor, and a storage state switching device corresponding to Non-volatile memory 904 for instructions.
  • the embodiment of the device may be implemented by software, or by hardware or a combination of software and hardware.
  • software implementation as an example, as a device in a logical sense, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory for operation by the processor of the electronic device where it is located.
  • the electronic device where the device in the embodiment is usually based on the actual function of the electronic device can also include other Hardware, no more details on this.
  • the corresponding instructions of the image processing device may also be directly stored in the memory, which is not limited herein.
  • the present application proposes a computer-readable storage medium, the storage medium stores a computer program, and the computer program can be used to cause a processor to execute the image processing method shown in any one of the foregoing embodiments.
  • one or more embodiments of the present application may be provided as a method, system or computer program product. Accordingly, one or more embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) having computer-usable program code embodied therein.
  • each embodiment in the present application is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments.
  • the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.
  • Embodiments of the subject matter and functional operations described in this application can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this application and their structural equivalents, or in A combination of one or more of .
  • Embodiments of the subject matter described in this application can be implemented as one or more computer programs, i.e. one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of data processing apparatus. Multiple modules.
  • the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for transmission by the data
  • the processing means executes.
  • a computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
  • the processes and logic flows described in this application can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing system.
  • a central processing system will receive instructions and data from read only memory and/or random access memory.
  • the basic components of a computer include a central processing system for implementing or executing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic or magneto-optical disks, or optical disks, to receive data therefrom or to It transmits data, or both.
  • mass storage devices for storing data, such as magnetic or magneto-optical disks, or optical disks, to receive data therefrom or to It transmits data, or both.
  • a computer is not required to have such a device.
  • a computer may be embedded in another device such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a device such as a Universal Serial Bus (USB) ) portable storage devices like flash drives, to name a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB Universal Serial Bus
  • Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disk), magneto-optical disk, and 0xCD_00ROM and DVD-ROM disks.
  • semiconductor memory devices such as EPROM, EEPROM, and flash memory devices
  • magnetic disks such as internal hard disks or removable disk
  • magneto-optical disk and 0xCD_00ROM and DVD-ROM disks.
  • the processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geometry (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Generation (AREA)

Abstract

本申请提出一种图像处理方法、装置、设备以及存储介质。其中，所述方法可以包括：获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象。将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分。对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。

Description

图像处理方法、装置、设备以及存储介质
相关申请的交叉引用
本公开要求于2021年6月30日提交的、申请号为202110739170.1的中国专利申请的优先权，该中国专利申请公开的全部内容以引用的方式并入本文中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种图像处理方法、装置、设备以及存储介质。
背景技术
在增强现实领域中，需要将渲染素材与现实图像进行融合完成渲染。为了提升渲染效果，在渲染过程中，需要将渲染素材中，被现实图像中的实物对象遮挡的部分进行剔除，以使得渲染效果尽量真实。
目前通常采用以下两种方式进行被遮挡部分的剔除。
方式一：通过遮挡平面进行剔除。方式一中可以预先设置遮挡平面，将渲染素材中处于遮挡平面之后的部分全部作为被遮挡部分进行剔除。如此“一刀切”式的剔除方法，不能很好的模拟真实的遮挡场景，渲染效果欠佳。
方式二：通过深度测试的方式进行剔除。方式二中需要利用可以预测深度的专用硬件进行深度测试，然后利用渲染素材与所述实物对象的深度关系进行渲染素材被遮挡的部分的判断，并对确定的被遮挡的部分进行剔除。如此需要利用专用硬件，对于不含该专用硬件的设备则无法实现剔除，普适性较差。
发明内容
有鉴于此，本申请公开一种图像处理方法。所述方法可以包括：获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象；将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分；对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
在一些实施例中，所述渲染素材标识有多个第一关键点，所述现实图像内包括待渲染的目标渲染对象，所述目标渲染对象中标识有与所述多个第一关键点一一对应的多个第二关键点；所述将所述渲染素材投影至所述现实图像，包括：获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系；根据所述位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间，所述第一空间包括基于所述现实图像进行三维建模得到的空间；将映射至所述第一空间的所述渲染素材投影至所述现实图像。
在一些实施例中，所述方法还包括：接收针对遮挡平面的配置信息，所述配置信息至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度信息和朝向信息；基于所述配置信息，确定所述预设遮挡平面。
在一些实施例中，所述第一空间与所述第二空间为不同的空间，所述将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的 部分，包括：将所述预设遮挡平面映射至所述第一空间；将所述投影部分中位置处于映射至所述第一空间的所述预设遮挡平面之后的部分，确定为所述渲染素材的被遮挡部分。
在一些实施例中，所述方法还包括：将所述投影部分包括的各顶点分别作为当前顶点；根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与所述预设遮挡平面相交的交点；响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后；响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之前。
在一些实施例中，所述对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理包括以下至少一项方式：删除所述渲染素材中被所述实物对象遮挡的部分；调整所述渲染素材中被所述实物对象遮挡的部分的透明度；修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式。
在一些实施例中，所述获取现实图像中的目标区域，包括：利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域；所述方法还包括：获取包括多个训练样本的训练样本集，所述训练样本包括目标区域的标注信息；所述目标区域包括根据业务需求预先设定的区域；基于所述训练样本集，对所述分割网络进行训练。
在一些实施例中，所述渲染素材包括三维虚拟人头；或者，所述目标区域包括所述现实图像中的前景区域；或者，所述实物对象包括人物身体；或者，所述现实图像包括的目标渲染对象为真实人头。
本申请还提出一种图像处理装置，所述装置可以包括：获取模块，用于获取现实图像、所述现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象；确定模块，用于将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分；渲染模块，用于对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
在一些实施例中，所述渲染素材标识有多个第一关键点，所述现实图像内包括待渲染的目标渲染对象，所述目标渲染对象中标识有与所述多个第一关键点一一对应的多个第二关键点；所述确定模块，具体用于：获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系；根据所述位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间，所述第一空间包括基于所述现实图像进行三维建模得到的空间；将映射至所述第一空间的所述渲染素材投影至所述现实图像。
在一些实施例中，所述装置还包括：配置模块，用于接收针对遮挡平面的配置信息，所述配置信息至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度信息和朝向信息；基于所述配置信息，确定所述预设遮挡平面。
在一些实施例中，所述第一空间与所述第二空间为不同的空间，所述确定模块，具体用于：将所述预设遮挡平面映射至所述第一空间；将所述投影部分中位置处于映射至所述第一空间的所述预设遮挡平面之后的部分，确定为所述渲染素材的被遮挡部分。
在一些实施例中，所述装置还包括：将所述投影部分包括的各顶点分别作为当前顶点；根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与 所述预设遮挡平面相交的交点；响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后；响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之前。
在一些实施例中，所述对所述渲染素材中对被所述实物对象遮挡的部分进行遮挡剔除处理包括以下至少一项方式：删除所述渲染素材中被所述实物对象遮挡的部分；调整所述渲染素材中被所述实物对象遮挡的部分的透明度；修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式。
在一些实施例中，所述获取模块，具体用于：利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域；所述装置还包括：训练模块，用于获取包括多个训练样本的训练样本集，所述训练样本包括目标区域的标注信息；所述目标区域包括根据业务需求预先设定的区域；基于所述训练样本集，对所述分割网络进行训练。
在一些实施例中，所述渲染素材包括三维虚拟人头；或者，所述目标区域包括所述现实图像中的前景区域；或者，所述实物对象包括人物身体；或者，所述现实图像包括的目标渲染对象为真实人头。
本申请还提出一种电子设备，所述设备包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器通过运行所述可执行指令以实现如前述任一实施例示出的图像处理方法。
本申请还提出一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序用于使处理器执行如前述任一实施例示出的图像处理方法。
在前述公开的技术方案中，可以获取现实图像中与渲染素材在同一空间产生遮挡关系的实物对象的目标区域；然后将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分；之后将所述投影部分中，位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分，并进行后续遮挡剔除处理与图像渲染处理。
一方面，在确定渲染素材被遮挡部分的过程中，可以无需利用专用硬件设备进行深度测试，从而可以在普通的设备中进行遮挡渲染，提升渲染普适性；
另一方面，在确定被遮挡部分的过程中，可以将包含可能与渲染素材发生遮挡关系的实物对象的目标区域内、处于预设遮挡平面之后的部分确定为被实物对象遮挡的部分，从而可以通过目标区域框定渲染素材可能被遮挡的部分，使渲染素材中不可能与实物对象发生遮挡关系的部分不会被确定为被遮挡部分，与“一刀切”式的提出方式相比，可以准确地确定渲染素材中的被遮挡部分，进而提升遮挡渲染效果。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
为了更清楚地说明本申请一个或多个实施例的技术方案，下面将附图作简单地介绍，显而易见地，下面描述中的附图仅仅具有示例作用，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请示出的一种图像处理方法的流程图；
图2为本申请示出的一种分割网络训练方法的流程图；
图3为本申请示出的一种渲染素材空间映射方法的流程示意图；
图4为本申请示出的一种预设遮挡平面配置方法的流程示意图；
图5为本申请示出的一种被遮挡部分确定方法的流程示意图；
图6A为本申请示出的一种虚拟人头的示意图；
图6B为本申请示出的一种直播图像的示意图；
图6C为本申请示出的一种人物前景区域的示意图；
图6D为本申请示出的一种渲染图像的示意图；
图7为本申请示出的一种图像渲染方法流程图；
图8为本申请示出的一种图像处理装置的结构示意图；
图9为本申请示出的一种电子设备的硬件结构示意图。
具体实施方式
下面将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的设备和方法的例子。
在本申请使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。还应当理解，本文中所使用的词语“如果”，取决于语境，可以被解释成为“在……时”或“当……时”或“响应于确定”。
有鉴于此，本申请提出一种图像处理方法。该方法可以获取现实图像中包括与渲染素材在同一空间产生遮挡关系的实物对象的目标区域；然后将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分；之后将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分，并进行后续遮挡剔除处理与图像渲染处理。
一方面，在确定渲染素材被遮挡部分的过程中，可以无需利用专用硬件设备进行深度测试，从而可以在普通的设备中进行遮挡渲染，提升渲染普适性；
另一方面，在确定被遮挡部分的过程中，可以将包含可能与渲染素材发生遮挡关系的实物对象的目标区域内处于预设遮挡平面之后的部分，确定为被实物对象遮挡的部分，从而可以通过目标区域框定渲染素材可能被遮挡的部分，使渲染素材中不可能与实物对象发生遮挡关系的部分不会被确定为被遮挡部分，与“一刀切”式的提出方式相比，可以准确地确定渲染素材中的被遮挡部分，进而提升遮挡渲染效果。
请参见图1，图1为本申请示出的一种图像处理方法的流程图。
如图1所示，所述方法可以包括：
S102，获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象。
S104，将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部 分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分。
S106，对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
该方法可以应用于电子设备中。其中，所述电子设备可以通过搭载与图像处理方法对应的软件装置执行所述方法。所述电子设备的类型可以是笔记本电脑，计算机，服务器，手机，PAD终端等。本申请不对所述电子设备的具体类型进行特别限定。所述电子设备可以是客户端或服务端一侧的设备。所述服务端可以是由服务器、服务器集群或分布式服务器集群提供的服务端或云端。以下以执行主体为电子设备(以下简称设备)为例进行说明。
在一些实施例中，用户可以通过所述设备提供的客户端程序，将现实图像传输至所述设备。所述现实图像可以包括针对现实世界采集的现实图像。例如，所述现实图像可以包括针对人物、车辆、房屋等目标采集的图像。
在一些实施例中，所述现实图像也可以是用户通过所述设备搭载的图像采集硬件采集的图像。例如，所述设备可以是手机终端，所述图像采集硬件可以是手机终端搭载的摄像头。用户通过摄像头可以采集现实图像。
接收到所述现实图像后，所述设备可以执行S102。
本申请记载的渲染素材，具体是指对现实图像进行渲染的素材。所述渲染素材可以是二维或三维素材。以下以三维素材为例进行说明。
根据不同的渲染场景，可以设置不同的渲染素材。在一些实施例中，所述渲染素材可以是一些虚拟道具。例如，在利用虚拟人头对人物图像中的人脸进行图像渲染的场景中，渲染素材可以是虚拟人头。再例如，在利用虚拟动物头对人物图像中的人脸进行图像渲染的场景中，渲染素材可以是虚拟动物头。在一些实施例中，渲染素材可以对现实图像中的目标渲染对象进行渲染。所述目标渲染对象可以是指现实图像中待渲染的对象。在渲染过程中，渲染素材会替换该目标渲染对象，呈现在渲染之后的渲染图像中。例如，在利用虚拟人头或虚拟动物头对人物图像中的人脸进行图像渲染的场景中，目标渲染对象可以是真实人头。虚拟人头或虚拟动物头可以替换真实人头呈现在渲染图像中。本申请中记载的目标区域，可以是指现实图像中包括实物对象的区域。所述实物对象是指与渲染素材在同一空间产生遮挡关系的对象。
举例来说，在利用虚拟人头进行现实图像渲染的场景中，虚拟人头的头发可能会被现实图像中的人物身体遮挡。此时，可以将包含人物身体(实物对象)的前景区域确定为所述目标区域。然后可以确定虚拟人头的头发部分中，投影至所述目标区域内的部分，并将该部分中处于预设遮挡平面之后的部分确定为被人物身体遮挡的部分。
如此将投影在包含实物对象的目标区域内、处于预设遮挡平面之后的部分确定为被实物对象遮挡的部分，可以通过目标区域框定渲染素材可能被遮挡的部分，使渲染素材中不可能与实物对象发生遮挡关系的部分不会被确定为被遮挡部分。从而一方面提升被遮挡部分确认效率，另一方面，与“一刀切”式的提出方式相比，可以准确地确定渲染素材中的被遮挡部分，进而提升遮挡渲染效果。
在一些实施例中，在执行S102时，可以利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域。
所述分割网络，可以是基于神经网络或深度学习生成的像素级分割网络。所述像素级分割网络的输出结果可以包括对输入图像包括的每一个像素点进行二分类，以区分该 像素点是否处于目标区域内的结果。本申请不对分割网络的网络结构进行特别限定。
在一些实施例中，可以先利用多个包括目标区域的标注信息的训练样本，对所述分割网络进行训练。
请参见图2，图2为本申请示出的一种分割网络训练方法的流程图。
在一些实施例中，如图2所示，可以先执行S202，获取包括多个训练样本的训练样本集。
其中，所述训练样本集包括的训练样本可以包括目标区域的标注信息。所述目标区域包括根据业务需求预先设定的区域。在一些实施例中，可以根据业务需求确定图像中的目标区域。然后对获取的一些图像进行目标区域的标注，以得到训练样本。
例如，在利用虚拟人头对人物图像中的人脸进行图像渲染的场景中，可以将人物图像中包括身体对象的区域确定为目标区域。比如，所述目标区域可以是人物图像的前景区域。前景区域通常可以涵盖完整的人物，也即涵盖整个身体对象。然后可以对获取的人物图像中的每个像素点进行标注，确定各像素点是否处于目标区域内，完成人物图像的标注得到训练样本。
之后可以执行S204，基于所述训练样本集，对所述分割网络进行训练。完成训练后，可以利用分割网络，对所述现实图像进行分割处理，得到所述目标区域。
由此可以在构建训练样本时，根据业务需求灵活确定目标区域，使得利用这些训练样本训练得到的分割网络可以从现实图像中获取符合业务需求的目标区域，进而提升渲染灵活性。
通常本申请记载的渲染素材维护在第二空间中。所述第二空间可以包括生成素材的三维空间，以第二坐标系表示。在设备中可以存储渲染素材包括的各顶点在所述第二坐标系的位置信息，以存储该素材。
在一些实施例中，对所述现实图像进行三维建模可以得到第一空间。该第一空间可以理解为现实图像的成像三维空间。在一些实施例中，可以采用单目相机或双目相机法，基于图像采集硬件的设备参数，对现实图像进行三维建模，得到该第一空间。该第一空间以第一坐标系表示。
其中，所述第一空间与所述第二空间可以是同一空间，也可以是不同的空间。如果第一空间与第二空间是相同的空间，则可以直接执行S104。如果第一空间与第二空间是不同的空间，在执行S104时，可以先将渲染素材映射至第一空间内。
在一些实施例中，所述渲染素材可以标识有多个第一关键点。在一些实施例中，所述渲染素材为虚拟道具时，所述多个第一关键点可以是虚拟道具轮廓上的多个关键点。例如，所述渲染素材为虚拟人头时，所述多个第一关键点可以是头顶，耳朵，鼻尖，面部等预设位置的关键顶点。其中，所述第一关键点在第二空间对应的位置为第一位置。
在一些实施例中，所述现实图像内包括待渲染的目标渲染对象。在渲染过程中，渲染素材会替换该目标渲染对象，呈现在渲染图像中。
所述目标渲染对象标识有与所述多个第一关键点一一对应的多个第二关键点。在一些实施例中，所述多个第一关键点与多个第二关键点可以是在轮廓上位置相对应的点。例如，多个第一关键点是虚拟人头的头顶，耳朵，鼻尖，面部等预设位置的关键顶点时，所述多个第二关键点可以是现实图像中真实人头的头顶，耳朵，鼻尖，面部等相应位置的关键顶点。其中，所述第二关键点在现实图像中对应的位置为第二位置。
请参见图3，图3为本申请示出的一种渲染素材空间映射方法的流程示意图。
在执行S104时，如图3所示，可以执行S302，获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系。
在一些实施例中，可以采用映射关系求解算法，利用多个第一关键点在第二空间的第一位置，多个第二关键点在第一空间的第二位置，以及采集现实图像的图像采集硬件参数等信息，得到将第二空间中的顶点映射到第一空间中的映射关系。
在一些实施例中，所述映射关系求解算法可以包括PNP(Perspective-N-Point，多点透视成像)算法。所述位置映射关系可以包括，同一像素点在从第二空间映射至第一空间时的平移量和旋转量。本申请不对映射关系求解算法进行特别限定。
之后可以执行S304，根据位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间。
本步骤中，在获取位置映射关系后，可以利用该关系，将渲染素材包括的各顶点映射至第一空间。由此一方面可以使该素材与现实图像中的实物对象处于同一三维空间，进而便于进行遮挡判断，图像渲染等处理，另一方面，可以使渲染素材更贴近目标渲染对象真实的位置、朝向等细节状态，提升渲染效果。
在将渲染素材映射至第一空间内后，可以执行S306，将映射至所述第一空间的所述渲染素材投影至所述现实图像。
本步骤中，可以将渲染素材在第一空间的三维坐标位置，转变为二维坐标位置，实现投影至所述现实图像。
之后执行S308，确定处于所述目标区域内的投影部分。
本步骤中，可以确定渲染素材包括的各顶点中，在现实图像中的投影位置处在目标区域内的顶点，然后可以将这些顶点在第一空间中对应的三维部分确定为所述投影部分。所述投影部分可以理解为渲染素材中，可能与目标区域内的实物对象产生遮挡关系的部分。
在得到投影部分后，可以继续执行S104，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分。
所述预设遮挡平面，可以是根据业务需求预先设定的平面。渲染素材在预设遮挡平面之后的部分在现实中可能会被实物对象遮挡。其中，在不同的场景中，可以设置不同的遮挡平面。例如，在利用虚拟人头对人物图像中的人脸进行渲染的场景中，可以着重考虑虚拟人头的头发部分可能被身体部分遮挡。因此可以将身体平面作为所述预设遮挡平面。在一些实施例中，所述身体平面可以指身体的前表面。
在一些实施例中，可以对预设遮挡平面进行配置。
请参见图4，图4为本申请示出的一种预设遮挡平面配置方法的流程示意图。
如图4所示，可以先执行S402，接收针对遮挡平面的配置信息。在本步骤中，用户(例如，技术人员)可以通过设备提供的接口，输入所述配置信息，所述设备可以接收该配置信息。
所述配置信息可以至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度和朝向信息。在三维空间中，通常可以利用深度和朝向信息指示平面的位置信息。所述深度可以指示第二空间对应的第二坐标系的原点到所述预设遮挡平面的距离。所述朝向可以指示所述预设遮挡平面的法向量的角度。通过这两个参数可以唯一指定预设遮挡平面在第二空间的位置。
例如，在利用虚拟人头对人物图像中的人脸进行图像渲染的场景中，可以将身体前 表面设置为所述预设遮挡平面，用户可以根据经验将身体前表面在第二空间中的深度和朝向信息打包至配置信息中，并通过设备提供的接口将配置信息传递至设备。
设备在接收到配置信息后，可以执行S404，基于所述配置信息，确定所述预设遮挡平面。本步骤中，设备可以利用搭载的图像处理软件，完成遮挡平面的配置。
在一些实施例中，所述第一空间与所述第二空间为不同的空间，在执行S104时，可以将第二空间中的预设遮挡平面映射至所述第一空间，以将遮挡平面与渲染素材置于同一三维空间中进行遮挡判断。
请参见图5，图5为本申请示出的一种被遮挡部分确定方法的流程示意图。
在执行S104时，如图5所示，可以执行S502，将所述投影部分包括的各顶点分别作为当前顶点。本步骤中，可以按照预设顺序将各顶点作为当前顶点。
然后可以执行S504，根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与所述预设遮挡平面相交的交点。本步骤中可以利用所述原点与所述当前顶点确定一条直线，并将该直线与遮挡平面的交点确定为所述交点。
之后可比较所述原点到所述交点的第一距离，和所述原点到所述当前顶点的第二距离的大小，并执行S506与S508，响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后。响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之前。这里，仅仅是举例，并不具有限制性，在实际实现中，根据第一空间的坐标系的设置，也可以是确定当第一距离小于第二距离时，所述当前顶点的位置位于预设遮挡平面之前，当第一距离大于第二距离时，所述当前顶点的位置位于预设遮挡平面之后。
由此可以通过判断渲染素材投影部分中的顶点，与预设遮挡平面之间的位置关系，来准确地确定投影部分中确实被该遮挡平面遮挡的部分。
在确定渲染素材确实被遮挡的部分后，可以执行S106。
在一些实施例中，可以采用以下至少一项方式进行遮挡剔除处理：
删除所述渲染素材中被所述实物对象遮挡的部分；
调整所述渲染素材中被所述实物对象遮挡的部分的透明度；
修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式。
其中，所述删除所述渲染素材中被所述实物对象遮挡的部分，可以包括通过运行预先编写的遮挡剔除程序，删除渲染素材被遮挡的部分的顶点以及顶点对应的像素，从而不会在渲染图像中展现该部分，实现对该部分的剔除效果。
所述调整所述渲染素材中被所述实物对象遮挡的部分的透明度，可以包括通过运行预先编写的遮挡剔除程序，调整渲染素材被遮挡的部分的顶点的像素值，使被遮挡的部分的透明度足够大，以至于不会在渲染图像中展现该部分，实现对该部分的剔除效果。
所述修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式，可以包括通过运行预先编写的遮挡剔除程序，修改所述像素混合模式。通过修改所述像素混合模式，可以调整被遮挡的部分与背景部分的展现效果，在视觉上使被遮挡的部分融合至背景部分，从而在渲染图像中不会展现该部分，实现对该部分的剔除效果。
在一些实施例中，在执行S106时，可以通过光栅化渲染的方式利用渲染素材对现实图像进行渲染。在渲染过程中，可以采用前述任一遮挡剔除处理方法，对渲染素材中被所述实物对象遮挡的部分进行剔除处理，以使得渲染图像可以展现出与真实场景中遮挡关系匹配的渲染效果。
在前述实施例公开的方案中，可以获取现实图像中与渲染素材在同一空间产生遮挡关系的实物对象的目标区域；然后将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分；之后将所述投影部分中，位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分，并进行后续遮挡剔除处理与图像渲染处理。
一方面，在确定渲染素材被遮挡部分的过程中，可以无需利用专用硬件设备进行深度测试，从而可以在普通的设备中进行遮挡渲染，提升渲染普适性；
另一方面，在确定被遮挡部分的过程中，可以将包含可能与渲染素材发生遮挡关系的实物对象的目标区域内、处于预设遮挡平面之后的部分确定为被实物对象遮挡的部分，从而可以通过目标区域框定渲染素材可能被遮挡的部分，使渲染素材中不可能与实物对象发生遮挡关系的部分不会被确定为被遮挡部分，与“一刀切”式的提出方式相比，可以准确地确定渲染素材中的被遮挡部分，进而提升遮挡渲染效果。
以下结合直播场景进行实施例说明。
在直播场景中可以利用虚拟道具库中提供的虚拟人头实时对采集的直播图像进行渲染。
直播过程使用的直播客户端可以搭载于移动终端中。该移动终端可以搭载普通的摄像头(不要求该摄像头具有深度测试功能)，用于实时采集直播图像。
所述虚拟道具库可以安装在移动终端(以下简称终端)本地，或与直播客户端(以下简称客户端)对应的服务端中。开发人员预先在所述虚拟道具库中，为虚拟人头配置遮挡平面，以及将虚拟人头可被遮挡的部分的类别配置为头发。
用户可以通过客户端选择心仪的虚拟人头。请参见图6A，图6A为本申请示出的一种虚拟人头的示意图。假设用户选择的虚拟人头如图6A所示。
请参见图7，图7为本申请示出的一种图像渲染方法流程图。
如图7所示，在直播过程中，终端可以执行S71，实时接收摄像头采集的直播图像。请参见图6B，图6B为本申请示出的一种直播图像的示意图。假设采集的直播图像如图6B所示。
然后可以执行S72，利用预先训练的人物前景区域分割网络，得到直播图像的人物前景区域(即前述实施例中的目标区域)。该区域中可能包括人物的身体部分，该身体部分可能对虚拟人头的头发造成遮挡，通过获取人物前景区域，一方面可以缩小遮挡判断的范围，提升被遮挡部分确认效率，另一方面，可以通过人物前景区域框定虚拟人头可能被遮挡的部分，使虚拟人头处于该人物前景区域之外的部分不会被确定为被遮挡的部分，与“一刀切”式的提出方式相比，可以准确地确定渲染素材中的被遮挡部分，进而提升遮挡渲染效果。
请参见图6C，图6C为本申请示出的一种人物前景区域的示意图。假设经过S72后得到的人物前景区域如图6C所示
然后可以执行S73，采集直播图像中真实人头上的多个第二关键点的二维坐标，并利用摄像头的参数，以及虚拟人头上的多个对应第一关键点的三维坐标，将用户选择 的虚拟人头以及该虚拟人头对应的遮挡平面，映射在摄像头形成的成像空间(即前述第一空间)中。由此一方面可以使该素材与现实图像中的人物身体对象(实物对象)处于同一三维空间，进而便于进行遮挡判断，图像渲染等处理，另一方面，可以使渲染素材更贴近真实人头(目标渲染对象)真实的位置、朝向等细节状态，提升渲染效果。
之后可以执行S74，将虚拟人头投影到所述直播图像，并确定处于所述人物前景区域内的，可能被身体遮挡的投影部分。
之后可以执行S75，根据该投影部分与遮挡平面的位置关系，确定虚拟人头中被身体遮挡的头发部分。由此可以将虚拟人头中，处于人物前景区域内，并且处于遮挡平面之后的部分确定为被身体遮挡的头发部分，从而可以确定出准确的，符合真实场景的被遮挡部分，进而使渲染效果更真实。
之后可以执行S76，在对直播图像进行光栅化渲染的过程中，将被身体遮挡的头发部分进行剔除处理，得到更真实的渲染图像。请参见图6D，图6D为本申请示出的一种渲染图像的示意图。通过S71-S76的步骤后可以得到如图6D所示的渲染图像。由此可以得到与头发和身体真实遮挡关系匹配的渲染效果。
与前述实施例相应的，本申请提出一种图像处理装置80。
请参见图8，图8为本申请示出的一种图像处理装置的结构示意图。
如图8所示，所述装置80可以包括：获取模块81，用于获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象；确定模块82，用于将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分；渲染模块83，用于对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
在一些实施例中，所述渲染素材标识有多个第一关键点，所述现实图像内包括待渲染的目标渲染对象，所述目标渲染对象中标识有与所述多个第一关键点一一对应的多个第二关键点；所述确定模块82，具体用于：获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系；根据所述位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间，所述第一空间包括基于所述现实图像进行三维建模得到的空间；将映射至所述第一空间的所述渲染素材投影至所述现实图像。
在一些实施例中，所述装置80还包括：配置模块，用于接收针对遮挡平面的配置信息，所述配置信息至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度信息和朝向信息；基于所述配置信息，确定所述预设遮挡平面。
在一些实施例中，所述第一空间与所述第二空间为不同的空间，所述确定模块82，具体用于：将所述预设遮挡平面映射至所述第一空间；将所述投影部分中位置处于映射至所述第一空间的所述预设遮挡平面之后的部分，确定为所述渲染素材的被遮挡部分。
在一些实施例中，所述装置80还包括：将所述投影部分包括的各顶点分别作为当前顶点；根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与所述预设遮挡平面相交的交点；响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后；响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平 面之前。
在一些实施例中，所述对所述渲染素材中对被所述实物对象遮挡的部分进行遮挡剔除处理包括以下至少一项方式：删除所述渲染素材中被所述实物对象遮挡的部分；调整所述渲染素材中被所述实物对象遮挡的部分的透明度；修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式。
在一些实施例中，所述获取模块81，具体用于：利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域；所述装置80还包括：训练模块，用于获取包括多个训练样本的训练样本集，所述训练样本包括目标区域的标注信息；所述目标区域包括根据业务需求预先设定的区域；基于所述训练样本集，对所述分割网络进行训练。
在一些实施例中，所述渲染素材包括三维虚拟人头；或者，所述目标区域包括所述现实图像中的前景区域；或者，所述实物对象包括人物身体；或者，所述现实图像包括的目标渲染对象为真实人头。
本申请示出的图像处理装置的实施例可以应用于电子设备上。相应地，本申请公开了一种电子设备，该设备可以包括：处理器，用于存储处理器可执行指令的存储器。其中，所述处理器被配置为调用所述存储器中存储的可执行指令，实现前述任一实施例示出的图像处理方法。
请参见图9，图9为本申请示出的一种电子设备的硬件结构示意图。
如图9所示，该电子设备可以包括用于执行指令的处理器901，用于进行网络连接的网络接口902，用于为处理器存储运行数据的内存903，以及用于存储状态切换装置对应指令的非易失性存储器904。
其中，所述装置的实施例可以通过软件实现，也可以通过硬件或者软硬件结合的方式实现。以软件实现为例，作为一个逻辑意义上的装置，是通过其所在电子设备的处理器将非易失性存储器中对应的计算机程序指令读取到内存中运行形成的。从硬件层面而言，除了图9所示的处理器、内存、网络接口、以及非易失性存储器之外，实施例中装置所在的电子设备通常根据该电子设备的实际功能，还可以包括其他硬件，对此不再赘述。
可以理解的是，为了提升处理速度，所述图像处理装置对应指令也可以直接存储于内存中，在此不作限定。
本申请提出一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序可以用于使处理器执行前述任一实施例示出的图像处理方法。
本领域技术人员应明白，本申请一个或多个实施例可提供为方法、系统或计算机程序产品。因此，本申请一个或多个实施例可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本申请一个或多个实施例可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、光学存储器等)上实施的计算机程序产品的形式。
本申请中记载的“和/或”表示至少具有两者中的其中一个，例如，“A和/或B”包括三种方案：A、B、以及“A和B”。
本申请中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于数据处理设备实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
所述对本申请特定实施例进行了描述。其它实施例在所附权利要求书的范围内。在一些情况下，在权利要求书中记载的行为或步骤可以按照不同于实施例中的顺序来执行并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要求示出的特定顺序或者连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的或者可能是有利的。
本申请中描述的主题及功能操作的实施例可以在以下中实现：数字电子电路、有形体现的计算机软件或固件、包括本申请中公开的结构及其结构性等同物的计算机硬件、或者它们中的一个或多个的组合。本申请中描述的主题的实施例可以实现为一个或多个计算机程序，即编码在有形非暂时性程序载体上以被数据处理装置执行或控制数据处理装置的操作的计算机程序指令中的一个或多个模块。可替代地或附加地，程序指令可以被编码在人工生成的传播信号上，例如机器生成的电、光或电磁信号，该信号被生成以将信息编码并传输到合适的接收机装置以由数据处理装置执行。计算机存储介质可以是机器可读存储设备、机器可读存储基板、随机或串行存取存储器设备、或它们中的一个或多个的组合。
本申请中描述的处理及逻辑流程可以由执行一个或多个计算机程序的一个或多个可编程计算机执行，以通过根据输入数据进行操作并生成输出来执行相应的功能。所述处理及逻辑流程还可以由专用逻辑电路—例如FPGA(现场可编程门阵列)或ASIC(专用集成电路)来执行，并且装置也可以实现为专用逻辑电路。
适合用于执行计算机程序的计算机包括，例如通用和/或专用微处理器，或任何其他类型的中央处理系统。通常，中央处理系统将从只读存储器和/或随机存取存储器接收指令和数据。计算机的基本组件包括用于实施或执行指令的中央处理系统以及用于存储指令和数据的一个或多个存储器设备。通常，计算机还将包括用于存储数据的一个或多个大容量存储设备，例如磁盘、磁光盘或光盘等，或者计算机将可操作地与此大容量存储设备耦接以从其接收数据或向其传送数据，抑或两种情况兼而有之。然而，计算机不是必须具有这样的设备。此外，计算机可以嵌入在另一设备中，例如移动电话、个人数字助理(PDA)、移动音频或视频播放器、游戏操纵台、全球定位系统(GPS)接收机、或例如通用串行总线(USB)闪存驱动器的便携式存储设备，仅举几例。
适合于存储计算机程序指令和数据的计算机可读介质包括所有形式的非易失性存储器、媒介和存储器设备，例如包括半导体存储器设备(例如EPROM、EEPROM和闪存设备)、磁盘(例如内部硬盘或可移动盘)、磁光盘以及0xCD_00ROM和DVD-ROM盘。处理器和存储器可由专用逻辑电路补充或并入专用逻辑电路中。
虽然本申请包含许多具体实施细节，但是这些不应被解释为限制任何公开的范围或所要求保护的范围，而是主要用于描述特定公开的具体实施例的特征。本申请内在多个实施例中描述的某些特征也可以在单个实施例中被组合实施。另一方面，在单个实施例中描述的各种特征也可以在多个实施例中分开实施或以任何合适的子组合来实施。此外，虽然特征可以如上所述在某些组合中起作用并且甚至最初如此要求保护，但是来自所要求保护的组合中的一个或多个特征在一些情况下可以从该组合中去除，并且所要求保护的组合可以指向子组合或子组合的变型。
类似地，虽然在附图中以特定顺序描绘了操作，但是这不应被理解为要求这些操作以所示的特定顺序执行或顺次执行、或者要求所有例示的操作被执行，以实现期望的结果。在某些情况下，多任务和并行处理可能是有利的。此外，所述实施例中的各种系统模块和组件的分离不应被理解为在所有实施例中均需要这样的分离，并且应当理解，所描述的程序组件和系统通常可以一起集成在单个软件产品中，或者封装成多个软件产品。
由此，主题的特定实施例已被描述。其他实施例在所附权利要求书的范围以内。在某些情况下，权利要求书中记载的动作可以以不同的顺序执行并且仍实现期望的结果。此外，附图中描绘的处理并非必需所示的特定顺序或顺次顺序，以实现期望的结果。在某些实现中，多任务和并行处理可能是有利的。
以上所述仅为本申请一个或多个实施例的较佳实施例而已，并不用以限制本申请一个或多个实施例，凡在本申请一个或多个实施例的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请一个或多个实施例保护的范围之内。

Claims (18)

  1. 一种图像处理方法，其特征在于，所述方法包括：
    获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象；
    将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分；
    对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
  2. 根据权利要求1所述的方法，其特征在于，所述渲染素材标识有多个第一关键点，所述现实图像内包括待渲染的目标渲染对象，所述目标渲染对象中标识有与所述多个第一关键点一一对应的多个第二关键点；
    所述将所述渲染素材投影至所述现实图像，包括：
    获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系；
    根据所述位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间，所述第一空间包括基于所述现实图像进行三维建模得到的空间；
    将映射至所述第一空间的所述渲染素材投影至所述现实图像。
  3. 根据权利要求2所述的方法，其特征在于，所述方法还包括：
    接收针对遮挡平面的配置信息，所述配置信息至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度信息和朝向信息；
    基于所述配置信息，确定所述预设遮挡平面。
  4. 根据权利要求3所述的方法，其特征在于，所述第一空间与所述第二空间为不同的空间，所述将所述投影部分中，位置处于所述预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分，包括：
    将所述预设遮挡平面映射至所述第一空间；
    将所述投影部分中位置处于映射至所述第一空间的所述预设遮挡平面之后的部分，确定为所述渲染素材的被遮挡部分。
  5. 根据权利要求1-4任一所述的方法，其特征在于，所述方法还包括：
    将所述投影部分包括的各顶点分别作为当前顶点；
    根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与所述预设遮挡平面相交的交点；
    响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后；
    响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之前。
  6. 根据权利要求1-5任一所述的方法，其特征在于，所述对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理包括以下至少一项方式：
    删除所述渲染素材中被所述实物对象遮挡的部分；
    调整所述渲染素材中被所述实物对象遮挡的部分的透明度；
    修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像素混合模式。
  7. 根据权利要求1-6任一所述的方法，其特征在于，所述获取现实图像中的目标区域，包括：
    利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域；
    所述方法还包括：
    获取包括多个训练样本的训练样本集，所述训练样本包括目标区域的标注信息，所述目标区域包括根据业务需求预先设定的区域；
    基于所述训练样本集，对所述分割网络进行训练。
  8. 根据权利要求1-7任一所述的方法，其特征在于，所述渲染素材包括三维虚拟人头；或者，所述目标区域包括所述现实图像中的前景区域；或者，所述实物对象包括人物身体；或者，所述现实图像包括的目标渲染对象为真实人头。
  9. 一种图像处理装置，其特征在于，所述装置包括：
    获取模块，用于获取现实图像中的目标区域及渲染素材，所述目标区域包括与所述渲染素材在同一空间产生遮挡关系的实物对象；
    确定模块，用于将所述渲染素材投影至所述现实图像，并确定处于所述目标区域内的投影部分，将所述投影部分中位置处于预设遮挡平面之后的部分，确定为所述渲染素材中被所述实物对象遮挡的部分；
    渲染模块，用于对所述渲染素材中被所述实物对象遮挡的部分进行遮挡剔除处理，并利用遮挡剔除处理后的渲染素材对所述现实图像进行渲染处理，得到渲染图像。
  10. 根据权利要求9所述的装置，其特征在于，所述渲染素材标识有多个第一关键点，所述现实图像内包括待渲染的目标渲染对象，所述目标渲染对象中标识有与所述多个第一关键点一一对应的多个第二关键点；
    所述确定模块，具体用于：
    获取所述多个第一关键点与所述多个第二关键点之间的位置映射关系；
    根据所述位置映射关系，将所述渲染素材映射至与所述目标渲染对象关联的第一空间，所述第一空间包括基于所述现实图像进行三维建模得到的空间；
    将映射至所述第一空间的所述渲染素材投影至所述现实图像。
  11. 根据权利要求10所述的装置，其特征在于，所述装置还包括：
    配置模块，用于接收针对遮挡平面的配置信息，所述配置信息至少包括所述遮挡平面在与所述渲染素材关联的第二空间中的深度信息和朝向信息；
    基于所述配置信息，确定所述预设遮挡平面。
  12. 根据权利要求11所述的装置，其特征在于，所述第一空间与所述第二空间为不同的空间，所述确定模块，具体用于：
    将所述预设遮挡平面映射至所述第一空间；
    将所述投影部分中位置处于映射至所述第一空间的所述预设遮挡平面之后的部分，确定为所述渲染素材的被遮挡部分。
  13. 根据权利要求9-12任一所述的装置，其特征在于，所述装置还包括：
    将所述投影部分包括的各顶点分别作为当前顶点；
    根据基于所述现实图像进行三维建模得到的第一空间所对应坐标系的原点与所述当前顶点，确定穿过映射至所述第一空间的所述预设遮挡平面的直线，并确定所述直线与所述预设遮挡平面相交的交点；
    响应于所述原点至所述交点的第一距离小于所述原点至所述当前顶点的第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之后；
    响应于所述第一距离大于所述第二距离，确定所述当前顶点的位置处于所述预设遮挡平面之前。
  14. 根据权利要求9-13任一所述的装置，其特征在于，所述对所述渲染素材中对被所述实物对象遮挡的部分进行遮挡剔除处理包括以下至少一项方式：
    删除所述渲染素材中被所述实物对象遮挡的部分；
    调整所述渲染素材中被所述实物对象遮挡的部分的透明度；
    修改所述渲染素材中被所述实物对象遮挡的部分与所述现实图像中背景部分的像 素混合模式。
  15. 根据权利要求9-14任一所述的装置，其特征在于，所述获取模块，具体用于：
    利用基于神经网络生成的分割网络，对所述现实图像进行分割处理，得到所述目标区域；
    所述装置还包括：
    训练模块，用于获取包括多个训练样本的训练样本集，所述训练样本包括目标区域的标注信息；所述目标区域包括根据业务需求预先设定的区域；
    基于所述训练样本集，对所述分割网络进行训练。
  16. 根据权利要求9-15任一所述的装置，其特征在于，所述渲染素材包括三维虚拟人头；或者，所述目标区域包括所述现实图像中的前景区域；或者，所述实物对象包括人物身体；或者，所述现实图像包括的目标渲染对象为真实人头。
  17. 一种电子设备，其特征在于，所述设备包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器通过运行所述可执行指令以实现如权利要求1-8任一所述的图像处理方法。
  18. 一种计算机可读存储介质，其特征在于，所述存储介质存储有计算机程序，所述计算机程序用于使处理器执行如权利要求1-8任一所述的图像处理方法。
PCT/CN2022/080677 2021-06-30 2022-03-14 图像处理方法、装置、设备以及存储介质 WO2023273414A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110739170.1 2021-06-30
CN202110739170.1A CN113240692B (zh) 2021-06-30 2021-06-30 一种图像处理方法、装置、设备以及存储介质

Publications (1)

Publication Number Publication Date
WO2023273414A1 true WO2023273414A1 (zh) 2023-01-05

Family

ID=77141151

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/080677 WO2023273414A1 (zh) 2021-06-30 2022-03-14 图像处理方法、装置、设备以及存储介质

Country Status (3)

Country Link
CN (1) CN113240692B (zh)
TW (1) TW202303520A (zh)
WO (1) WO2023273414A1 (zh)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240692B (zh) * 2021-06-30 2024-01-02 北京市商汤科技开发有限公司 一种图像处理方法、装置、设备以及存储介质
CN114419299A (zh) * 2022-01-21 2022-04-29 北京字跳网络技术有限公司 虚拟物体的生成方法、装置、设备及存储介质
CN114494556A (zh) * 2022-01-30 2022-05-13 北京大甜绵白糖科技有限公司 一种特效渲染方法、装置、设备及存储介质
CN114693780A (zh) * 2022-04-11 2022-07-01 北京字跳网络技术有限公司 图像的处理方法、装置、设备、存储介质和程序产品
CN117173305A (zh) * 2022-05-25 2023-12-05 北京字节跳动网络技术有限公司 三维效果形象的生成方法、装置、电子设备及存储介质
CN114820906B (zh) * 2022-06-24 2022-11-22 北京百度网讯科技有限公司 图像渲染方法、装置、电子设备及存储介质
CN115174985B (zh) * 2022-08-05 2024-01-30 北京字跳网络技术有限公司 特效展示方法、装置、设备及存储介质
CN116228952A (zh) * 2023-02-24 2023-06-06 北京字跳网络技术有限公司 一种虚拟对象挂载方法、装置、设备及介质
CN116542847B (zh) * 2023-07-05 2023-10-10 海豚乐智科技(成都)有限责任公司 一种低小慢目标高速图像模拟方法、存储介质及装置

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101162524A (zh) * 2006-08-11 2008-04-16 佳能株式会社 图像处理设备及方法
EP2275993A2 (en) * 2009-07-14 2011-01-19 Siemens Aktiengesellschaft Method for estimating the visibility of features on surfaces of object instances in multi-object scenes and method for perception planning in multi-object scenes
US20150325040A1 (en) * 2012-12-27 2015-11-12 Nokia Corporation Method, apparatus and computer program product for image rendering
CN109299658A (zh) * 2018-08-21 2019-02-01 腾讯科技（深圳）有限公司 脸部检测方法、脸部图像渲染方法、装置及存储介质
US20190206119A1 (en) * 2016-06-30 2019-07-04 Center Of Human-Centered Interaction For Coexistence Mixed reality display device
CN111815755A (zh) * 2019-04-12 2020-10-23 Oppo广东移动通信有限公司 虚拟物体被遮挡的区域确定方法、装置及终端设备
CN113240692A (zh) * 2021-06-30 2021-08-10 北京市商汤科技开发有限公司 一种图像处理方法、装置、设备以及存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103489214A (zh) * 2013-09-10 2014-01-01 北京邮电大学 增强现实系统中基于虚拟模型预处理的虚实遮挡处理方法
CN108765542B (zh) * 2018-05-31 2022-09-09 Oppo广东移动通信有限公司 图像渲染方法、电子设备和计算机可读存储介质
CN110136082B (zh) * 2019-05-10 2021-06-01 腾讯科技（深圳）有限公司 遮挡剔除方法、装置及计算机设备
CN111369655B (zh) * 2020-03-02 2023-06-30 网易（杭州）网络有限公司 渲染方法、装置和终端设备
CN112233215B (zh) * 2020-10-15 2023-08-22 网易（杭州）网络有限公司 轮廓渲染方法、装置、设备及存储介质
CN113034655A (zh) * 2021-03-11 2021-06-25 北京字跳网络技术有限公司 基于增强现实的试鞋方法、装置和电子设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101162524A (zh) * 2006-08-11 2008-04-16 佳能株式会社 图像处理设备及方法
EP2275993A2 (en) * 2009-07-14 2011-01-19 Siemens Aktiengesellschaft Method for estimating the visibility of features on surfaces of object instances in multi-object scenes and method for perception planning in multi-object scenes
US20150325040A1 (en) * 2012-12-27 2015-11-12 Nokia Corporation Method, apparatus and computer program product for image rendering
US20190206119A1 (en) * 2016-06-30 2019-07-04 Center Of Human-Centered Interaction For Coexistence Mixed reality display device
CN109299658A (zh) * 2018-08-21 2019-02-01 腾讯科技（深圳）有限公司 脸部检测方法、脸部图像渲染方法、装置及存储介质
CN111815755A (zh) * 2019-04-12 2020-10-23 Oppo广东移动通信有限公司 虚拟物体被遮挡的区域确定方法、装置及终端设备
CN113240692A (zh) * 2021-06-30 2021-08-10 北京市商汤科技开发有限公司 一种图像处理方法、装置、设备以及存储介质

Also Published As

Publication number Publication date
CN113240692B (zh) 2024-01-02
CN113240692A (zh) 2021-08-10
TW202303520A (zh) 2023-01-16

Similar Documents

Publication Publication Date Title
WO2023273414A1 (zh) 图像处理方法、装置、设备以及存储介质
CN115699114B (zh) 用于分析的图像增广的方法和装置
CN109561296B (zh) 图像处理装置、图像处理方法、图像处理系统和存储介质
US10373380B2 (en) 3-dimensional scene analysis for augmented reality operations
JP7387202B2 (ja) ３次元顔モデル生成方法、装置、コンピュータデバイス及びコンピュータプログラム
CN108830894A (zh) 基于增强现实的远程指导方法、装置、终端和存储介质
US20180276882A1 (en) Systems and methods for augmented reality art creation
US10621777B2 (en) Synthesis of composite images having virtual backgrounds
US20130095920A1 (en) Generating free viewpoint video using stereo imaging
US20190130648A1 (en) Systems and methods for enabling display of virtual information during mixed reality experiences
EP3533218B1 (en) Simulating depth of field
CN112258610B (zh) 图像标注方法、装置、存储介质及电子设备
US20190130633A1 (en) Systems and methods for using a cutting volume to determine how to display portions of a virtual object to a user
KR101652594B1 (ko) 증강 현실 컨텐츠 제공 장치 및 방법
US20230298280A1 (en) Map for augmented reality
US20220335684A1 (en) Finite aperture omni-directional stereo light transport
WO2023226370A1 (zh) 一种目标对象的三维重现方法及系统
EP4296947A1 (en) Calibration information determination method and apparatus, and electronic device
CN114881901A (zh) 视频合成方法、装置、设备、介质及产品
EP3676801B1 (en) Electronic devices, methods, and computer program products for controlling 3d modeling operations based on pose metrics
JP2022518402A (ja) 三次元再構成の方法及び装置
WO2024032165A1 (zh) 3d模型生成方法、系统及电子设备
CN109816791B (zh) 用于生成信息的方法和装置
CN116342831A (zh) 三维场景重建方法、装置、计算机设备及存储介质
WO2011069285A1 (en) Concave surface modeling in image-based visual hull

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22831245

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22831245

Country of ref document: EP

Kind code of ref document: A1