WO2020186935A1 - 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质 - Google Patents

虚拟对象的显示方法、装置、电子设备和计算机可读存储介质 Download PDF

Info

Publication number
WO2020186935A1
WO2020186935A1 PCT/CN2020/074447 CN2020074447W WO2020186935A1 WO 2020186935 A1 WO2020186935 A1 WO 2020186935A1 CN 2020074447 W CN2020074447 W CN 2020074447W WO 2020186935 A1 WO2020186935 A1 WO 2020186935A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual object
key point
image
display
virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/074447
Other languages
English (en)
French (fr)
Inventor
李润祥
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing ByteDance Network Technology Co Ltd
Original Assignee
Beijing ByteDance Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing ByteDance Network Technology Co Ltd filed Critical Beijing ByteDance Network Technology Co Ltd
Publication of WO2020186935A1 publication Critical patent/WO2020186935A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2016—Rotation, translation, scaling

Definitions

  • the present disclosure relates to the field of information processing, and in particular to a method, device, electronic device, and computer-readable storage medium for displaying virtual objects.
  • some APPs can display virtual objects and control the movement of the virtual objects according to the user's operation, for example, display the virtual object at a fixed position on the display screen, and then realize the steering movement of the virtual object according to the user's operation, etc.
  • this kind of display The form of the virtual object is relatively simple, and it cannot flexibly make the virtual object show more actions.
  • the embodiments of the present disclosure provide a method, apparatus, electronic device, and computer-readable storage medium for displaying virtual objects.
  • virtual objects By processing image frames in a video, virtual objects can be flexibly displayed according to the image frames in the video.
  • an embodiment of the present disclosure provides a method for displaying a virtual object, which is characterized in that it includes: acquiring an image frame in a video; determining, in the image frame, the image feature corresponding to the key point of the virtual object. The position of the key point of the virtual object; render the virtual object according to the rendering information corresponding to the key point of the virtual object and the position of the key point of the virtual object; display the virtual object in the first display area of the display interface .
  • determining the position of the key point of the virtual object in the image frame according to the image feature corresponding to the key point of the virtual object includes: determining the position of the key point that matches the image feature in the image frame; The position of the key point matching the image feature is taken as the position of the key point of the virtual object.
  • the image feature includes the image feature of the person object.
  • the rendering information of the virtual object includes one or more of the following information: pixel position information of the virtual object; pixel color information of the virtual object; contour information of the virtual object; The texture information of the object; the modeling rule information of the virtual object.
  • the displaying the virtual object in the first display area of the display interface includes: zooming in or out displaying the virtual object in the first display area of the display interface.
  • the method further includes: displaying the image frame in a second display area of the display interface.
  • first display area overlaps the second display area, or the first display area does not overlap the second display area.
  • an embodiment of the present disclosure provides a display device for a virtual object, which is characterized in that it includes: a video acquisition module for acquiring image frames in a video; The image feature corresponding to the key point of the object determines the position of the key point of the virtual object; a rendering module is configured to render the key point of the virtual object according to the rendering information corresponding to the key point of the virtual object and the position of the key point of the virtual object.
  • Virtual object a display module that displays the virtual object in the first display area of the display interface.
  • the determining module is further configured to determine the position of the key point matching the image feature in the image frame, and use the position of the key point matching the image feature as the key point of the virtual object s position.
  • the image feature includes the image feature of the person object.
  • the rendering information of the virtual object includes one or more of the following information: pixel position information of the virtual object; pixel color information of the virtual object; contour information of the virtual object; The texture information of the object; the modeling rule information of the virtual object.
  • the display module is further configured to zoom in or out to display the virtual object in the first display area of the display interface.
  • the display module is further configured to display the image frame in the second display area of the display interface.
  • first display area overlaps the second display area, or the first display area does not overlap the second display area.
  • embodiments of the present disclosure provide an electronic device, including: a memory for storing computer readable instructions; and one or more processors coupled with the memory for running the computer readable instructions, When the processor is running, the method for displaying any one of the virtual objects in the foregoing first aspect is realized.
  • the embodiments of the present disclosure provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by the computer, the The computer executes any one of the virtual object display methods in the first aspect.
  • the present disclosure discloses a virtual object display method, device, electronic equipment and computer-readable storage medium.
  • the method for displaying the virtual object is characterized in that it includes: acquiring an image frame in a video; determining the position of the key point of the virtual object in the image frame according to the image feature corresponding to the key point of the virtual object; Rendering the virtual object according to the rendering information corresponding to the key points of the virtual object and the positions of the key points of the virtual object; displaying the virtual object in the first display area of the display interface.
  • the virtual object display method, device, electronic device, and computer-readable storage medium provided by the embodiments of the present disclosure can flexibly display the virtual object according to the image frame in the video by processing the image frame in the video.
  • FIG. 1 is a flowchart of an embodiment of a method for displaying virtual objects provided by an embodiment of the disclosure
  • FIG. 2 is a schematic diagram of key points and/or positions of key points of virtual objects provided by an embodiment of the disclosure
  • FIG. 3 is a schematic structural diagram of an embodiment of a display device for virtual objects provided by an embodiment of the disclosure
  • Fig. 4 is a schematic structural diagram of an electronic device provided according to an embodiment of the present disclosure.
  • illustrations provided in the following embodiments only illustrate the basic concept of the present disclosure in a schematic manner.
  • the illustrations only show components related to the present disclosure rather than the number, shape, and shape of the components in actual implementation.
  • the type, quantity, and ratio of each component can be changed at will during actual implementation, and the component layout type may also be more complicated.
  • the method for displaying the virtual object provided in this embodiment can be executed by a virtual object display device, which can be implemented as software, as hardware, or as a combination of software and hardware, such as
  • the display device includes a computer device, so that the method for displaying the virtual object provided in this embodiment is executed by the computer device.
  • the computer device can be a desktop or portable computer device, or a mobile terminal device. Wait.
  • Fig. 1 is a flowchart of the first embodiment of a method for displaying virtual objects provided by an embodiment of the present disclosure. As shown in Fig. 1, the method for displaying virtual objects in an embodiment of the present disclosure includes the following steps:
  • Step S101 acquiring image frames in the video
  • the display device of the virtual object may obtain the image frame in the video in various ways, which is not limited in the embodiment of the present disclosure.
  • the display device of the virtual object includes a camera device, so that the video is captured by the camera device to obtain the image frames in the video, and the display device of the virtual object can also obtain the image frames in the video through the network.
  • the storage device or the web camera obtains the image frames in the video.
  • the video in the embodiments of the present disclosure is composed of a series of image frames. When the video is played, the series of image frames may be displayed in the sequence of the video. Column image frames, for example, 24 image frames per second are displayed, of course, the image frames may also be called images.
  • Step S102 determining the position of the key point of the virtual object in the image frame according to the image feature corresponding to the key point of the virtual object;
  • the display device of the virtual object needs to acquire the position of the key point of the virtual object in order to display the virtual object. Therefore, in step S102, the display device of the virtual object uses the image feature corresponding to the key point of the virtual object in step S102. The position of the key point of the virtual object is determined in the image frame obtained in S101.
  • the virtual object in the embodiment of the present disclosure includes a preset number of key points, and each key point of the virtual object corresponds to an image feature, so that the key point of each virtual object corresponds to the image feature , Determining the position of the key point of the virtual object in the image frame.
  • image features may be characterized by color features and/or shape features, that is, image features include color features and/or shape features.
  • the image feature includes the image feature of the video object or the image feature of the image frame object, so that the image feature of the video object or the image feature of the image frame object can be determined from the image frame in the video.
  • the location of the key point corresponding to or matched with the image feature of the video object or the image feature of the image frame object is the key point positioning.
  • the video object or the image frame object may include any object in the image frame of the video, such as a person object and an object object, etc.
  • the image feature may include the image feature of the person object and the object Image features, etc.
  • determining the position of the key point of the virtual object in the image frame according to the image feature corresponding to the key point of the virtual object includes: determining the key point matching the image feature in the image frame And use the position of the key point matching the image feature as the position of the key point of the virtual object.
  • the image feature includes the image feature of the character object, so the virtual object display device determines the position of the key point of the virtual object according to the image feature of the character object corresponding to the key point of the virtual object.
  • the image feature of a person object can include face image feature, arm image feature, bone image feature, joint image feature, organ image feature, etc., so that it can be based on the image feature of the person object.
  • the position of the key points corresponding to or matching the image features of these person objects (also called the key points of the person objects) is determined in the image frame. Further, the positions of the key points of the person objects determined can be used as the key of the virtual object. The location of the point.
  • the virtual object includes five key points P0-P4, where the image feature corresponding to the key point P0 of the virtual object includes a face image feature, the image feature corresponding to the key point P1 of the virtual object includes a left-hand image feature, and The image feature corresponding to the key point P2 of the virtual object includes the right-hand image feature, the image feature corresponding to the key point P3 of the virtual object includes the left foot image feature, and the image feature corresponding to the key point P4 of the virtual object includes the right foot image feature.
  • step S102 the position of the key point of the person object determined in the image frame according to the face image feature is taken as the position of P0, and the person determined in the image frame according to the left-hand image feature
  • the position of the key point of the object is taken as the position of P1
  • the position of the key point of the person object determined in the image frame according to the right-hand image feature is taken as the position of P2
  • the position is determined in the image frame according to the left foot image feature
  • the position of the key point of the character object is taken as the position of P3
  • the position of the key point of the character object determined in the image frame according to the image characteristics of the right foot is taken as the position of P4.
  • the position of the character object can be located first by using the color feature.
  • the character object can be segmented from the background, and then the position of the key point of the character object corresponding to the image feature of the character object is determined according to the image feature of the character object, so as to determine the position of the key point of the character object As the location of the key point of the virtual object.
  • the image frames in the video are captured by a camera device.
  • the image sensor of the camera device records color information and color position information for each image frame during the shooting process, and then inputs the color information and color position information based on
  • the classifier of the convolutional neural network if there is a person object in the image frame, then the outline of the person object can be recognized by the classifier, so that the person object is segmented from the background, and then according to the segmented person Object, and perform search and comparison in the image frame based on the aforementioned image features to determine the position of the key point of the person object. This can also be called determining the key point of the person object, because the key point of the person object is determined
  • the point process is the process of positioning on the image frame according to the image features corresponding to the key points of the character object.
  • the key points of the person object are determined based on the image features corresponding to the key points of the person object. Therefore, the key points of the person object correspond to the image feature, and the image feature also corresponds to The key points of the character object; since the key points of the character object occupy only a very small area in the image (usually only a few to dozens of pixels in size), the image features corresponding to the key points of the character object are on the image The occupied area is usually very limited and local.
  • ASM and AAM methods statistical energy function methods, regression analysis methods, deep learning methods, classifier methods, batch extraction methods, and so on.
  • the position of the key point of the character object and the position of the key point of the virtual object may be represented by coordinates.
  • the camera device that shoots a video will set a corresponding coordinate system according to the frame during the shooting process.
  • the coordinate system is also called the coordinate system corresponding to the video or the image frame in the video.
  • the coordinate system so that for each pixel of the captured image frame, the corresponding coordinate can be marked based on the coordinate system.
  • the key point position of the character object determined from the image frame according to the image feature of the character object can also be based on The coordinate of one or more pixels occupied by the key point of the character object is represented (if the key point of the character object occupies multiple pixels, the average value of the coordinates of the multiple pixels can be used to represent the The position of the key point of the character object).
  • the position of the key point of the virtual object can also be represented based on the coordinates of one or more pixels occupied by the key point of the character object.
  • the coordinates of a key point of a character object can be expressed as (x, y), where x and y are the abscissa and ordinate respectively; for example, for an image frame of a video taken by a camera device with a depth sensor, Recording depth information for each pixel, then the coordinates of a key point of a character object in the image frame can also be expressed as (x, y, z), where x and y are the abscissa and ordinate, and z is Correspondingly, since the position of the key point of the character object is taken as the position of the key point of the virtual object, the coordinates of the key point of the virtual object can be expressed as (x, y), (x , Y, z).
  • the coordinates of the key points of the character object in the image frame refers to the coordinate system set by the aforementioned camera device during the shooting process according to the frame, but those skilled in the art can understand that for the For the image frame, other coordinate systems may also be established to indicate the position of the key point of the character object and the position of the key point of the virtual object, which is not limited in the embodiment of the present disclosure.
  • the embodiments of the present disclosure can use color features to locate the position of the character object to segment the character object from the background, and then determine the position of the key point of the character object from the image frame according to the image characteristic of the character object.
  • the position of the key point of the character object is taken as the position of the key point of the virtual object.
  • the embodiments of the present disclosure may also extract key points of the character object from the image frame by other methods.
  • the image feature of the character object corresponding to the key point of the virtual object determines the position of the key point corresponding to the image feature of the character object (that is, the aforementioned key point of the character object) from the image frame, and takes the position as the position Describe the location of the key points of the virtual object.
  • FIG. 2 is a schematic diagram of determining the position of the key point of the virtual object according to the image feature of the character object in step S102.
  • Figure 2 shows the key points of a total of 11 virtual objects that are determined from A to K, and the position of the key point of each virtual object can be represented by coordinates.
  • the key points A and B (positions) of the virtual object are determined from the image frame according to the torso image characteristics, and the key points C (position) of the virtual object are determined from the image frame according to the head image characteristics.
  • the key points D and E (locations) of the virtual object are determined from the image frame based on the image features of one arm, and the key points F and G (locations) of the virtual object are based on the image features of the other arm Determined from the image frame, the key points H and I (locations) of the virtual object are determined from the image frame according to the image characteristics of a leg, and the key points J and K (locations) of the virtual object It is determined from the image frame according to the image characteristics of the other leg.
  • the key points of these virtual objects and/ Or the positions of the key points of the virtual object constitute the posture or posture of the virtual object.
  • Step S103 Render the virtual object according to the rendering information corresponding to the key points of the virtual object and the positions of the key points of the virtual object.
  • the virtual object in the embodiment of the present disclosure can be any two-dimensional or three-dimensional virtual object.
  • the embodiment of the present disclosure does not make specific restrictions on this. Any virtual object that can be displayed on the display interface can be introduced into the present disclosure.
  • the virtual object In order to display the two-dimensional or three-dimensional virtual object, the virtual object needs to be rendered in step S103.
  • the rendering information corresponding to the key points of the virtual object includes at least one of the following information: pixel position information of the virtual object, pixel color information of the virtual object, contour information of the virtual object, The texture information of the virtual object, the modeling rule information of the virtual object. Since the position of the key point of the virtual object is determined in step S102, the virtual object can be rendered based on the position of the key point of the virtual object and the rendering information corresponding to the key point of the virtual object.
  • the key point position and contour information determine the size of the part of the virtual object corresponding to the virtual key point position, and adjust the size of the texture according to the size, so as to render a complete virtual object at the pixel level;
  • the key of the virtual object can be
  • the position of the point and the modeling rule information are modeled by triangles or polygons to form the outline of the virtual object, and textures are applied to achieve the rendering of the virtual object; it can also be generated by generating a confrontation network (GAN) or progressive structural condition GAN according to the said
  • GAN confrontation network
  • the position of the key point of the virtual object, and the texture information corresponding to the key point of each virtual object render the virtual object according to the zoom ratio.
  • Step S104 Display the virtual object in the first display area of the display interface.
  • step S104 is executed to display the virtual object in the first display area of the display interface.
  • displaying the virtual object in the first display area of the display interface includes: zooming in or out displaying the virtual object in the first display area of the display interface.
  • the virtual object represented by the position of the pixel and the color of the pixel can be obtained.
  • step S104 in the first display area of the display interface, according to the pixel position of the virtual object And pixel color, zoom in or zoom out to display the virtual object.
  • the pixel position of the virtual object can be enlarged or enlarged in the first display area of the display interface through coordinate transformation.
  • Display the virtual object in a zoomed-out manner; if the display interface adopts a new coordinate system, the pixels of the virtual object can be transformed into the coordinate system to zoom in or zoom out the display in the first display area of the display interface.
  • Virtual object in addition, when the virtual object is enlarged or reduced to display the virtual object, an increase or decrease in resolution may also be involved, but the embodiments of the present disclosure limit the method of displaying the virtual object.
  • the virtual object Since the position of the key point of the virtual object is determined in the image frame according to the image feature corresponding to the key point of the virtual object, the virtual object is rendered in step S103 and displayed in step S104
  • the action, posture, or posture of may be associated with the image feature.
  • the position determined in the image frame that is to say, the position of the key point of the head of the character object in the image frame is taken as the position of the key point C of the virtual object, and accordingly, the position in the image frame
  • the positions of the key points of the torso, hands, and legs of the character object are used as the key points A and B, D and E, F and G, H and I, and J and K of the virtual object respectively, so according to The position of the key point of the virtual object and the rendering information corresponding to the key point of the virtual object.
  • step S102- of the method for displaying virtual objects of the present disclosure may be executed for each image frame in the image frame.
  • the virtual objects rendered according to each image frame are displayed in the sequence, and when the virtual objects rendered according to each image frame are displayed in the first display area at a preset time interval (for example, 24 frames per second)
  • a preset time interval for example, 24 frames per second
  • the virtual object is equivalent to playing a video of the movement of the virtual object, in which the movement, posture, or posture of the virtual object in the played video corresponds to the movement of the character object in the video corresponding to step S101,
  • the posture or posture is consistent or corresponding.
  • the image frame is displayed in the second display area of the display interface.
  • the first display area overlaps the second display area, for example, the second display area fills the display interface, and the first display area occupies a corner of the display interface, such as the upper right corner;
  • the first display area and the second display area do not overlap, for example, the first display area fills the left half of the display interface, and the second display area fills the right half of the display interface .
  • the video in step S101 is played in the second display area of the display interface, and the virtual object displayed in the first display area of the display interface is based on the second display area.
  • the image frame currently displayed in the display area is rendered.
  • the video in step S101 is played in the second display area of the display interface
  • the image frames in the video are sequentially displayed at a preset speed (for example, 24 frames per second)
  • the virtual object displayed in the first display area of the display interface determines the key points of the virtual object according to the image characteristics corresponding to the key points of the virtual object in the image frame currently displayed in the second display area Position (that is, step S102), and then render the virtual object according to the rendering information corresponding to the key point of the virtual object and the position of the key point of the virtual object.
  • FIG. 3 is a schematic structural diagram of an embodiment of a virtual object display device 300 provided by an embodiment of the disclosure. As shown in FIG. 3, the device includes a video acquisition module 301, a determination module 302, a rendering module 303, and a display module 304 .
  • the video acquisition module 301 is used to acquire image frames in the video; the determination module 302 is used to determine the key points of the virtual object in the image frame according to the image characteristics corresponding to the key points of the virtual object.
  • the rendering module 303 is configured to render the virtual object according to the rendering information corresponding to the key points of the virtual object and the position of the key points of the virtual object; the display module 304 is configured to display The first display area of the interface displays the virtual object.
  • the display module 304 is further configured to display the image frame in the second display area of the display interface.
  • the device shown in FIG. 3 can execute the method of the embodiment shown in FIG. 1.
  • parts that are not described in detail in this embodiment refer to the related description of the embodiment shown in FIG.
  • the implementation process and technical effects of this technical solution refer to the description in the embodiment shown in FIG. 1, which will not be repeated here.
  • Electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, Mobile terminals such as car navigation terminals) and fixed terminals such as digital TVs, desktop computers, etc.
  • PDAs personal digital assistants
  • PADs tablets
  • PMPs portable multimedia players
  • vehicle-mounted terminals for example, Mobile terminals such as car navigation terminals
  • fixed terminals such as digital TVs, desktop computers, etc.
  • the electronic device shown in FIG. 4 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.
  • the electronic device 400 may include a processing device (such as a central processing unit, a graphics processor, etc.) 401, which may be loaded into a random access device according to a program stored in a read-only memory (ROM) 402 or from a storage device 408.
  • the programs in the memory (RAM) 403 execute various appropriate actions and processes.
  • the RAM 403 also stores various programs and data required for the operation of the electronic device 400.
  • the processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus or a communication line 404.
  • An input/output (I/O) interface 405 is also connected to the bus or communication line 404.
  • the following devices can be connected to the I/O interface 405: including input devices 406 such as touch screens, touch pads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; including, for example, liquid crystal displays (LCD), speakers, An output device 407 such as a vibrator; a storage device 408 such as a magnetic tape and a hard disk; and a communication device 409.
  • the communication device 409 may allow the electronic device 400 to perform wireless or wired communication with other devices to exchange data.
  • FIG. 4 shows an electronic device 400 having various devices, it should be understood that it is not required to implement or have all the illustrated devices. It may alternatively be implemented or provided with more or fewer devices.
  • the process described above with reference to the flowchart can be implemented as a computer software program.
  • the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart.
  • the computer program may be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402.
  • the processing device 401 the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
  • the aforementioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two.
  • the computer-readable storage medium includes, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable signal medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wire, optical cable, RF (Radio Frequency), etc., or any suitable combination of the above.
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device.
  • the foregoing computer-readable medium carries one or more programs, and when the foregoing one or more programs are executed by the electronic device, the electronic device is caused to execute the virtual object display method in the foregoing embodiment.
  • the computer program code used to perform the operations of the present disclosure may be written in one or more programming languages or a combination thereof.
  • the above-mentioned programming languages include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the program code can be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet connection).
  • LAN local area network
  • WAN wide area network
  • each block in the flowchart or block diagram can represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions.
  • the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved.
  • each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.
  • the units involved in the embodiments described in the present disclosure may be implemented in a software manner, or may be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation on the unit itself under certain circumstances.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种虚拟对象的显示方法，包括：获取视频中的图像帧(S101)；在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置(S102)；根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象(S103)；在显示界面的第一显示区域显示所述虚拟对象(S104)。通过对视频中的图像帧进行处理，能够根据视频中的图像帧灵活地显示虚拟对象。

Description

虚拟对象的显示方法、装置、电子设备和计算机可读存储介质
相关申请的交叉引用
本申请要求于2019年03月20日提交的，申请号为201910215466.6、发明名称为“虚拟对象的显示方法、装置、电子设备和计算机可读存储介质”的中国专利申请的优先权，该申请的全文通过引用结合在本申请中。
技术领域
本公开涉及信息处理领域，尤其涉及一种虚拟对象的显示方法、装置、电子设备及计算机可读存储介质。
背景技术
随着互联网相关技术的进步，视频的相关应用正成为重要的交互手段，例如用户可以制作卡通人物等虚拟对象的视频并发布到视频分享平台，但是制作虚拟对象的视频需要具备专业的美工技术和计算机图像处理技术，对于虚拟对象的每个动作都需要花费大量的工作进行编辑绘制，普通用户很难实现。
现有技术中，一些APP可以实现显示虚拟对象并根据用户的操作控制虚拟对象运动，例如在显示屏幕的固定位置显示虚拟对象，然后根据用户的操作实现虚拟对象的转向移动等，但是这种显示虚拟对象 的形式比较单一，且无法灵活地使虚拟对象显示出更多的动作。
发明内容
本公开实施例提供虚拟对象的显示方法，装置，电子设备，和计算机可读存储介质，通过对视频中的图像帧进行处理，能够根据视频中的图像帧灵活地显示虚拟对象。
第一方面，本公开实施例提供一种虚拟对象的显示方法，其特征在于，包括：获取视频中的图像帧；在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；在显示界面的第一显示区域显示所述虚拟对象。
进一步的，在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定虚拟对象的关键点的位置，包括：在所述图像帧中确定与所述图像特征匹配的关键点的位置；将与所述图像特征匹配的关键点的位置作为所述虚拟对象的关键点的位置。
进一步的，所述图像特征包括人物对象的图像特征。
进一步的，所述虚拟对象的渲染信息包括如下信息中的一种或多种：所述虚拟对象的像素位置信息；所述虚拟对象的像素颜色信息；所述虚拟对象的轮廓信息；所述虚拟对象的贴图信息；所述虚拟对象的建模规则信息。
进一步的，所述在显示界面的第一显示区域显示所述虚拟对象，包括：在所述显示界面的第一显示区域放大或缩小显示所述虚拟对 象。
进一步的，所述方法还包括：在所述显示界面的第二显示区域显示所述图像帧。
进一步的，所述第一显示区域与所述第二显示区域重叠，或者所述第一显示区域与所述第二显示区域不重叠。
第二方面，本公开实施例提供一种虚拟对象的显示装置，其特征在于，包括：视频获取模块，用于获取视频中的图像帧；确定模块，用于在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；渲染模块，用于根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；显示模块，在显示界面的第一显示区域显示所述虚拟对象。
进一步的，所述确定模块还用于在所述图像帧中确定与所述图像特征匹配的关键点的位置，并将与所述图像特征匹配的关键点的位置作为所述虚拟对象的关键点的位置。
进一步的，所述图像特征包括人物对象的图像特征。
进一步的，所述虚拟对象的渲染信息包括如下信息中的一种或多种：所述虚拟对象的像素位置信息；所述虚拟对象的像素颜色信息；所述虚拟对象的轮廓信息；所述虚拟对象的贴图信息；所述虚拟对象的建模规则信息。
进一步的，所述显示模块还用于在所述显示界面的第一显示区域放大或缩小显示所述虚拟对象。
进一步的，所述显示模块还用于在所述显示界面的第二显示区域显示所述图像帧。
进一步的，所述第一显示区域与所述第二显示区域重叠，或者所述第一显示区域与所述第二显示区域不重叠。
第三方面，本公开实施例提供一种电子设备，包括：存储器，用于存储计算机可读指令；以及与所述存储器耦合的一个或多个处理器，用于运行所述计算机可读指令，使得所述处理器运行时实现前述第一方面中的任一所述虚拟对象的显示方法。
第四方面，本公开实施例提供一种非暂态计算机可读存储介质，其特征在于，所述非暂态计算机可读存储介质存储计算机指令，当所述计算机指令被计算机执行时，使得所述计算机执行前述第一方面中的任一所述虚拟对象的显示方法。
本公开公开了一种虚拟对象的显示方法、装置、电子设备和计算机可读存储介质。其中所述虚拟对象的显示方法，其特征在于，包括：获取视频中的图像帧；在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；在显示界面的第一显示区域显示所述虚拟对象。本公开实施例提供的虚拟对象的显示方法、装置、电子设备和计算机可读存储介质，通过对视频中的图像帧进行处理，能够根据视频中的图像帧灵活地显示虚拟对象。
上述说明仅是本公开技术方案的概述，为了能更清楚了解本公开的技术手段，而可依照说明书的内容予以实施，并且为让本公开的上述和其他目的、特征和优点能够更明显易懂，以下特举较佳实施例，并配合附图，详细说明如下。
附图说明
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作一简单地介绍，显而易见地，下面描述中的附图是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的虚拟对象的显示方法实施例的流程图；
图2为本公开实施例提供的虚拟对象的关键点和/或关键点的位置的示意图；
图3为本公开实施例提供的虚拟对象的显示装置实施例的结构示意图；
图4为根据本公开实施例提供的电子设备的结构示意图。
具体实施方式
以下通过特定的具体实例说明本公开的实施方式，本领域技术人员可由本说明书所揭露的内容轻易地了解本公开的其他优点与功效。显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。本公开还可以通过另外不同的具体实施方式加以实施或应用，本说明书中的各项细节也可以基于不同观点与应用，在没有背离本公开的精神下进行各种修饰或改变。需说明的是，在不冲突的情况下，以下实施例及实施例中的特征可以相互组合。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有 其他实施例，都属于本公开保护的范围。
需要说明的是，下文描述在所附权利要求书的范围内的实施例的各种方面。应显而易见，本文中所描述的方面可体现于广泛多种形式中，且本文中所描述的任何特定结构及/或功能仅为说明性的。基于本公开，所属领域的技术人员应了解，本文中所描述的一个方面可与任何其它方面独立地实施，且可以各种方式组合这些方面中的两者或两者以上。举例来说，可使用本文中所阐述的任何数目个方面来实施设备及/或实践方法。另外，可使用除了本文中所阐述的方面中的一或多者之外的其它结构及/或功能性实施此设备及/或实践此方法。
还需要说明的是，以下实施例中所提供的图示仅以示意方式说明本公开的基本构想，图示中仅显示与本公开中有关的组件而非按照实际实施时的组件数目、形状及尺寸绘制，其实际实施时各组件的型态、数量及比例可为一种随意的改变，且其组件布局型态也可能更为复杂。
另外，在以下描述中，提供具体细节是为了便于透彻理解实例。然而，所属领域的技术人员将理解，可在没有这些特定细节的情况下实践所述方面。
本实施例提供的该虚拟对象的显示方法可以由一个虚拟对象的显示装置来执行，该装置可以实现为软件，可以实现为硬件，还可以实现为软件和硬件的组合，例如所述虚拟对象的显示装置包括计算机设备，从而通过该计算机设备来执行本实施例提供的该虚拟对象的显示方法，如本领域技术人员所理解的，计算机设备可以是台式或便携计算机设备，还可以是移动终端设备等。
图1为本公开实施例提供的虚拟对象的显示方法实施例一的流 程图，如图1所示，本公开实施例的虚拟对象的显示方法包括如下步骤：
步骤S101，获取视频中的图像帧；
在步骤S101中，虚拟对象的显示装置可以通过多种方式获取视频中的图像帧，本公开实施例对此不作限定。例如虚拟对象的显示装置包括摄像装置，从而通过摄像装置拍摄视频以获取视频中的图像帧，虚拟对象的显示装置还可以通过网络获取视频中的图像帧，例如虚拟对象的显示装置通过网络从其他存储设备或者网络摄像装置获取视频中的图像帧。如同本领域技术人员的理解，本公开实施例中的视频是由一系列图像帧组成，当播放所述视频时，可以按照所述一系列图像帧在所述视频中的先后顺序显示所述一些列图像帧，例如每秒显示24个图像帧，当然，所述图像帧还可以被称为图像。
步骤S102，在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；
虚拟对象的显示装置需要获取所述虚拟对象的关键点位置，以便显示所述虚拟对象，因此在步骤S102中，所述虚拟对象的显示装置根据与虚拟对象的关键点对应的图像特征，在步骤S101中获取的图像帧中确定虚拟对象的关键点的位置。
在一个可选的实施例中，本公开实施例的虚拟对象包括预设数量的关键点，每个虚拟对象的关键点对应有图像特征，从而根据与每个虚拟对象的关键点对应的图像特征，在所述图像帧中确定所述虚拟对象的关键点的位置。
如同本领域技术人员所理解的，本公开实施例中，图像特征可以通过颜色特征和/或形状特征来表征，即图像特征包括颜色特征和/ 或形状特征。可选的，所述图像特征包括视频对象的图像特征或图像帧对象的图像特征，从而可以根据视频对象的图像特征或图像帧对象的图像特征在所述视频中的图像帧中确定与所述视频对象的图像特征或图像帧对象的图像特征对应或匹配的关键点的位置，也就是进行关键点定位。其中，所述视频对象或者所述图像帧对象可以包括所述视频的图像帧中的任何对象，例如人物对象和物体对象等，相应地，所述图像特征可以包括人物对象的图像特征和物体对象的图像特征等。
可选的，在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置，包括：在所述图像帧中确定与所述图像特征匹配的关键点的位置，并将与所述图像特征匹配的关键点的位置作为所述虚拟对象的关键点的位置。可选的，所述图像特征包括人物对象的图像特征，从而所述虚拟对象显示装置根据与所述虚拟对象的关键点对应的人物对象的图像特征确定所述虚拟对象的关键点的位置。
以图像特征包括人物对象的图像特征为例，人物对象的图像特征可以包括人脸图像特征，手臂图像特征，骨骼图像特征，关节图像特征，器官图像特征等，从而可以根据人物对象的图像特征在图像帧中确定与这些人物对象的图像特征对应或匹配的关键点(也称为人物对象的关键点)的位置，进一步的，可以将确定出的人物对象的关键点的位置作为虚拟对象的关键点的位置。作为一个示例，虚拟对象包括五个关键点P0-P4，其中与虚拟对象的关键点P0对应的图像特征包括人脸图像特征，与虚拟对象的关键点P1对应的图像特征包括左手图像特征，与虚拟对象的关键点P2对应的图像特征包括右手图像特征，与虚拟对象的关键点P3对应的图像特征包括左脚图像特征，与 虚拟对象的关键点P4对应的图像特征包括右脚图像特征。那么在步骤S102中，将根据所述人脸图像特征在所述图像帧中确定的人物对象的关键点的位置作为P0的位置，将根据所述左手图像特征在所述图像帧中确定的人物对象的关键点的位置作为P1的位置，将根据右手图像特征在所述图像帧中确定的人物对象的关键点的位置作为P2的位置，将根据左脚图像特征在所述图像帧中确定的人物对象的关键点的位置作为P3的位置，将根据所述右脚图像特征在所述图像帧中确定的人物对象的关键点的位置作为P4的位置。
在上述根据与虚拟对象的关键点对应的人物对象的图像特征确定所述虚拟对象的关键点的位置的过程中，作为一个可选的实施例，首先可以使用颜色特征定位人物对象的位置，这样可以将人物对象从背景中分割出来，进而根据所述人物对象的图像特征确定与所述人物对象的图像特征对应的人物对象的关键点的位置，从而将确定出的人物对象的关键点的位置作为所述虚拟对象的关键点的位置。具体的，视频中的图像帧是通过摄像装置拍摄获得，摄像装置的图像传感器在拍摄过程中对于每一个图像帧都会记录颜色信息以及颜色的位置信息，然后将颜色信息以及颜色的位置信息输入基于卷积神经网络的分类器，如果图像帧中存在人物对象，那么通过所述分类器就能够识别出人物对象的轮廓，这样就将人物对象从背景中分割出来，之后根据所述分割出的人物对象，并且基于前述的图像特征在图像帧中进行搜索比对，确定所述人物对象的关键点的位置，这也可以称为确定所述人物对象的关键点，因为确定所述人物对象的关键点的过程就是根据与该人物对象的关键点对应的图像特征在图像帧上进行定位的过程。如同本领域技术人员所理解的，人物对象的关键点是基于与人物对象的关键点对应的图像特征确定的，因此所述人物对象的关键点对应于 所述图像特征，所述图像特征也对应所述人物对象的关键点；由于人物对象的关键点在图像中仅占据非常小的面积(通常只有几个至几十个像素的大小)，与人物对象的关键点对应的图像特征在图像上所占据的区域通常也是非常有限和局部的，目前用的特征提取方式有两种：(1)沿轮廓垂向的一维范围图像特征提取；(2)关键点方形邻域的二维范围图像特征提取。上述两种方式有很多种实现方法，如ASM和AAM类方法、统计能量函数类方法、回归分析方法、深度学习方法、分类器方法、批量提取方法等等。
作为一个可选的实施例，可以通过坐标表示人物对象的关键点的位置和虚拟对象的关键点的位置。如同本领域技术人员的理解，拍摄视频的摄像装置在拍摄过程中会根据画幅设定相应的坐标系，该坐标系也称为所述视频对应的坐标系或者所述视频中的图像帧对应的坐标系，从而对于拍摄的图像帧的每个像素，都可以基于该坐标系标记相应的坐标，那么根据人物对象的图像特征从所述图像帧中确定的人物对象的关键点的位置也可以基于所述人物对象的关键点所占据的一个或多个像素的坐标来表示(如果所述人物对象的关键点占据多个像素，则可以通过所述多个像素的坐标的平均值来表示所述人物对象的关键点的位置)，相应地，所述虚拟对象的关键点的位置也可以基于所述人物对象的关键点所占据的一个或多个像素的坐标来表示。例如某个人物对象的关键点的坐标可以表示为(x，y)，其中x和y分别为横坐标和纵坐标；还例如对于具有深度传感器的摄像装置所拍摄的视频的图像帧，还会对每个像素记录深度信息，那么所述图像帧中的某个人物对象的关键点的坐标还可以表示为(x，y，z)，其中x和y分别为横坐标和纵坐标，z为深度坐标，相应地，由于将所述人物对象的关键点的位置作为所述虚拟对象的关键点的位置，那么所述虚 拟对象的关键点的坐标就可以表示为(x，y)，(x，y，z)。为便于后续的描述和理解，“人物对象的关键点在图像帧中的坐标”，“虚拟对象的关键点在图像帧中的坐标”，“人物对象的关键点的坐标”，和“虚拟对象的关键点的坐标”的表述时，该坐标所对应的坐标系均指前述拍摄视频的摄像装置在拍摄过程中根据画幅所设定的坐标系，但是本领域技术人员可以理解，对于视频中的图像帧，也可以建立其他坐标系来表示所述人物对象的关键点的位置和所述虚拟对象的关键点的位置，本公开实施例对此不作限定。
如前所述，本公开实施例可以使用颜色特征定位人物对象的位置，以将人物对象从背景中分割出来，然后再根据人物对象的图像特征从图像帧中确定人物对象的关键点的位置，从而将所述人物对象的关键点的位置作为所述虚拟对象的关键点的位置。但是，这不应理解为对本公开实施例的限制，本公开实施例还可以通过其他方法从图像帧中提取人物对象的关键点，例如不需要使用颜色特征定位人物的对象的位置，而直接根据与虚拟对象的关键点对应的人物对象的图像特征从图像帧中确定与所述人物对象的图像特征对应的关键点(即前述的人物对象的关键点)的位置，并将所述位置作为所述虚拟对象的关键点的位置。
图2所示为步骤S102中根据人物对象的图像特征确定虚拟对象的关键点的位置的示意图。图2所示包括确定的A到K共11个虚拟对象的关键点，每个虚拟对象的关键点的位置可以通过坐标表示。其中虚拟对象的关键点A和B(的位置)是根据躯干图像特征从所述图像帧中确定的，虚拟对象的关键点C(的位置)是根据头部图像特征从所述图像帧中确定的，虚拟对象的关键点D和E(的位置)是根据一条手臂的图像特征从所述图像帧中确定的，虚拟对象的关键点F和 G(的位置)是根据另一条手臂的图像特征从所述图像帧中确定的，虚拟对象的关键点H和I(的位置)是根据一条腿部的图像特征从所述图像帧中确定的，虚拟对象的关键点J和K(的位置)是根据另一条腿部的图像特征从所述图像帧中确定的。本领域有技术人员可以理解，根据需要和图像特征的不同等因素，可以提取更多或者更少的与所述图像特征对应的关键点，在一些实施例中，这些虚拟对象的关键点和/或虚拟对象的关键点的位置构成所述虚拟对象的姿势或姿态。
步骤S103，根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象。
本公开实施例中的虚拟对象可以是任何二维或者三维的虚拟对象，本公开实施例对此不做具体的限制，任何可以在显示界面上显示的虚拟对象都可以引入到本公开中，相应地，为了显示所述二维或者三维的虚拟对象，需要在步骤S103中对所述虚拟对象进行渲染。
在一个可选的实施例中，与所述虚拟对象的关键点对应的渲染信息包括如下信息中的至少一种：虚拟对象的像素位置信息，虚拟对象的像素颜色信息，虚拟对象的轮廓信息，虚拟对象的贴图信息，虚拟对象的建模规则信息。由于在步骤S102中确定了虚拟对象的关键点的位置，进而可以基于虚拟对象的关键点的位置和与所述虚拟对象的关键点对应的渲染信息渲染所述虚拟对象，例如可以根据虚拟对象的关键点位置和轮廓信息确定该虚拟关键点位置所对应的虚拟对象的部位的大小，并根据该大小调整贴图大小，以便在像素级别上渲染出完整的虚拟对象；可以根据所述虚拟对象的关键点的位置和建模规则信息通过三角形或多边形建模以形成虚拟对象的轮廓，并施加贴图以实现对虚拟对象的渲染；还可以通过生成对抗网络(GAN)或渐进结构条件GAN等根据所述虚拟对象的关键点的位置，和各虚拟对象的关 键点所对应的贴图信息、按照缩放比例渲染虚拟对象。本领域技术人员可知，任何根据关键点和渲染信息渲染二维或者三维对象的方法都可以引入本公开，本公开实施例对此不做具体的限制。
步骤S104，在显示界面的第一显示区域显示所述虚拟对象。
在通过步骤S103完成对虚拟对象的渲染之后，执行步骤S104，在显示界面的第一显示区域显示所述虚拟对象。在一个可选的实施例中，在所述显示界面的第一显示区域显示所述虚拟对象包括：在所述显示界面的第一显示区域放大或缩小显示所述虚拟对象。例如完成步骤S103对虚拟对象的渲染后，可以得到通过像素的位置和像素的颜色表示的虚拟对象，那么在步骤S104中，可以在显示界面的第一显示区域，根据所述虚拟对象的像素位置和像素颜色，放大或缩小显示所述虚拟对象。如本领域技术人员所理解的，在步骤S104中通过显示界面显示所述虚拟对象时，可以对所述虚拟对象的像素位置通过坐标变换的方式，在所述显示界面的第一显示区域放大或缩小显示所述虚拟对象；如果所述显示界面采用了新的坐标系，可以对所述虚拟对象的像素通过坐标系变换的方式，在所述显示界面的第一显示区域放大或缩小显示所述虚拟对象；另外在放大或缩小显示所述虚拟对象时还可能涉及到分辨率的升高或者降低，但是本公开实施例对显示所述虚拟对象的方法做出限定。
应用上述实施方式，通过对视频中的图像帧进行处理，并根据图像帧的人物对象信息显示虚拟对象，不需要用户具备专业的美工技术和计算机图像处理技术，就能够实现根据视频中的图像帧灵活地显示虚拟对象。
由于所述虚拟对象的关键点的位置是根据与所述虚拟对象的关 键点对应的图像特征在所述图像帧中确定的，因此在步骤S103中渲染并在步骤S104中显示的所述虚拟对象的动作、姿势、或者姿态可以与所述图像特征存在关联。以图2中根据人物对象的图像特征在图像帧中确定的虚拟对象的关键点的位置的示意图为例，图2中的虚拟对象的关键点C的位置是根据与其对应的头部图像特征从所述图像帧中确定的，也就是说将所述图像帧中的人物对象的头部关键点所在的位置作为所述虚拟对象的关键点C的位置，相应地，将所述图像帧中的人物对象的躯干，手部，和腿部关键点的位置分别作为所述虚拟对象的关键点A和B，D和E，F和G，H和I，以及J和K的位置，因此在根据上述虚拟对象的关键点的位置和与所述虚拟对象的关键点对应的渲染信息渲染所述虚拟对象的过程中，如果与关键点C对应的渲染信息也包括与虚拟对象的头部对应的渲染信息，相应地，与关键点A和B，D和E，F和G，H和I，以及J和K对应的渲染信息也分别包括与虚拟对象的躯干，手部，和腿部对应的的渲染信息，那么所渲染出的虚拟对象将会与所述图像帧中的人物对象具有一致的或者对应的动作、姿势、或者姿态。相应地，在一个可选的实施例中，可以根据所述图像帧在所述视频中的先后顺序，对所述图像帧中每一个图像帧执行本公开的虚拟对象的显示方法的步骤S102-S104，从而按照所述先后顺序显示根据每个图像帧所渲染的所述虚拟对象，当按照预设的时间间隔(例如24帧每秒)在所述第一显示区域显示根据每个图像帧所渲染的所述虚拟对象时，相当于播放所述虚拟对象运动的视频，在该播放的视频中所述虚拟对象运动的动作、姿势、或者姿态与步骤S101对应的视频中的人物对象的动作、姿势、或者姿态保持一致或对应。
在一个可选的实施例中，在所述显示界面的第二显示区域显示所 述图像帧。可选的，所述第一显示区域与所述第二显示区域重叠，例如所述第二显示区域充满所述显示界面，而所述第一显示区域占据所述显示界面的角落如右上角；可选的，所述第一显示区域与所述第二显示区域不重叠，例如所述第一显示区域充满所述显示界面的左半边，所述第二显示区域充满所述显示界面的右半边。
在又一个可选的实施例中，在所述显示界面的第二显示区域播放所述步骤S101中的视频，在所述显示界面的第一显示区域所显示的虚拟对象是根据所述第二显示区域当前所显示的图像帧渲染的。也就是说，在所述显示界面的第二显示区域播放所述步骤S101中的视频时，是按照预设的速度(例如24帧每秒)依序显示所述视频中的图像帧，而在所述显示界面的第一显示区域所显示的虚拟对象，是在所述第二显示区域当前所显示的图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置(即步骤S102)，然后根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置所渲染的虚拟对象。相当于在所述显示界面的第一显示区域播放所述虚拟对象运动的视频，并且在所述显示界面的第二显示区域播放所述步骤S101中的视频，在所述第一显示区域播放的视频中的所述虚拟对象运动的动作、姿势、或者姿态与在所述显示界面的第二区域中播放的视频中的人物对象的动作、姿势、或者姿态保持一致或对应。
图3所示为本公开实施例提供的虚拟对象的显示装置300实施例的结构示意图，如图3所示，所述装置包括视频获取模块301，确定模块302，渲染模块303，和显示模块304。
其中所述视频获取模块301，用于获取视频中的图像帧；所述确定模块302，用于在所述图像帧中根据与虚拟对象的关键点对应的图 像特征确定所述虚拟对象的关键点的位置；所述渲染模块303，用于根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；所述显示模块304，用于在显示界面的第一显示区域显示所述虚拟对象。
在一个可选的实施例中，所述显示模块304，还用于在所述显示界面的第二显示区域显示所述图像帧。
图3所示装置可以执行图1所示实施例的方法，本实施例未详细描述的部分，可参考对图1所示实施例的相关说明。该技术方案的执行过程和技术效果参见图1所示实施例中的描述，在此不再赘述。
下面参考图4，其示出了适于用来实现本公开实施例的电子设备400的结构示意图。本公开实施例中的电子设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图4示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图4所示，电子设备400可以包括处理装置(例如中央处理器、图形处理器等)401，其可以根据存储在只读存储器(ROM)402中的程序或者从存储装置408加载到随机访问存储器(RAM)403中的程序而执行各种适当的动作和处理。在RAM 403中，还存储有电子设备400操作所需的各种程序和数据。处理装置401、ROM 402以及RAM 403通过总线或通信线路404彼此相连。输入/输出(I/O)接口405也连接至总线或通信线路404。
通常，以下装置可以连接至I/O接口405：包括例如触摸屏、触 摸板、键盘、鼠标、图像传感器、麦克风、加速度计、陀螺仪等的输入装置406；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置407；包括例如磁带、硬盘等的存储装置408；以及通信装置409。通信装置409可以允许电子设备400与其他设备进行无线或有线通信以交换数据。虽然图4示出了具有各种装置的电子设备400，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置409从网络上被下载和安装，或者从存储装置408被安装，或者从ROM 402被安装。在该计算机程序被处理装置401执行时，执行本公开实施例的方法中限定的上述功能。
需要说明的是，本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如包括但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以 包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备执行上述实施例中的虚拟对象的显示方法。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、 方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元的名称在某种情况下并不构成对该单元本身的限定。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。

Claims (11)

  1. 一种虚拟对象的显示方法，其特征在于，包括：
    获取视频中的图像帧；
    在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；
    根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；
    在显示界面的第一显示区域显示所述虚拟对象。
  2. 根据权利要求1所述的虚拟对象的显示方法，其特征在于，在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定虚拟对象的关键点的位置，包括：
    在所述图像帧中确定与所述图像特征匹配的关键点的位置；
    将与所述图像特征匹配的关键点的位置作为所述虚拟对象的关键点的位置。
  3. 根据权利要求1所述的虚拟对象的显示方法，其特征在于，所述图像特征包括人物对象的图像特征。
  4. 根据权利要求1所述的许你对象的显示方法，其特征在于，所述虚拟对象的渲染信息包括如下信息中的一种或多种：
    所述虚拟对象的像素位置信息；
    所述虚拟对象的像素颜色信息；
    所述虚拟对象的轮廓信息；
    所述虚拟对象的贴图信息；
    所述虚拟对象的建模规则信息。
  5. 根据权利要求1所述的虚拟对象的显示方法，其特征在于，所述在显示界面的第一显示区域显示所述虚拟对象，包括：
    在所述显示界面的第一显示区域放大或缩小显示所述虚拟对象。
  6. 根据权利要求1-5中任一所述的虚拟对象的显示方法，其特征在于，所述方法还包括：
    在所述显示界面的第二显示区域显示所述图像帧。
  7. 根据权利要求6所述的虚拟对象的显示方法，其特征在于，所述第一显示区域与所述第二显示区域重叠。
  8. 根据权利要求6所述的虚拟对象的显示方法，其特征在于，所述第一显示区域与所述第二显示区域不重叠。
  9. 一种虚拟对象的显示装置，其特征在于，包括：
    视频获取模块，用于获取视频中的图像帧；
    确定模块，用于在所述图像帧中根据与虚拟对象的关键点对应的图像特征确定所述虚拟对象的关键点的位置；
    渲染模块，用于根据与所述虚拟对象的关键点对应的渲染信息和所述虚拟对象的关键点的位置渲染所述虚拟对象；
    显示模块，在显示界面的第一显示区域显示所述虚拟对象。
  10. 一种电子设备，包括：
    存储器，用于存储计算机可读指令；以及
    处理器，用于运行所述计算机可读指令，使得所述处理器运行时实现根据权利要求1-8中任意一项所述的虚拟对象的显示方法。
  11. 一种非暂态计算机可读存储介质，用于存储计算机可读指令，当所述计算机可读指令由计算机执行时，使得所述计算机执行权利要求1-8中任意一项所述的虚拟对象的显示方法。
PCT/CN2020/074447 2019-03-20 2020-02-06 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质 Ceased WO2020186935A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910215466.6A CN110058685B (zh) 2019-03-20 2019-03-20 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质
CN201910215466.6 2019-03-20

Publications (1)

Publication Number Publication Date
WO2020186935A1 true WO2020186935A1 (zh) 2020-09-24

Family

ID=67316302

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/074447 Ceased WO2020186935A1 (zh) 2019-03-20 2020-02-06 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN110058685B (zh)
WO (1) WO2020186935A1 (zh)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110058685B (zh) * 2019-03-20 2021-07-09 北京字节跳动网络技术有限公司 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质
CN110570502A (zh) * 2019-08-05 2019-12-13 北京字节跳动网络技术有限公司 显示图像帧的方法、装置、电子设备和计算机可读存储介质
CN110660032A (zh) * 2019-09-24 2020-01-07 Oppo广东移动通信有限公司 一种对象屏蔽方法、对象屏蔽装置及电子设备
CN110807728B (zh) * 2019-10-14 2022-12-13 北京字节跳动网络技术有限公司 对象的显示方法、装置、电子设备及计算机可读存储介质
CN110827379A (zh) * 2019-10-31 2020-02-21 北京字节跳动网络技术有限公司 虚拟形象的生成方法、装置、终端及存储介质
CN112784622B (zh) * 2019-11-01 2023-07-25 抖音视界有限公司 图像的处理方法、装置、电子设备及存储介质
CN110910512B (zh) * 2019-11-29 2024-04-30 北京达佳互联信息技术有限公司 虚拟物体自适应调整方法、装置、计算机设备和存储介质
CN111210485B (zh) * 2020-01-06 2023-03-28 北京字节跳动网络技术有限公司 图像的处理方法、装置、可读介质和电子设备
CN111669502B (zh) 2020-06-19 2022-06-24 北京字节跳动网络技术有限公司 目标对象显示方法、装置及电子设备
CN112199016B (zh) * 2020-09-30 2023-02-21 北京字节跳动网络技术有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN112333464B (zh) * 2020-10-30 2022-08-02 北京字跳网络技术有限公司 一种交互数据生成方法、装置以及计算机存储介质
CN112714337A (zh) * 2020-12-22 2021-04-27 北京百度网讯科技有限公司 视频处理方法、装置、电子设备和存储介质
CN114419298B (zh) * 2022-01-21 2026-02-06 北京字跳网络技术有限公司 虚拟物体的生成方法、装置、设备及存储介质
CN115442637A (zh) * 2022-09-06 2022-12-06 北京字跳网络技术有限公司 直播特效渲染方法、装置、设备、可读存储介质及产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102221886A (zh) * 2010-06-11 2011-10-19 微软公司 通过化身与用户界面交互
CN107831902A (zh) * 2017-11-23 2018-03-23 腾讯科技（上海）有限公司 一种运动控制方法及其设备、存储介质、终端
CN108629843A (zh) * 2017-03-24 2018-10-09 成都理想境界科技有限公司 一种实现增强现实的方法及设备
US20190026922A1 (en) * 2017-07-24 2019-01-24 Visom Technology, Inc. Markerless augmented reality (ar) system
CN110058685A (zh) * 2019-03-20 2019-07-26 北京字节跳动网络技术有限公司 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108273265A (zh) * 2017-01-25 2018-07-13 网易（杭州）网络有限公司 虚拟对象的显示方法及装置
CN108229282A (zh) * 2017-05-05 2018-06-29 商汤集团有限公司 关键点检测方法、装置、存储介质及电子设备
US20240412460A1 (en) * 2017-06-21 2024-12-12 Quantum Interface, Llc Systems, apparatuses, interfaces, and methods for virtual control constructs, eye movement object controllers, and virtual training
CN108227931A (zh) * 2018-01-23 2018-06-29 北京市商汤科技开发有限公司 用于控制虚拟人物的方法、设备、系统、程序和存储介质
CN108537867B (zh) * 2018-04-12 2020-01-10 北京微播视界科技有限公司 根据用户肢体运动的视频渲染方法和装置
CN109035373B (zh) * 2018-06-28 2022-02-01 北京市商汤科技开发有限公司 三维特效程序文件包的生成及三维特效生成方法与装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102221886A (zh) * 2010-06-11 2011-10-19 微软公司 通过化身与用户界面交互
CN108629843A (zh) * 2017-03-24 2018-10-09 成都理想境界科技有限公司 一种实现增强现实的方法及设备
US20190026922A1 (en) * 2017-07-24 2019-01-24 Visom Technology, Inc. Markerless augmented reality (ar) system
CN107831902A (zh) * 2017-11-23 2018-03-23 腾讯科技（上海）有限公司 一种运动控制方法及其设备、存储介质、终端
CN110058685A (zh) * 2019-03-20 2019-07-26 北京字节跳动网络技术有限公司 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质

Also Published As

Publication number Publication date
CN110058685A (zh) 2019-07-26
CN110058685B (zh) 2021-07-09

Similar Documents

Publication Publication Date Title
WO2020186935A1 (zh) 虚拟对象的显示方法、装置、电子设备和计算机可读存储介质
JP7604669B2 (ja) 特殊効果展示方法、装置、機器および媒体
CN111242881B (zh) 显示特效的方法、装置、存储介质及电子设备
US11776209B2 (en) Image processing method and apparatus, electronic device, and storage medium
WO2020248900A1 (zh) 全景视频的处理方法、装置及存储介质
WO2020228406A1 (zh) 图像风格化生成方法、装置及电子设备
WO2020220809A1 (zh) 目标对象的动作识别方法、装置和电子设备
CN112672185B (zh) 基于增强现实的显示方法、装置、设备及存储介质
JP2022505118A (ja) 画像処理方法、装置、ハードウェア装置
US20220159197A1 (en) Image special effect processing method and apparatus, and electronic device and computer readable storage medium
CN111833459B (zh) 一种图像处理方法、装置、电子设备及存储介质
CN110069191B (zh) 基于终端的图像拖拽变形实现方法和装置
CN112132859B (zh) 贴纸生成方法、装置、介质和电子设备
CN111862349B (zh) 虚拟画笔实现方法、装置和计算机可读存储介质
CN109600559B (zh) 一种视频特效添加方法、装置、终端设备及存储介质
CN114782593B (zh) 图像处理方法、装置、电子设备及存储介质
CN113163135A (zh) 视频的动画添加方法、装置、设备及介质
CN115937010A (zh) 一种图像处理方法、装置、设备及介质
CN111652675A (zh) 展示方法、装置和电子设备
CN112270242B (zh) 轨迹的显示方法、装置、可读介质和电子设备
CN116527993A (zh) 视频的处理方法、装置、电子设备、存储介质和程序产品
JP2023504803A (ja) 画像合成方法、装置、およびストレージ媒体
WO2021208764A1 (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
WO2020077912A1 (zh) 图像处理方法、装置、硬件装置
CN114820878B (zh) 图像编辑方法及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20773022

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07/02/2022)

122 Ep: pct application non-entry in european phase

Ref document number: 20773022

Country of ref document: EP

Kind code of ref document: A1