WO2023045710A1

WO2023045710A1 - Multimedia display and matching methods and apparatuses, device and medium

Info

Publication number: WO2023045710A1
Application number: PCT/CN2022/115521
Authority: WO
Inventors: 黄造军; 徐之俊; 冯宇飞; 邓子建; 吴铭泽
Original assignee: 北京字节跳动网络技术有限公司
Priority date: 2021-09-27
Filing date: 2022-08-29
Publication date: 2023-03-30
Also published as: CN113870133B; CN113870133A

Abstract

The preset disclosure relates to multimedia display and matching methods and apparatuses, a device and a medium. The multimedia display method comprises: receiving original multimedia data; performing special effect editing on the original multimedia data to obtain multimedia data to be matched; generating synthetic multimedia data on the basis of the multimedia data to be matched and target multimedia data, the target multimedia data being obtained by matching according to a first multimedia feature of the multimedia data to be matched; and displaying the synthetic multimedia data. According to the embodiments of the present disclosure, the beautification effect of multimedia data is enriched, and the display appeal of the multimedia data is improved. Moreover, users may interact by means of the multimedia data, diversified interaction between the users is implemented, and the use experience of the users is improved.

Description

Multimedia display and matching method, device, equipment and medium

Cross References to Related Applications

This application is based on the Chinese patent application with the application number 202111136435.5, the filing date is September 27, 2021, and the title is "Multimedia Display and Matching Method, Device, Equipment and Medium", and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

technical field

The present disclosure relates to the technical field of multimedia processing, in particular to a multimedia display and matching method, device, equipment and medium.

Background technique

With the rapid development of computer technology and mobile communication technology, various network platforms based on electronic equipment have been widely used, greatly enriching people's daily life. More and more users are willing to beautify multimedia data such as images or videos on network platforms to obtain photos or videos with satisfactory effects.

At present, although users can use preset special effect templates to beautify multimedia data, the interaction mode between users is relatively simple, which lacks interest and reduces user experience.

Contents of the invention

In order to solve the above technical problems or at least partly solve the above technical problems, the present disclosure provides a multimedia display and matching method, device, equipment and medium.

In a first aspect, the present disclosure provides a multimedia display method, including:

receiving raw multimedia data;

Perform special effect editing on the original multimedia data to obtain the multimedia data to be matched;

Based on the multimedia data to be matched and the target multimedia data, synthetic multimedia data is generated, and the target multimedia data is obtained by matching the first multimedia feature of the multimedia data to be matched;

Display composite multimedia data.

In a second aspect, the present disclosure provides a multimedia matching method, including:

receiving the multimedia data to be matched, the multimedia data to be matched is obtained by performing special effect processing on the original multimedia data;

Extracting the first multimedia feature from the multimedia data to be matched;

Obtain a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

Among the plurality of candidate multimedia data, the target multimedia data matching the first multimedia feature is queried, and the target multimedia data is used to generate merged multimedia data with the multimedia data to be matched.

In a third aspect, the present disclosure provides a multimedia display device, including:

a data receiving unit configured to receive original multimedia data;

The special effect editing unit is configured to perform special effect editing on the original multimedia data to obtain the multimedia data to be matched;

The data synthesis unit is configured to generate synthesized multimedia data based on the multimedia data to be matched and the target multimedia data, and the target multimedia data is obtained by matching the first multimedia feature of the multimedia data to be matched;

The data display unit is configured to display synthesized multimedia data.

In a fourth aspect, the present disclosure provides a multimedia matching device, including:

The data receiving unit is configured to receive the multimedia data to be matched, and the multimedia data to be matched is obtained after performing special effect processing on the original multimedia data;

A feature extraction unit configured to extract a first multimedia feature from the multimedia data to be matched;

A data acquisition unit configured to acquire a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

The data query unit is configured to query target multimedia data that matches the first multimedia feature among multiple candidate multimedia data, and the target multimedia data is used to generate merged multimedia data with the multimedia data to be matched.

In a fifth aspect, the present disclosure provides a computing device, including:

processor;

memory for storing executable instructions;

Wherein, the processor is used to read executable instructions from the memory, and execute the executable instructions to implement the multimedia display method of the first aspect, or to implement the multimedia matching method of the second aspect.

In a sixth aspect, the present disclosure provides a computer-readable storage medium, the storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the multimedia display method of the first aspect, or implements the second aspect The multimedia matching method.

Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

The multimedia display and matching method, device, device and medium of the disclosed embodiments generate and display synthesized multimedia data based on the edited multimedia data to be matched and target multimedia data after performing special effect editing on the received original multimedia data. Since the target pair of media data is obtained based on the first multimedia feature matching of the multimedia data to be matched, in addition to the special effects in the multimedia data to be matched, the synthesized multimedia data obtained based on the original multimedia data may also include The multimedia data matches the content of the target multimedia data, so that the multimedia data image has multiple elements, enriches the beautification effect of the multimedia data, improves the interest of the multimedia data display, and enables users to interact through the multimedia data to achieve It improves the diversity of interactions between users and improves the user experience.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 shows an architecture diagram of a multimedia display system provided by an embodiment of the present disclosure;

FIG. 2 shows an architecture diagram of another multimedia display system provided by an embodiment of the present disclosure;

FIG. 3 shows a schematic flowchart of a multimedia display method provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of a shooting preview interface provided by an embodiment of the present disclosure;

Fig. 5 shows a schematic diagram of a special effect editing interface provided by an embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of a display interface of multimedia data to be matched provided by an embodiment of the present disclosure;

FIG. 7 shows a schematic diagram of matching logic of multimedia data provided by an embodiment of the present disclosure;

FIG. 8 shows a schematic diagram of a display interface for synthesizing multimedia data provided by an embodiment of the present disclosure;

FIG. 9 shows a schematic flowchart of another multimedia display method provided by an embodiment of the present disclosure;

FIG. 10 shows a schematic flowchart of another multimedia display method provided by an embodiment of the present disclosure;

FIG. 11 shows a schematic flowchart of a multimedia matching method provided by an embodiment of the present disclosure;

FIG. 12 shows a schematic structural diagram of a multimedia display device provided by an embodiment of the present disclosure;

FIG. 13 shows a schematic structural diagram of a multimedia matching device provided by an embodiment of the present disclosure;

Fig. 14 shows a schematic structural diagram of a computing device provided by an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variants are open-ended, i.e. "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

With the rapid development of computer technology and mobile communication technology, various network platforms based on electronic equipment have been widely used, greatly enriching people's daily life. More and more users are willing to beautify and shoot multimedia data such as images or videos on a network platform to obtain photos or videos with satisfactory effects.

Currently, users can use preset special effect templates to beautify multimedia data. For example, preset stickers or preset special effects can be added to the captured pictures.

However, in this way, the user can only select a special effect template from preset special effect tools, and the beautification effect is relatively simple, lacks interest, and reduces user experience.

In order to solve the above problems, embodiments of the present disclosure provide a multimedia display and matching method, device, device and medium capable of displaying synthesized multimedia data generated by matching multimedia data and target multimedia data.

The multimedia display method provided by the present disclosure can be applied to the architecture shown in FIG. 1 and FIG. 2 , and will be described in detail with reference to FIG. 1 and FIG. 2 .

Fig. 1 shows a structure diagram of a multimedia display system provided by an embodiment of the present disclosure.

As shown in Figure 1, the multimedia display system may include at least one electronic device 101 at the client end and at least one server 102 at the server end. The electronic device 101 can establish a connection with the server 102 and exchange information through a network protocol such as Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS). Among them, the electronic device 101 may be a mobile phone, a tablet computer, a desktop computer, a notebook computer, a vehicle terminal, a wearable device, an all-in-one machine, a smart home device, and other devices with communication functions, or it may be a device simulated by a virtual machine or a simulator. . The server 102 may be a device with storage and computing functions such as a cloud server or a server cluster.

Based on the above architecture, the user can edit original multimedia data with special effects on the electronic device 101 through a specific service platform, and generate and display synthesized multimedia data. Wherein, the specific service platform may be a specific application program or a specific website, such as a social platform or a video playing platform with social functions.

In some embodiments, after the user logs in to the specific service platform through the electronic device 101, the electronic device 101 can obtain original multimedia data such as images or videos, and edit the original multimedia data with special effects to obtain the multimedia data to be matched. And after obtaining multiple pieces of candidate multimedia data including the target multimedia data P11 from the server 102, the electronic device 101 may query the target multimedia data P11 from the candidate multimedia data based on the first multimedia feature of the multimedia data to be matched. Then, the electronic device 101 may generate composite multimedia data P12 by matching the target multimedia data obtained based on the first multimedia feature of the multimedia data to be matched with the multimedia data to be matched. Optionally, continuing to refer to FIG. 1 , the electronic device 101 may upload the generated synthesized multimedia data P12 to the server 102 .

In some other embodiments, the electronic device 101 may upload the multimedia data to be matched to the server 102 . Then, after receiving the multimedia data to be matched, the server 102 can match the target multimedia data P11 from multiple pieces of candidate multimedia data, and send the target multimedia data P11 to the electronic device 101 . Then, the electronic device 101 may generate synthesized multimedia data P12 based on the target multimedia data obtained by matching the first multimedia feature of the multimedia data to be matched with the multimedia data to be matched.

In addition, the multimedia display method provided by the present disclosure can be applied to a specific scenario where users of multiple electronic devices interact through multimedia data, and the architecture shown in FIG. 2 will be described below.

FIG. 2 shows a structural diagram of another multimedia display system provided by an embodiment of the present disclosure.

As shown in FIG. 2 , the multimedia display system may include at least one first electronic device 201 and at least one second electronic device 202 on the client side, and at least one server 203 on the server side. The first electronic device 201, the second electronic device 202, and the server 203 can respectively establish connections and perform information exchange through network protocols such as HTTPS. Wherein, the first electronic device 201 and the second electronic device 202 may be devices with communication functions such as mobile phones, tablet computers, desktop computers, notebook computers, vehicle-mounted terminals, wearable devices, all-in-one machines, and smart home devices, or It is a device simulated by a virtual machine or an emulator. The server 203 may be a device with storage and computing functions such as a cloud server or a server cluster.

Based on the above architecture, the first user can log in to a specific service platform on the first electronic device 201 , and the second user can log in to the same specific service platform on the second electronic device 202 . During the interaction between the first user and the first user through the specific service platform, the second user can use the second electronic device 202 to send the request to the first user through the server 203 of the specific social platform in the specific social platform. The target multimedia data P22 subjected to synthesis processing. Wherein, the specific social platform may be a specific application or a specific website with social functions.

In one embodiment, after the second user sends the target multimedia data P22 edited by special effects to the server 203 through the second electronic device 202, the server 203 may send the candidate multimedia data including the target multimedia data P22 to the first electronic equipment 201. If the first electronic device 201 determines that the target multimedia data P21 matches the to-be-matched multimedia data after special effect processing, it can generate synthesized multimedia data P23 and send the synthesized multimedia data P21 to the second electronic device 202 through the server 203 .

In another embodiment, the server 23 receives the specially edited multimedia data P21 sent by the first user through the first electronic device 201 and the specially edited multimedia data P21 sent by the second user through the second electronic device 202 . After the target multimedia data P22, if the target to-be-matched multimedia data P22 matches the to-be-matched multimedia data P21, then the target multimedia data P22 is sent to the first electronic device 201. After the first electronic device 201 generates the synthesized multimedia data P23 , it sends the synthesized multimedia data P21 to the second electronic device 202 through the server 203 .

After introducing the architecture of the multimedia display system according to the embodiment of the present disclosure through FIG. 1 and FIG. 2 , the multimedia display method provided by the embodiment of the present disclosure will be described below with reference to FIG. 3 to FIG. 8 .

Fig. 3 shows a schematic flowchart of a multimedia display method provided by an embodiment of the present disclosure.

In the embodiment of the present disclosure, the multimedia display method may be executed by an electronic device. Among them, the electronic equipment may include but not limited to such as mobile phone, notebook computer, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), vehicle terminal (such as vehicle navigation terminal) , wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc.

As shown in Fig. 3, the multimedia display method may include the following steps.

S310. Receive original multimedia data.

In the embodiment of the present disclosure, the user may trigger related operations in the target application program when he wants to edit special effects on the image or wants to, and the electronic device may receive the original multimedia data in response to the related triggered operations. Wherein, the target application program may be a social platform or a video publishing platform. Specifically, the original multimedia data may be multimedia data including visual information, such as video data or image data.

In some embodiments, the original multimedia data may be collected by the user in real time. Correspondingly, the above-mentioned related operation may be an operation of opening the shooting page by the user. Alternatively, the above related operation may be a shooting operation performed by the user on the shooting page. Alternatively, the above related operations may be triggered by the user on the multimedia synthesis function on the live broadcast page or the shooting page.

In other embodiments, the original multimedia data may be locally stored by the electronic device. Correspondingly, the relevant operation may be the user's selection operation on an image or video in the electronic album.

In yet other embodiments, the original multimedia data may be downloaded by the user. Correspondingly, the related operation may be a user's download operation for images or videos in a download page of a browser, a target application program, or a third-party application program.

In still other embodiments, the original multimedia data may be sent to the electronic device by other devices. Correspondingly, after the electronic device receives the multimedia data sent by other devices, it can use it as the original multimedia data.

In some embodiments, the original multimedia data may be multimedia data including target objects such as people, animals, plants or objects. For example, the original multimedia data may be a user's selfie, a user's selfie video, and the like. Optionally, the original multimedia data may include a partial image or an overall image of the target object. For example, the original multimedia data may only include a person's face image, or include images of a face and other body parts.

S320. Perform special effect editing on the original multimedia data to obtain multimedia data to be matched.

In the embodiment of the present disclosure, after receiving the original multimedia data, the electronic device may perform special effect editing on the original multimedia data in response to the user's trigger operation of the special effect editing function or the multimedia co-shooting function to obtain the multimedia data to be matched.

In some embodiments, special effect editing can change the features of the target object itself in the original multimedia data by adding features or replacing original features, or can be used to change the features of the accessory parts of the target object. Specifically, special effect editing may be to perform special effect editing on the original multimedia image by using at least one of special effect editing tools such as beautification, image modification, special effect props, filters, image style transfer tools, and stickers. Wherein, the special effect editing tool may be provided by a target application program, a third-party application program, a web page, and the like. It should be noted that, in order to facilitate the description of the subsequent part, in the following part of the embodiments of the present disclosure, the target object after the special effect processing in the multimedia data to be matched is referred to as the first special effect object.

In one example, the electronic device can change the face features of the target object by means of beautification, image modification, special effects and props. For example, you can adjust the features of the target object's facial contours, eyes, skin, nose, mouth, etc.

In another example, features such as height, overall or partial fatness and thinness of the target object can be changed through functions such as beautification, image modification, and special effects props.

In some other embodiments, the features of accessory components such as clothing, headgear, glasses, makeup, masks, and facial special effects that do not change the original features of the face can be added or changed through texture maps, special effect props, etc. Among them, the facial special effects that do not change the features of the original parts of the face may include animal beards and the like.

In some other embodiments, the image style of the entire original multimedia data, or the overall or partial image style of the target object can be style transferred by using an image style transfer tool or filter. For example, the image style of the original multimedia data can be converted into an animation style, and correspondingly, the target object in the original multimedia data becomes a cartoon character.

In some embodiments, a target special effect template may be selected from multiple selectable special effect templates of the special effect editing tool to perform special effect editing on the original multimedia data. Specifically, if the original multimedia data is an image, the target special effect template can be used to perform static or dynamic special effect editing on the partial image or the overall image of the original image to generate the multimedia data to be matched in image or video format. Or, if the original multimedia data is a video, one or more key video frames can be extracted from the original video, and the target special effect template can be used to perform static or dynamic special effects editing on the partial image or the overall image of the key video frame to generate an image Or the multimedia data to be matched in video format. Optionally, the key video frame may be a video frame containing the target object in the original video.

Further, according to different ways of editing special effects, S120 may include at least the following two implementation manners.

In some embodiments, S120 may specifically include: in response to the template selection operation on the target special effect template, performing special effect editing on the original multimedia data based on the target special effect template to obtain the multimedia data to be matched.

Specifically, after receiving the original multimedia data, if the user wants to edit special effects on the original multimedia data, the electronic device may respond to the user's trigger operation of selecting a target special effect template from multiple optional special effect templates in the special effect editing tool, The target special effect template selected by the user is used to perform special effect editing on the original multimedia data to obtain a multimedia template to be matched.

In some other embodiments, S120 may specifically include: performing special effect editing on the original multimedia data based on the target special effect template corresponding to the original multimedia data, to obtain the multimedia data to be matched.

Specifically, if the user pre-selects the target special effect template, the target special effect template can be directly used to perform special effect editing on the original multimedia data to obtain the multimedia data to be matched. Alternatively, based on the multimedia characteristics of the original multimedia data, an appropriate special effect template can be matched as the target special effect template. Alternatively, if the user first selects a target special effect template, multimedia data can be photographed in the template, and correspondingly, the multimedia data to be matched is directly displayed on the photographing interface.

S330. Generate composite multimedia data based on the multimedia data to be matched and the target multimedia data.

In the embodiment of the present disclosure, after receiving the multimedia data to be matched and the target multimedia data, the electronic device may splice the target image part of the multimedia data to be matched with the target image part of the target multimedia data directly or after a certain conversion. Or the way of image fusion is added to the target image area in the target multimedia template to obtain composite multimedia data, so that the composite multimedia data can have the first special effect role in the multimedia data to be matched and the second special effect role in the target multimedia data at the same time. At least some of the features in the target multimedia template.

To facilitate the explanation of the synthesized multimedia data, before introducing the synthesized multimedia data, the following part of the embodiments of the present disclosure will specifically describe the first multimedia feature of the multimedia data to be matched and the target multimedia data.

The first multimedia feature of the multimedia data to be matched may be a feature of the first special effect character itself or a feature of an accessory.

In some embodiments, the characteristics of the first special effects character may include facial features or body characteristics such as height, fatness and thinness of the first special effects character.

Optionally, the facial features may include facial features such as head aspect ratio, face shape, ratio of chin to head width, ratio of forehead length to head length, and the like. Alternatively, eye characteristics such as eye size, distance between eyes, pupil color, pupil size, eye shape, etc. may be included. Alternatively, nasal features such as nose length, nose wing width, nose bridge height, nose bridge width, etc. may be included. Still alternatively, hair characteristics such as hair length, hair color, hair shape (curly, straight), etc. may be included. Alternatively, skin information such as skin color, skin roughness, etc. may be included.

In some embodiments, the characteristics of the accessory accessories may include whether to wear glasses, whether to wear a mask, whether to wear accessories, whether to wear headgear, whether to wear makeup, whether to have facial effects that do not change the characteristics of the original parts of the face, and the like. Exemplarily, if the first special effect object wears an accessory, specific features of the accessory may be included. For example, if the first special effect object wears a mask, the first multimedia feature may also include the model and name of the mask.

Through the first multimedia feature shown above, target multimedia data having the same or matching features with the multimedia data to be matched can be obtained through matching.

For the target multimedia data, optionally, the target multimedia data may include an object edited with special effects. Exemplarily, the object in the target multimedia data and the target object in the original multimedia data may be different objects. For example, the object in the target multimedia data may be the image of the second user edited with special effects, and the target object in the multimedia data to be matched may be the image of the first user edited with special effects. For ease of description, the edited object in the target multimedia data may be referred to as a second special effect object.

In some embodiments, the target multimedia data may be pre-stored data in a target application program, a third-party application program, or a multimedia database of a web page.

In other embodiments, the target multimedia data may be multimedia data uploaded by other users and edited with special effects.

In addition, the manner in which other users generate target multimedia data is similar to the manner in which to-be-matched multimedia data is generated, which will not be repeated here.

In some embodiments, the multimedia data to be matched and the target multimedia data may be the same type of multimedia data. For example, both are images, or both are videos. Alternatively, the multimedia data to be matched and the target multimedia data may be different types of multimedia data. One of the two is an image, and the other of the two is a video.

After the multimedia data to be matched and the target multimedia data are introduced in detail, the embodiment of the present disclosure will describe synthesizing multimedia data.

In some embodiments, if the target multimedia template is a scene template, the synthesized multimedia data can be used to present the interaction between the first special effect role in the multimedia data to be matched and the second special effect role in the target multimedia data in the scene corresponding to the target multimedia template behavior or interaction. Wherein, the target multimedia template may be an image scene template or a video scene template, and its specific type is not limited.

In an example, the electronic device may generate synthesized multimedia data based on the user's operation of selecting a target multimedia template from multiple optional scene templates. Wherein, the optional scene template may be obtained from a scene template library of a target application program, a third-party application program, or a web page.

In another example, the electronic device may determine a matching target scene template according to features of the first special effect object in the multimedia data to be matched and features of the second special effect object in the target multimedia data. Wherein, the special effect of the first special effect object and the feature of the second special effect object may be their actions.

For example, if the action of the first special effect object is a toast, and the action of the second special effect object is also a toast, then the partial or overall images of the first special effect object and the second special effect object can be added to scene templates such as parties, bars, etc. Synthetic multimedia data such as cheers are generated.

For another example, if the action of the first special effect object is to be hugged by a princess, and the action of the second special effect object is to hug a person, then the partial or overall images of the first special effect object and the second special effect object can be added to images such as weddings, beautiful sky, etc. In romantic scene templates such as , generate synthetic multimedia data such as a spinning princess hug.

For another example, if the action of the first special effect object is kicking and shooting, and the action of the second special effect object is defense, then the partial or overall features of the first special effect object and the second special effect object can be added to scene templates such as sports fields, Generate synthetic multimedia data such as football matches.

In yet another example, the electronic device may generate synthesized multimedia data based on a target multimedia template uploaded by a user.

In some embodiments, text, music, special effects, etc. may also be added to the synthesized multimedia data in order to improve interest.

S340, displaying the synthesized multimedia data.

In an embodiment of the present disclosure, the electronic device may display and display the composite multimedia data in response to a user's composite operation on the multimedia data or a user's trigger operation for displaying the composite multimedia data. Alternatively, the electronic device does not need to respond to the trigger operation, and directly displays the synthesized multimedia data on a relevant interface after generating the synthesized multimedia data.

The video display method, device, device and medium of the embodiments of the present disclosure generate and display synthesized multimedia data based on the edited multimedia data to be matched and target multimedia data after performing special effect editing on the received original multimedia data. Since the target pair of media data is obtained based on the first multimedia feature matching of the multimedia data to be matched, in addition to the special effects in the multimedia data to be matched, the synthesized multimedia data obtained based on the original multimedia data may also include The multimedia data matches the content of the target multimedia data, so that the multimedia data image has multiple elements, thereby enriching the beautification effect of the multimedia data, thereby improving the fun of multimedia data display, and enabling users to interact through multimedia data , realize the diversified interaction between users, and improve the user experience.

For ease of understanding, the following embodiments of the present disclosure will specifically describe the multimedia display method provided by the embodiments of the present disclosure through FIGS. 4-8 .

Fig. 4 shows a schematic diagram of a shooting preview interface provided by an embodiment of the present disclosure.

As shown in FIG. 4 , the electronic device can display a target object 41 in the shooting preview interface 40, and various special effect editing tools such as a filter tool 401, a beauty tool 402, and a special effect tool 403, and can also display a multimedia synthesis tool 404. . Wherein, the filter tool 401, the beauty tool 402, and the special effect tool 403 may each include one or more special effect templates.

When the user clicks the special effect tool 403, the displayed interface may be as shown in FIG. 5 . Fig. 5 shows a schematic diagram of a special effect editing interface provided by an embodiment of the present disclosure.

As shown in FIG. 5 , multiple special effect templates 4031 to 4034 of the special effect tool 403 may be displayed on the special effect editing interface 50 . After the user selects the mask special effect template 4033, the generated multimedia data to be matched is shown in FIG. 6 . Fig. 6 shows a schematic diagram of a display interface of multimedia data to be matched provided by an embodiment of the present disclosure.

As shown in FIG. 6 , the display interface 60 of the multimedia data to be matched may include a first special effect character 61 after special effect processing and a multimedia synthesis tool 404 . After the user clicks on the multimedia synthesis tool 404, the matching step of the multimedia data can be performed by the electronic device or the server. Fig. 7 shows a schematic diagram of matching logic of multimedia data provided by an embodiment of the present disclosure.

As shown in FIG. 7 , after matching the target multimedia data P72 containing the second special effect character 73 according to the matching multimedia data P71 containing the first special effect character 61 , the generated composite multimedia data is shown in FIG. 8 .

FIG. 8 shows a schematic diagram of a display interface for synthesizing multimedia data provided by an embodiment of the present disclosure. As shown in 8, the synthesized multimedia data P81 may present a scene in which the first special effect character 61 and the second special effect character 73 toast in the masquerade scene in the form of images or videos.

In some embodiments provided by the embodiments of the present disclosure, FIG. 9 shows a schematic flowchart of another multimedia display method provided by the embodiments of the present disclosure.

As shown in FIG. 9, the multimedia display method may include the following steps.

S910. Receive original multimedia data. Wherein, the specific content of S910 is similar to the specific content of S310, which will not be repeated here.

S920, performing special effect editing on the original multimedia data to obtain the multimedia data to be matched. Wherein, the specific content of S920 is similar to the specific content of S320, which will not be repeated here.

S930. Extract a first multimedia feature from the multimedia data to be matched.

In some embodiments, image feature extraction technology or video frame feature extraction technology may be used to extract the first multimedia feature from the multimedia data to be matched.

For the specific content of the first multimedia feature, refer to the relevant description of S330 in the above part of the embodiment of the present disclosure, and details will not be repeated again.

S940. Acquire a plurality of candidate multimedia data corresponding to the multimedia data to be matched.

In some embodiments, candidate multimedia data may be obtained from a multimedia database of a target application, a third-party application, or a webpage.

In one example, the candidate multimedia data may be pre-stored data in a multimedia database.

In another example, the candidate multimedia data may be multimedia data uploaded by other users and edited with special effects.

S950. Query target multimedia data matching the first multimedia feature among multiple candidate multimedia data.

In some embodiments, target multimedia data can be determined from multiple candidate multimedia data by means of feature matching.

In an embodiment, S950 may specifically include the following steps.

Step A1, determining at least one feature tag corresponding to the first multimedia feature.

Optionally, the feature label may be a label obtained by classifying the first special effect object from one or more dimensions based on one or a type of first multimedia feature. Exemplarily, the feature tag can classify the first special effect object from the dimension of the first special effect object itself or the accessory component.

For example, the feature tags of the first special effect object may include tags for characterizing the nose, eyes, gender, action, skin state, etc. The first special effect object can be classified from the characteristics of the character itself.

For another example, the attached tags of the first special effect object may include whether to wear glasses, whether to wear a mask, whether to wear makeup, and so on.

Step A2, for each candidate multimedia data, determine a common label that is the same as at least one feature label.

That is to say, if the same label exists between the label of the multimedia data to be matched and the label of the candidate multimedia data, the same label may be used as a common label of the multimedia data to be matched and the candidate multimedia data.

Exemplarily, if user A's tags include wearing glasses, tall nose, yellow skin, tall, and female; user B's tags include no glasses, small mouth, thin, and male. Then the common tags of the two may include glasses tags (wearing glasses or not wearing glasses), gender tags (male or female).

Step A3, calculating the tag matching score between the multimedia data to be matched and each candidate multimedia data according to the weight values corresponding to the common tags.

In one example, the weight value of the common tag may be preset.

In yet another example, the weight value of the common tag may be set according to the user's selection. For tags that users do not pay attention to, set a low weight value a. For tags that the user values or pays attention to (such as whether the user likes or dislikes whether other characters wear glasses and whether they have ponytails), a high weight value b can be set for the glasses tag and hairstyle tag. Wherein, the weight value b is greater than the weight value a. Alternatively, a high weight value c can be set for tags that the user is interested in, and a low weight value d can be set for tags that the user dislikes. Wherein, the weight value c is greater than the weight value a, and the weight value a is greater than the weight value d.

The tag matching score is used to reflect the degree of matching between each candidate multimedia data and the multimedia data to be matched in terms of a feature or a type of feature corresponding to the tag.

In some embodiments, for each feature tag, a tag score of the multimedia data corresponding to the tag may be generated according to the feature. For example, for the glasses tag, if the first special effect object in the multimedia data to be matched wears glasses, the tag score of the glasses tag can be 100; if the first special effect object does not wear glasses, the tag score of the glasses tag can be 0.

It should be noted that the calculation method of the tag score of the candidate multimedia data is the same as that of the multimedia data to be matched, and will not be repeated here.

Correspondingly, after acquiring the tag scores of the candidate multimedia data and the tag scores of the multimedia data to be matched, the similarity score of the two can be calculated according to the tag scores of the common tags of the two. Then calculate the label matching score of the two according to the similarity score and the weight value between the two.

Optionally, for some feature tags, the closeness of tag scores of each candidate multimedia data to the multimedia data to be matched is positively correlated with the similarity score between the two. That is to say, the closer the tag scores of each candidate multimedia data and the multimedia data to be matched are, the higher the similarity score between them will be. For example, if both are wearing glasses, their matching score is high. Exemplarily, the similarity score corresponding to this type of feature label may be equal to the preset value minus the target label score difference. Wherein, the target label score difference is the difference between the label score of each candidate multimedia data in this category of labels and the label score of the multimedia data to be matched in this category of labels.

For other feature labels, the closeness of the label scores of each candidate multimedia data to the multimedia data to be matched is negatively correlated with the similarity score between the two. That is to say, the greater the tag score difference between each candidate multimedia data and the multimedia data to be matched, the lower the similarity score between the two. For example, if both genders are the same, their similarity score is low. If the two genders are opposite, their similarity score is high. Exemplarily, the similarity score corresponding to this type of feature label may be equal to the target label score difference.

In one example, tag scores for multiple candidate multimedia data can be recorded in a matching table. Correspondingly, after the electronic device acquires the feature tags of the multimedia data to be matched and the tag scores of each feature tag, it calculates the tag matching scores of the multimedia data to be matched and each candidate multimedia data based on the above calculation method, so as to calculate from the matching table and Find the target multimedia data.

In some other embodiments, the tag matching score of each tag may be obtained according to the weight value of the tag and the feature matching score between each candidate multimedia data and the multimedia data to be matched. For example, the tag matching score of each tag may be equal to the product of the weight value of the tag and the feature matching score between each candidate multimedia data and the multimedia data to be matched.

Optionally, for some feature tags, the similarity between each candidate multimedia data and the multimedia data to be matched is positively correlated with the feature matching score between the two. That is to say, the higher the similarity between each candidate multimedia data and the multimedia data to be matched, the higher the feature matching score between the two. For example, if both wear glasses, the feature matching score is high.

For other feature tags, the similarity between each candidate multimedia data and the multimedia data to be matched is negatively correlated with the feature matching score between the two. That is to say, the lower the similarity between each candidate multimedia data and the multimedia data to be matched is, the higher the score of the two is. For example, if both genders are the same, their feature matching score will be low. If the two genders are opposite, the feature matching score is high.

It should be noted that the correlation between the similarity of each feature tag and the feature matching score can be set according to actual scenarios and specific requirements, and there is no specific limitation on this.

Step A4, sorting the tag matching scores of the candidate multimedia data to determine the target multimedia data.

In an example, the data with tag matching scores from high to low may be sorted, and the candidate multimedia data with the highest score is used as the target multimedia data. Wherein, the tag matching scores between the multimedia data to be matched and multiple candidate multimedia data can be recorded in the matching table in descending order or in descending order.

In some embodiments, in order to improve matching accuracy, the target multimedia data may also satisfy one or more of the following conditions.

Condition C1, the special effect editing method of the target multimedia data is the same as the special effect editing method of the to-be-matched multimedia data. Exemplarily, if both the target multimedia data and the multimedia data to be matched are edited with special effects of adding masks, the special effect editing methods of the two are the same. As another example, if a certain special effect template corresponds to the first special effect and the second special effect, the target multimedia data adopts the first special effect, and the to-be-matched multimedia data adopts the second special effect, then the two special effect editing methods are the same.

Condition C2, the user to which the target multimedia data belongs is an online user. That is to say, if the user opens the interface of the target application program through the electronic device, or the target application program is running in the background on the electronic device, the user is considered to be an online user.

Condition C3, the location distance between the distribution location of the target multimedia data and the distribution location of the original multimedia data is less than or equal to a preset distance threshold.

For example, the distance threshold may be a default value set by the system, or may be a target distance threshold selected by the user from multiple optional distance thresholds. Alternatively, if the user to which the target multimedia data belongs is in the same region as the user to which the original multimedia data belongs, such as the same district, city, or province, the distance between the two locations is considered to be less than or equal to a preset distance threshold. Wherein, the distance threshold may be set according to an actual situation or a specific scene, which is not limited.

The condition C4, the historical matching times of the target multimedia data satisfies the preset times filtering condition. Wherein, the frequency filtering condition may be that the historical matching frequency is within a preset value range. Wherein, the value range of the preset number of times may be the value set by the system by default, or may be the value range of the target number of times selected by the user from multiple optional value ranges of the number of times.

In one example, in order to improve the flexibility of matching, if the user cannot filter out the target multimedia data through steps A1 to A4, the target multimedia data can be selected from the candidate multimedia data through at least one of the above conditions C1 to C4.

In another example, in order to improve the accuracy of matching, if the user obtains multiple target multimedia data through the screening of steps A1 to A4, the multiple target multimedia data can be further screened through at least one of the above conditions C1 to C4 , to obtain the target multimedia data.

In still some examples, after acquiring the multimedia data to be matched, at least one of the above conditions C1 to C4 may be directly used to screen out target multimedia data from multiple candidate multimedia data.

In yet another example, in order to increase the matching rate, the candidate multimedia data may be obtained after screening by at least one of the above conditions C1 to C4.

In some embodiments, if the user screens the target multimedia data through at least two of the above-mentioned conditions C1 to C4, the candidate multimedia data can be screened in order according to the preset condition usage order until the last one is used. After a condition is filtered to obtain the target multimedia data. Alternatively, when the data of the target multimedia data obtained by using one more feature is within a preset quantity range, the target multimedia data can be obtained.

In some embodiments, the target multimedia data is also obtained by matching the second multimedia feature of the original multimedia data.

In one example, in order to improve the flexibility of matching, if the user cannot filter out the target multimedia data through steps A1 to A4, the target multimedia data can be matched from the candidate multimedia data by using the second multimedia feature of the original multimedia data.

In another example, in order to improve the accuracy of matching, if the user obtains multiple target multimedia data through steps A1 to A4, the multiple target multimedia data can be further processed through the second multimedia feature of the original multimedia data. Screening to obtain the target multimedia data.

In some examples, the second multimedia feature may be a multimedia feature of the target object in the original multimedia data. Wherein, the second multimedia feature is similar to the above-mentioned first multimedia feature, and the method of using the second multimedia feature to query the target multimedia data is similar to the method of using the first multimedia feature to query the target multimedia data. No longer.

S960. Generate composite multimedia data based on the multimedia data to be matched and the target multimedia data. The target multimedia data is obtained by matching according to the first multimedia feature of the multimedia data to be matched. Among them, the specific content of the S960 is similar to that of the S330, which will not be repeated here.

S970, displaying the synthesized multimedia data. Among them, the specific content of the S970 is similar to that of the S340, and will not be repeated here.

The multimedia display method of the embodiment of the present disclosure can use the first data feature of the multimedia data to be matched to accurately match the target multimedia data with the same feature from multiple candidate multimedia data, so that the generated composite multimedia data includes the feature matching degree The high first special effect object and the second special effect object improve the interest of the multimedia display method.

In some embodiments provided by the embodiments of the present disclosure, FIG. 10 shows a schematic flowchart of another multimedia display method provided by the embodiments of the present disclosure.

As shown in FIG. 10, the multimedia display method may include the following steps.

S1010. Receive original multimedia data. Wherein, the specific content of S1010 is similar to the specific content of S310, which will not be repeated here.

S1020. Perform special effect editing on the original multimedia data to obtain multimedia data to be matched. Wherein, the specific content of S1020 is similar to the specific content of S320, which will not be repeated here.

S1030. Generate composite multimedia data based on the multimedia data to be matched and the target multimedia data, where the target multimedia data is obtained by matching according to the first multimedia feature of the multimedia data to be matched. Wherein, the specific content of S1030 is similar to the specific content of S330, which will not be repeated here.

S1040, displaying the synthesized multimedia data. Wherein, the specific content of S1040 is similar to the specific content of S340, which will not be repeated here.

S1050. When a trigger operation on the composite multimedia data is detected, publish the composite multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs.

In some embodiments, when the user wants to interact with the user to whom the target multimedia data belongs, a trigger operation for synthesizing multimedia data is performed. Wherein, the triggering operation may be triggered when the synthesized multimedia data is generated, or after the synthesized multimedia data is previewed, and its trigger timing is not limited.

In an embodiment, the electronic device may distribute the synthesized multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs through the server.

In one embodiment, the electronic device can display the original multimedia data and city multimedia data in the image/video favorites or display column of the target application program of the user to which the original multimedia data belongs and the user to which the target multimedia data belongs. A mark is added to the corresponding icon to prompt the user to view the synthesized multimedia data.

In another embodiment, S1050 may include the following steps.

Step D1, sending first prompting information to the user to which the original multimedia data belongs, the first prompting information is used to trigger display of the synthesized multimedia data and display the social home page of the user to which the target multimedia data belongs.

Optionally, the first prompt information can be released in the form of text, picture, voice, etc. through a chat box, a display window on the interface, or a broadcast column on the interface. Exemplarily, the specific form of the first prompt message may be "You just participated in a masquerade party with XXX (the scene corresponding to the synthesized multimedia video), go to TA's homepage to see/chat with TA"

Exemplarily, the first prompt information may include links such as text/QR code of the synthesized multimedia data display interface, or the user can jump to the synthesized multimedia display interface by triggering the information bar of the first prompt information. Optionally, for the convenience of interaction, the first prompt information may also include links such as text/QR code of the user to which the target multimedia data belongs. Alternatively, the synthesized multimedia data display interface may include a control for accessing the user's social homepage to which the target multimedia data belongs, or the synthesized multimedia data display interface may include a control for adding friends of the user to whom the target multimedia data belongs, or, the synthesized multimedia data display interface may A control for establishing a chat with the user to whom the target multimedia data belongs may be included.

Step D2, sending second prompting information to the user to whom the target multimedia data belongs, the second prompting information being used to trigger playing the composite multimedia data and displaying the social homepage of the user to which the original multimedia data belongs.

Wherein, the second prompt information is similar to the first prompt information, which will not be repeated here.

Through the embodiments of the present disclosure, synthesized multimedia data can be released to the users who belong to the original multimedia data and the users who belong to the target multimedia data, so that the interaction between the users who belong to the original multimedia data and the users who belong to the target multimedia data can be realized by synthesizing multimedia data, and the multimedia data can be improved. The interestingness of the display improves the user experience.

Fig. 11 shows a schematic flowchart of a multimedia matching method provided by an embodiment of the present disclosure.

In the embodiment of the present disclosure, the multimedia matching method may be executed by a server. Wherein, the server may be a device with storage and computing functions such as a cloud server or a server cluster.

As shown in Fig. 11, the multimedia matching method may include the following steps.

S1110. Receive the multimedia data to be matched, where the multimedia data to be matched is obtained by performing special effect processing on the original multimedia data.

S1120. Extract a first multimedia feature from the multimedia data to be matched.

S1130. Acquire multiple candidate multimedia data corresponding to the multimedia data to be matched.

S1140. Search for target multimedia data that matches the first multimedia feature among multiple candidate multimedia data, and the target multimedia data is used to generate merged multimedia data with the multimedia data to be matched.

In some embodiments of the present disclosure, S1140 may include:

determining at least one feature label corresponding to the first multimedia feature;

For each candidate multimedia data, determine a common label identical to at least one feature label;

Calculate the label matching score between the multimedia data to be matched and each candidate multimedia data according to the weight value corresponding to the common label;

The tag matching scores of the candidate multimedia data are sorted to determine the target multimedia data.

It should be noted that the multimedia matching method shown in S1110 to S1140 is similar to the multimedia display method shown in conjunction with S910 to S970 above, and will not be repeated here.

In some embodiments of the present disclosure, after S1140, the multimedia matching method may further include: receiving a publishing instruction of synthesized multimedia data, and distributing the synthesized multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs. Wherein, the publishing instruction is generated by the electronic device after detecting a trigger operation on the synthesized multimedia data. Wherein, this step is similar to S1050 and will not be repeated here.

In some embodiments of the present disclosure, after S1140, the multimedia matching method may further include:

The first prompt information is sent to the user to which the original multimedia data belongs, and the first prompt information is used to trigger the display of the composite multimedia data and display the social home page of the user to which the target multimedia data belongs. Wherein, this step is similar to the above step D1 and will not be repeated here.

The second prompt information is sent to the user to which the target multimedia data belongs, and the second prompt information is used to trigger the playing of the composite multimedia data and display the social home page of the user to which the original multimedia data belongs. Wherein, this step is similar to the above step D2 and will not be repeated here.

In the multimedia matching method of the embodiment of the present disclosure, after special effect editing is performed on the received original multimedia data, composite multimedia data is generated and displayed based on the edited multimedia data to be matched and target multimedia data. Since the target pair of media data is obtained based on the first multimedia feature matching of the multimedia data to be matched, in addition to the special effects in the multimedia data to be matched, the synthesized multimedia data obtained based on the original multimedia data may also include The multimedia data matches the content of the target multimedia data, so that the multimedia data image has multiple elements, thereby enriching the beautification effect of the multimedia data, thereby improving the fun of multimedia data display, and enabling users to interact through multimedia data , realize the diversified interaction between users, and improve the user experience.

An embodiment of the present disclosure also provides a multimedia display device for implementing the above multimedia display method, which will be described below with reference to FIG. 12 .

In the embodiment of the present disclosure, the multimedia display device may be an electronic device, for example, the multimedia display device may be the first electronic device 101 in the client shown in FIG. 1 . Among them, electronic devices may include devices with communication functions such as mobile phones, tablet computers, desktop computers, notebook computers, vehicle terminals, wearable electronic devices, all-in-one computers, and smart home devices, and may also be devices simulated by virtual machines or simulators. .

FIG. 12 shows a schematic structural diagram of a multimedia display device provided by an embodiment of the present disclosure.

As shown in FIG. 12 , the multimedia display device 1200 may include a data receiving unit 1210 , a special effect editing unit 1220 , a data synthesis unit 1230 and a data display unit 1240 .

A data receiving unit 1210 configured to receive original multimedia data;

The special effect editing unit 1220 is configured to perform special effect editing on the original multimedia data to obtain the multimedia data to be matched;

The data synthesis unit 1230 is configured to generate synthesized multimedia data based on the multimedia data to be matched and the target multimedia data, and the target multimedia data is obtained by matching the first multimedia feature of the multimedia data to be matched;

The data display unit 1240 is configured to display synthesized multimedia data.

The multimedia display device of the disclosed embodiment generates and displays synthesized multimedia data based on the edited multimedia data to be matched and target multimedia data after performing special effect editing on the received original multimedia data. Since the target pair of media data is obtained based on the first multimedia feature matching of the multimedia data to be matched, in addition to the special effects in the multimedia data to be matched, the synthesized multimedia data obtained based on the original multimedia data may also include The multimedia data matches the content of the target multimedia data, so that the multimedia data image has multiple elements, thereby enriching the beautification effect of the multimedia data, thereby improving the fun of multimedia data display, and enabling users to interact through multimedia data , realize the diversified interaction between users, and improve the user experience.

In some embodiments of the present disclosure, the special effect editing unit 1220 may be further configured to: in response to the template selection operation on the target special effect template, perform special effect editing on the original multimedia data based on the target special effect template to obtain the multimedia data to be matched;

In other embodiments of the present disclosure, the special effect editing unit 1220 may be further configured to: based on the target special effect template corresponding to the original multimedia data, perform special effect editing on the original multimedia data to obtain the multimedia data to be matched.

In some embodiments of the present disclosure, the multimedia display device 1200 may further include a feature extraction unit, a data acquisition unit, and a data query unit.

The data query unit is configured to query the target multimedia data matching the first multimedia feature among the plurality of candidate multimedia data.

In some embodiments of the present disclosure, the data query unit may be further configured as:

In some embodiments of the present disclosure, the target multimedia data satisfies at least one of the following:

The special effect editing method of the target multimedia data is the same as the special effect editing method of the multimedia data to be matched;

The user to which the target multimedia data belongs is an online user;

The location distance between the distribution location of the target multimedia data and the distribution location of the original multimedia data is less than or equal to a preset distance threshold;

The historical matching times of the target multimedia data meet the preset times filtering condition.

In some embodiments of the present disclosure, the target multimedia data is also obtained by matching the second multimedia feature of the original multimedia data.

In some embodiments of the present disclosure, the multimedia display device 1200 may further include a data publishing unit.

The data distribution unit is configured to distribute the composite multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs when a trigger operation on the composite multimedia data is detected.

In some embodiments of the present disclosure, the data publishing unit may be further configured as:

Sending first prompt information to the user to whom the original multimedia data belongs, the first prompt information is used to trigger display of the composite multimedia data and display the social home page of the user to which the target multimedia data belongs;

Sending second prompting information to the user to which the target multimedia data belongs, the second prompting information is used to trigger playing the synthesized multimedia data and displaying the social home page of the user to which the original multimedia data belongs.

It should be noted that the multimedia display device 1200 shown in FIG. 12 can execute each step in the method embodiment shown in FIG. 3 to FIG. 10 , and realize each process and effects, which will not be described here.

An embodiment of the present disclosure also provides a multimedia matching device for implementing the above multimedia matching method, which will be described below with reference to FIG. 13 . In the embodiment of the present disclosure, the multimedia display device may be a server, for example, the multimedia matching device may be the server 102 in the client shown in FIG. 1 . Wherein, the server may be a device with storage and computing functions such as a cloud server or a server cluster.

Fig. 13 shows a schematic structural diagram of a multimedia matching device provided by an embodiment of the present disclosure.

As shown in FIG. 13 , the multimedia matching device 1300 may include a data receiving unit 1310 , a feature extraction unit 1320 , a data obtaining unit 1330 and a data query unit 1340 .

The data receiving unit 1310 is configured to receive the multimedia data to be matched, and the multimedia data to be matched is obtained based on special effects processing on the original multimedia data;

The feature extraction unit 1320 is configured to extract a first multimedia feature from the multimedia data to be matched;

The data obtaining unit 1330 is configured to obtain a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

The data query unit 1340 is configured to query the target multimedia data matching the first multimedia feature among the plurality of candidate multimedia data, and the target multimedia data is used to generate combined multimedia data with the multimedia data to be matched.

The multimedia matching device of the disclosed embodiment generates and displays synthesized multimedia data based on the edited multimedia data to be matched and target multimedia data after performing special effect editing on the received original multimedia data. Since the target pair of media data is obtained based on the first multimedia feature matching of the multimedia data to be matched, in addition to the special effects in the multimedia data to be matched, the synthesized multimedia data obtained based on the original multimedia data may also include The multimedia data matches the content of the target multimedia data, so that the multimedia data image has multiple elements, thereby enriching the beautification effect of the multimedia data, thereby improving the fun of multimedia data display, and enabling users to interact through multimedia data , realize the diversified interaction between users, and improve the user experience.

In some embodiments of the present disclosure, the data query unit 1340 may be further configured to:

The user to which the target multimedia data belongs is an online user;

In some embodiments of the present disclosure, the target multimedia data is also matched according to the second multimedia feature of the original multimedia data.

In some embodiments of the present disclosure, in some embodiments of the present disclosure, the target multimedia data is also obtained by matching the second multimedia feature of the original multimedia data.

In some embodiments of the present disclosure, the multimedia matching apparatus 1300 may further include a data publishing unit.

The data distribution unit is configured to distribute the composite multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs in response to the composite multimedia data distribution instruction. Wherein, the publishing instruction is generated by the electronic device after detecting a trigger operation on the synthesized multimedia data.

The second prompt information is sent to the user to which the target multimedia data belongs, and the second prompt information is used to trigger the playing of the composite multimedia data and display the social home page of the user to which the original multimedia data belongs.

It should be noted that the multimedia matching device 1300 shown in FIG. 13 can execute each step in the method embodiment shown in FIG. 11 , and realize each process and effect in the method embodiment shown in FIG. 11 , which will not be described here. repeat.

An embodiment of the present disclosure also provides a computing device, which may include a processor and a memory, and the memory may be used to store executable instructions. Wherein, the processor may be configured to read executable instructions from the memory, and execute the executable instructions to implement the multimedia display method and/or the multimedia matching method in the above embodiments.

Fig. 14 shows a schematic structural diagram of a computing device provided by an embodiment of the present disclosure. Referring to FIG. 14 in detail below, it shows a schematic structural diagram for implementing a computing device 1400 in an embodiment of the present disclosure.

In some embodiments, when implementing the multimedia display method in the above embodiments, the computing device 1400 in the embodiments of the present disclosure may include but not limited to mobile phones, notebook computers, digital broadcast receivers, PDA (Personal Digital Assistant) , PAD (Tablet Computer), PMP (Portable Multimedia Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc. mobile terminals and fixed terminals such as digital TVs, desktop computers, smart home devices, etc.

In other embodiments, when implementing the multimedia matching method in the above embodiments, the computing device 1400 in the embodiments of the present disclosure may include, but not limited to, devices with storage and computing functions such as cloud servers or server clusters.

It should be noted that the computing device 1400 shown in FIG. 14 is only an example, and should not limit the functions and scope of use of this embodiment of the present disclosure.

As shown in FIG. 14, the computing device 1400 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1401, which may be stored in a read-only memory (ROM) 1402 or loaded into a random Various appropriate actions and processes are executed by accessing programs in the memory (RAM) 1403 . In the RAM 1403, various programs and data necessary for the operation of the computing device 1400 are also stored. The processing device 1401, ROM 1402, and RAM 1403 are connected to each other through a bus 1404. An input/output (I/O) interface 1405 is also connected to the bus 1404 .

Typically, the following devices can be connected to the I/O interface 1405: input devices 1406 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speaker, vibration an output device 1407 such as a computer; a storage device 1408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1409. The communication means 1409 may allow the computing device 1400 to communicate with other devices wirelessly or by wire to exchange data. While FIG. 14 shows computing device 1400 having various means, it is to be understood that implementing or possessing all of the illustrated means is not a requirement. More or fewer means may alternatively be implemented or provided.

An embodiment of the present disclosure also provides a computer-readable storage medium, the storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the multimedia display method or the multimedia matching method in the foregoing embodiments.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs.

An embodiment of the present disclosure also provides a computer program product, the computer program product may include a computer program, and when the computer program is executed by a processor, the processor is made to implement the video editing method or the video playing method in the foregoing embodiments.

For example, embodiments of the present disclosure include a computer program product, which includes a computer program carried on a non-transitory computer readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via communication means 1409, or from storage means 1408, or from ROM 1402. When the computer program is executed by the processing device 1401, the above-mentioned functions defined in the multimedia display method or the multimedia matching method of the embodiment of the present disclosure are executed.

It should be noted that the above-mentioned computer-readable medium in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted by any appropriate medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, clients and servers can communicate using any currently known or future developed network protocol, such as HTTP, and can be interconnected with any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network of.

The above-mentioned computer-readable medium may be included in the above-mentioned computing device, or may exist independently without being assembled into the computing device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the computing device is made to execute:

Receive the original multimedia data; edit the original multimedia data with special effects to obtain the multimedia data to be matched; generate synthetic multimedia data based on the multimedia data to be matched and the target multimedia data, and the target multimedia data is matched according to the first multimedia feature of the multimedia data to be matched Get; display composite multimedia data.

Or, receiving the multimedia data to be matched, the multimedia data to be matched is obtained based on performing special effect processing on the original multimedia data; extracting the first multimedia feature from the multimedia data to be matched; obtaining a plurality of candidate multimedia data corresponding to the multimedia data to be matched; Among the plurality of candidate multimedia data, the target multimedia data matching the first multimedia feature is queried, and the target multimedia data is used to generate merged multimedia data with the multimedia data to be matched.

In the embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, C++, and also conventional procedural programming languages - such as "C" or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through an Internet service provider). Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or portion of code that contains one or more logical functions for implementing specified executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. Wherein, the name of a unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), System on Chips (SOCs), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principle. Those skilled in the art should understand that the disclosure scope involved in this disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, but also covers the technical solutions formed by the above-mentioned technical features or Other technical solutions formed by any combination of equivalent features. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, while operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or performed in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while the above discussion contains several specific implementation details, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

A multimedia display method, characterized in that, comprising:

receiving raw multimedia data;

Editing the original multimedia data with special effects to obtain the multimedia data to be matched;

Generate composite multimedia data based on the multimedia data to be matched and target multimedia data, and the target multimedia data is obtained by matching the first multimedia feature of the multimedia data to be matched;

The synthesized multimedia data is displayed.
The method according to claim 1, wherein said editing the original multimedia data with special effects to obtain the multimedia data to be matched comprises:

In response to a template selection operation on a target special effect template, perform special effect editing on the original multimedia data based on the target special effect template to obtain the multimedia data to be matched;

Or, based on the target special effect template corresponding to the original multimedia data, perform special effect editing on the original multimedia data to obtain the multimedia data to be matched.
The method according to claim 1, wherein, before generating composite multimedia data based on the multimedia data to be matched and the target multimedia data, the method further comprises:

extracting the first multimedia feature from the multimedia data to be matched;

Acquiring a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

Among the plurality of candidate multimedia data, query the target multimedia data matching the first multimedia feature.
The method according to claim 3, wherein, among the plurality of candidate multimedia data, searching for the target multimedia data matching the first multimedia feature comprises:

determining at least one feature tag corresponding to the first multimedia feature;

For each of the candidate multimedia data, determine a common label that is the same as the at least one feature label;

According to the weight value corresponding to the common label, calculate the label matching score between the multimedia data to be matched and each of the candidate multimedia data;

sorting the tag matching scores of the candidate multimedia data to determine target multimedia data.
The method according to claim 1, wherein the target multimedia data satisfies at least one of the following:

The special effect editing method of the target multimedia data is the same as the special effect editing method of the multimedia data to be matched;

The user to which the target multimedia data belongs is an online user;

The location distance between the distribution location of the target multimedia data and the distribution location of the original multimedia data is less than or equal to a preset distance threshold;

The historical matching times of the target multimedia data meet the preset times filtering condition.
The method according to claim 1, wherein the target multimedia data is also obtained by matching the second multimedia feature of the original multimedia data.
The method according to claim 1, characterized in that, after said displaying said synthesized multimedia data, said method further comprises:

When a trigger operation on the composite multimedia data is detected, the composite multimedia data is released to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs.
The method according to claim 7, wherein the distributing the synthesized multimedia data to the user to which the original multimedia data belongs and the user to which the target multimedia data belongs comprises:

Sending first prompt information to the user to which the original multimedia data belongs, the first prompt information is used to trigger the display of the synthesized multimedia data and display the social home page of the user to which the target multimedia data belongs;

Sending second prompt information to the user to which the target multimedia data belongs, where the second prompt information is used to trigger playing the synthesized multimedia data and display a social home page of the user to which the original multimedia data belongs.
A multimedia matching method, characterized in that, comprising:

Receiving multimedia data to be matched, the multimedia data to be matched is obtained based on special effects processing of the original multimedia data;

extracting the first multimedia feature from the multimedia data to be matched;

Acquiring a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

Among the plurality of candidate multimedia data, the target multimedia data matching the first multimedia feature is searched, and the target multimedia data is used to generate merged multimedia data with the multimedia data to be matched.
The method according to claim 9, wherein, among the plurality of candidate multimedia data, searching for the target multimedia data matching the first multimedia feature comprises:

determining at least one feature tag corresponding to the first multimedia feature;

For each of the candidate multimedia data, determine a common label that is the same as the at least one feature label;

According to the weight value corresponding to the common label, calculate the label matching score between the multimedia data to be matched and each of the candidate multimedia data;

sorting the tag matching scores of the candidate multimedia data to determine target multimedia data.
A multimedia display device, characterized in that it comprises:

a data receiving unit configured to receive original multimedia data;

A special effect editing unit configured to perform special effect editing on the original multimedia data to obtain the multimedia data to be matched;

The data synthesis unit is configured to generate synthesized multimedia data based on the multimedia data to be matched and the target multimedia data, and the target multimedia data is obtained by matching the first multimedia feature of the multimedia data to be matched;

A data display unit configured to display the synthesized multimedia data.
A multimedia matching device is characterized in that it comprises:

The data receiving unit is configured to receive the multimedia data to be matched, and the multimedia data to be matched is obtained based on special effect processing to the original multimedia data;

A feature extraction unit configured to extract the first multimedia feature from the multimedia data to be matched;

A data acquisition unit configured to acquire a plurality of candidate multimedia data corresponding to the multimedia data to be matched;

A data query unit configured to query, among the plurality of candidate multimedia data, the target multimedia data that matches the first multimedia feature, and the target multimedia data is used to generate the multimedia data to be matched Merge multimedia data.
A computing device, comprising:

processor;

memory for storing executable instructions;

Wherein, the processor is configured to read the executable instruction from the memory, and execute the executable instruction to implement the multimedia display method according to any one of the above claims 1-8 or the above claim 9 The multimedia matching method described in any one of -10.
A computer-readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the processor realizes the multimedia described in any one of claims 1-8 A display method or the multimedia matching method described in any one of claims 9-10 above.