WO2023134269A1

WO2023134269A1 - Display device, and virtual fitting system and method

Info

Publication number: WO2023134269A1
Application number: PCT/CN2022/128392
Authority: WO
Inventors: 黄玖法
Original assignee: 海信视像科技股份有限公司
Priority date: 2022-01-17
Filing date: 2022-10-28
Publication date: 2023-07-20
Also published as: CN116523579A

Abstract

A display device, and a virtual fitting system and method. According to the method, image data associated with a user can be collected in real time during use, and the image data associated with the user can be sent to a server to generate a human model. Clothing materials are added into the human model to synthesize a rendering model, and character motions are extracted in real time, so that a model posture of the rendering model is adjusted according to motion parameters to form a fitting picture. According to the method, a dynamic 3D virtual fitting function can be achieved, and the user motions are displayed in real time by means of the rendering model, thereby achieving the use effects of clothes moving with people and wearing recommendation, and solving the problem in traditional virtual fitting methods that the fitting effect cannot be displayed in real time.

Description

A display device, virtual fitting system and method

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202210051018.9 and the application title "a display device, virtual fitting system and method" submitted to the China Patent Office on January 17, 2022, the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the technical field of intelligent display devices, in particular to a display device, a virtual fitting system and a method.

Background technique

The virtual fitting system is a service platform that integrates hardware and software. It can build a virtual model through augmented reality (Augmented Reality, AR), artificial intelligence (AI), 3D vision and other technologies, and generate a display based on the virtual model. picture. The virtual fitting system can realize the 360-degree natural fit of virtual clothes and the effect of clothes moving with people. It can be widely used in online shopping, daily wear and other life scenes.

The virtual fitting system can be built into smart devices such as smart terminals and display devices. When using the virtual fitting system, the smart device is controlled to run the application of the virtual fitting system. Then input image data such as photos, then the virtual fitting system can form a synthetic effect map through image synthesis, and display it through smart devices to achieve the purpose of virtual fitting.

However, this virtual fitting method, which forms effect images through image synthesis, is only suitable for static image display, and cannot display the fitting effect in real time, nor can it display the effects of various wearing postures and viewing angles. The final effect of this kind of virtual fitting system is poor, and the degree of restoration of fitting is low.

Contents of the invention

The present disclosure provides a display device, including: a display, a camera, a communicator, and a controller. Wherein, the display is configured to display images and/or user interfaces; the camera is configured to collect image data in real time, and the image data includes images associated with the user; the communicator is configured to establish a communication connection with the server, and the server has a built-in model reconstruction application, using The human body model is generated according to the image data associated with the user; the controller is connected with the display, the camera, and the communicator, and the controller is configured to perform: obtaining the image data associated with the user; sending the image data associated with the user to the server for Make the server generate a human body model according to the image data associated with the user and send the human body model to the controller; add clothing materials to the human body model to synthesize the rendering model; extract action parameters from the image data associated with the user, and according to the The action parameters adjust the model pose of the rendered model to render a fitting picture.

The present disclosure also provides a virtual fitting system, including: a display device, an image acquisition device, and a server; wherein, the image acquisition device is connected to the display device, and the display device establishes a communication connection with the server; the image acquisition device is configured to collect image data in real time, The image data includes images associated with the user, and image signal processing is performed on the images associated with the user to generate image data associated with the user; the image acquisition device is also configured to send the image data associated with the user to the display device; the display device It is configured to obtain the image data associated with the user, and send the image data associated with the user to the server; a model reconstruction application is built in the server, and the server is configured to receive the image data associated with the user, and run the model reconstruction application; according to the The image data associated with the user generates a human body model, and the server is also configured to send the human body model to the display device; the display device is also configured to add clothing materials to the human body model to synthesize the rendered model; and, extract from the image data associated with the user Action parameters, and adjust the model pose of the rendered model according to the action parameters to render the fitting screen.

The present disclosure also provides a virtual fitting method, which is applied to a virtual fitting system. The virtual fitting system includes: a display device, an image acquisition device, and a server; wherein, the image acquisition device is connected to the display device, and the display device establishes a communication connection with the server; The virtual fitting method includes: the image acquisition device collects images associated with the user in real time, and performs image signal processing on the image associated with the user to generate image data associated with the user; the display device obtains the image data associated with the user, and The image data associated with the user is sent to the server; the server receives the image data associated with the user, generates a mannequin based on the image data associated with the user, and sends the mannequin to the display device; the display device adds clothing materials to the mannequin for composite rendering Model; the display device extracts action parameters from the image data associated with the user, and adjusts the model pose of the rendering model according to the action parameters to render a fitting picture.

An embodiment of the present disclosure also provides a computer-readable non-volatile storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the computer device executes the above method.

Description of drawings

FIG. 1 is a scene diagram of a virtual fitting system in an embodiment of the present disclosure;

FIG. 2 is a hardware configuration diagram of a display device in an embodiment of the present disclosure;

FIG. 3 is a schematic structural diagram of a home smart wardrobe in an embodiment of the present disclosure;

4 is a schematic structural diagram of a display device with a built-in camera in an embodiment of the present disclosure;

5 is a schematic structural diagram of a display device connected to an external image acquisition device in an embodiment of the present disclosure;

Fig. 6a is a schematic diagram of a virtual fitting interface in an embodiment of the present disclosure;

Fig. 6b is a display effect diagram of clothing option classification in an embodiment of the present disclosure;

Fig. 6c is a schematic diagram of an interface for identifying clothes in an embodiment of the present disclosure;

Fig. 6d is a schematic diagram of an interface for selecting a clothing color in an embodiment of the present disclosure;

FIG. 6e is a schematic diagram showing the effect of purchase links in an embodiment of the present disclosure;

FIG. 6f is a schematic diagram of a purchase interface displayed in an embodiment of the present disclosure;

Fig. 6g is a schematic diagram of the effect of displaying product sales positions in the smart fitting mirror in the embodiment of the present disclosure;

7 is a schematic flow diagram of a virtual fitting method in an embodiment of the present disclosure;

FIG. 8 is a software configuration diagram of a server in an embodiment of the present disclosure;

FIG. 9 is a schematic diagram of a fitting application interface in an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of a key frame-based action driving process in an embodiment of the present disclosure;

Fig. 11 is a schematic diagram of a process flow for matching associated clothes in an embodiment of the present disclosure;

FIG. 12 is a schematic diagram of an expression matching process in an embodiment of the present disclosure;

Fig. 13 is a sequence diagram of data interaction in the virtual fitting system in an embodiment of the present disclosure.

Detailed ways

In order to make the purpose and implementation of the application clearer, the following will clearly and completely describe the exemplary implementation of the application in conjunction with the accompanying drawings in the exemplary embodiment of the application. Obviously, the described exemplary embodiment is only the present application. Claim some of the examples, not all of them.

It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.

The terms "first", "second", and "third" in the description and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean limiting specific sequential or sequential unless otherwise noted. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The embodiments will be described in detail hereinafter, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following examples do not represent all implementations consistent with this disclosure. These are merely examples of systems and methods consistent with aspects of the disclosure as recited in the claims.

In the embodiment of the present disclosure, the virtual fitting system is a service platform integrating hardware and software, which can construct a virtual model through augmented reality (Augmented Reality, AR), artificial intelligence (Artificial Intelligence, AI) and 3D vision technologies. , and generate a display screen based on the virtual model.

Fig. 1 is an exemplary use scene diagram of the virtual fitting system in the embodiment of the present application of some embodiments of the present disclosure. As shown in FIG. 1 , the virtual fitting system provided by the present disclosure may include a control device 100 , a display device 200 , an intelligent terminal 300 , and a server 400 . The virtual fitting system can realize the virtual fitting function through the collaborative work among multiple devices. Wherein, the control device 100 and the smart terminal 300 can be used for user interaction, and can input control instructions to the display device 200 based on the virtual fitting user interface provided by the display device 200 . Both the display device 200 and the server 400 have data processing capability. The display device 200 is deployed locally, the server 400 is deployed in the cloud, and the display device 200 and the server 400 can exchange data.

The control device 100 can receive the user's input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 .

The control device 100 may be a remote controller, and the communication between the remote controller and the display device 200 includes infrared protocol communication, bluetooth protocol communication, and other short-distance communication methods, and the display device 200 is controlled wirelessly or wiredly. The user can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, and the like.

In some embodiments, a smart terminal 300 (such as a mobile terminal, a tablet computer, a computer, a notebook computer, etc.) can also be used to control the display device 200 . For example, the display device 200 is controlled using an application program running on a smart terminal.

In some embodiments, the display device may not use the above-mentioned smart terminal or control device to receive instructions, but may receive user control through touch or gesture.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart terminal 300. For example, the module for acquiring voice commands configured inside the display device 200 can directly receive the user's voice command control , the user's voice command control can also be received through the voice control device installed outside the display device 200.

It should be noted that the display device 200 generally refers to devices with data processing capabilities and screen display capabilities, including but not limited to smart TVs, personal computers, display terminals, smart advertising screens, Rubik's Cube screens, virtual reality devices, augmented reality devices, smart Wearable devices, robot assistants, smart fitting mirrors in shopping malls, smart wardrobes at home, etc. In some embodiments, FIG. 2 is an exemplary hardware configuration diagram of a display device in some embodiments of the present disclosure. As shown in FIG. 2 , the display device 200 includes one or more combinations of functional modules such as a power supply 210 , a communicator 220 , a memory 230 , an interface module 240 , a controller 250 , and a display 260 .

Wherein, the power supply 210 can supply power to the display device 200 so that each functional module can be powered on and run. The communicator 220 is used to establish a communication connection relationship between the display device 200 and the server 400, for example, the display device 200 communicates through a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 may provide various contents and interactions to the display device 200 . The memory 230 is used to store various information and application data. The interface module 240 is used to connect the display device 200 with peripheral devices to realize the input or output of specific types of signals. The controller 250 controls the operation of the display device 200 and responds to user's operations by running various software control programs stored in the memory 230 .

The display 260 is used to present a user interface, so that the display device 200 has a screen display function, and the display device 200 can present a specific display screen on the display 260 by running the application program in the memory 230 and displaying the process. For example, playback interface, user interface, application program interface, etc.

For different types of display devices 200, the display 260 may take different forms and have different display ratios. For display devices 200 such as smart TVs, personal computers, and display terminals, the display 260 may have a shape corresponding to a standard display screen ratio. For example, the display resolution of a smart TV is 3840×2160; the display resolution of a personal computer is 1920×1080; the display resolution of a display terminal is 2400×1080. As for display devices 200 such as virtual reality devices, smart dressing mirrors in shopping malls, and home smart wardrobes, the displays 260 included in them can be designed in a shape and proportion suitable for actual purposes. For example, the display 260 of the virtual reality device includes two square screens on the left and right; FIG. 3 shows a schematic structural diagram of a home smart wardrobe in some embodiments of the present disclosure. As shown in FIG. Elongated display equal to the width and height of the wardrobe door.

Because the virtual fitting system can synthesize images according to the portrait images associated with the user, thereby generating a fitting effect image. Therefore, in order to perform frame composition, the display device 200 should be able to acquire a portrait frame associated with the user. In some embodiments, the display device 200 can acquire portrait images through the built-in image acquisition module 270 , that is, the display device 200 further includes the image acquisition module 270 on the basis of including the above-mentioned various functional modules. For example, FIG. 4 shows a schematic structural diagram of a display device with a built-in camera in some embodiments of the present disclosure. As shown in FIG. 4 , the image acquisition module is a camera disposed on the top or bottom of the display device 200 .

In some embodiments, the display device 200 can acquire portrait images through an external image acquisition device 500 . FIG. 5 shows a schematic structural diagram of a display device externally connected to an image acquisition device in some embodiments of the present disclosure, that is, as shown in FIG. 5 , the display device 200 can be connected to the image acquisition device 500 through the interface module 240 . The image acquisition device 500 has a built-in camera and a transmission circuit, and can take pictures of the user through the camera, and then send the captured image or video to the display device 200 for display through the transmission circuit and the interface module 240 .

In the virtual fitting system, the display device 200 acts as a direct interaction device for the user, which can receive the user's control instructions, and perform data processing according to the control instructions to form a user interface containing different contents, which are presented through the display 260 . In some embodiments, the display device 200 can be used as a dedicated device of the virtual fitting system, that is, the display device 200 is only used to run the virtual fitting program and present the virtual fitting interface. For example, the display device 200 can be applied to a robot assistant in a shopping mall environment, and the robot assistant can perform voice interaction with the user to realize a virtual fitting function.

In some embodiments, the display device 200 can also be an implementation device of the virtual fitting system, that is, the display device 200 has many functions, and the virtual fitting function is one of the many functions. For example, the display device 200 may be a personal computer, and a virtual fitting application may be installed in the personal computer so that it can implement a virtual fitting function.

In some embodiments, various application programs can be installed in the display device 200 for realizing specific functions. The installed applications can be system applications or third-party applications. For example, a "virtual fitting" application program for users to download and install may be displayed in the application store provided by the operator of the display device 200 . Clothing material can be pre-built in the "virtual fitting" application program, then the display device 200 can run the application program in response to the user's input operation, and receive the image associated with the user, and the display device 200 can compare the image associated with the user with the pre-installed The constructed clothing material is synthesized, and the synthesis effect is displayed to achieve the purpose of fitting.

The fitting application program can be an independent application program or a functional module integrated in a specific application. For example, for a shopping application, in order to enable the user to understand the wearing effect of the clothing to be purchased during the online shopping process, the display device 200 may display a control or icon for activating the "virtual fitting" function in the shopping application. Compatible with independent applications, after enabling the "virtual fitting" function, the shopping application can display a prompt interface to the user, and the prompt interface includes controls or icons used to guide the user to input an image associated with the user. The display device 200 invokes the current material model with the purchased clothing, thereby using the "virtual fitting" function to synthesize the image associated with the user with the material model, and output a picture of the fitting effect.

During the process of implementing the virtual fitting function, the display device 200 may perform image synthesis processing based on the acquired portrait picture associated with the user. In some embodiments, the display device 200 synthesizes the portrait image and the clothing image by adding virtual clothing patterns based on the portrait image to realize virtual fitting. For example, after acquiring the portrait picture associated with the user, the display device 200 can perform feature recognition on the portrait picture to identify the wearable position of the portrait pattern, including upper limbs, lower limbs, hands, feet, neck, top of the head, etc. Then extract the virtual clothing material from the clothing material library, so as to add the corresponding clothing material at each dressing position to complete the virtual fitting.

In some embodiments, the display device 200 may also perform screen synthesis by adding portrait patterns based on virtual clothing patterns. For example, the virtual fitting application can add a portrait display area to the head area corresponding to the virtual clothing pattern, and display the head image in the acquired portrait pattern picture in the display area, thereby synthesizing a virtual fitting picture.

Wherein, the virtual clothing pattern can be obtained through clothing materials stored in the memory 230 or the cloud server 400 . For example, the operator of the virtual fitting system can generate virtual clothing materials by performing multi-angle image shooting, 3D modeling and other processing methods according to the popular clothing currently on sale. The generated virtual clothing material can be stored in the server 400 of the virtual fitting system. When the display device 200 starts the virtual fitting application and selects the clothing to be tried on, the display device 200 can request the virtual clothing material from the server 400 according to the user's selection. .

After obtaining the virtual clothing material, the display device 200 may cache the requested virtual clothing material according to the usage of its own memory 230 . Therefore, when using the virtual fitting system subsequently, according to the clothing to be tried on selected by the user, the corresponding virtual clothing material can be first matched in the local cache of the memory 230, and when the local cache includes the corresponding virtual clothing material, the corresponding virtual clothing material can be extracted Virtual clothing material. And when the corresponding virtual clothing material is not matched in the local cache, the corresponding virtual clothing material may be requested to the server 400 again.

In some embodiments, the virtual clothing material may include parameters with multiple dimensions independent of each other, including: clothing style, color, material, and the like. The display device 200 can present a variety of different clothing styles through the arrangement and combination of parameters in different dimensions. For example, the same virtual clothing material can be used for the same type of clothing, and the appearance of different clothing can be obtained by adjusting parameters such as color and material during virtual try-on. Therefore, the display device 200 can combine multiple clothing materials with less model data, reducing the construction amount of clothing models and the amount of data transmission when requesting clothing materials.

In order to synthesize a virtual fitting effect, in some embodiments, the display device 200 may present a virtual fitting interface for the user after running the virtual fitting application. Fig. 6a shows a schematic diagram of a virtual fitting interface of some embodiments of the present disclosure. As shown in Fig. 6a, the virtual fitting interface may include display windows, clothing options, control options, and outfit recommendation options. The virtual fitting interface can display different options to the user, and the display device 200 responds to the user's selection operation on the virtual fitting interface to control and synthesize a virtual fitting image. For example, the portrait image of the user collected by the image acquisition device 500 can be displayed in the display window in real time, and if the selected clothing option is "clothing A", the display device 200 can call the "clothing A" from the local cache or the server 400. ” and display the corresponding virtual clothing material through the display window. Similarly, if the option of "clothing B" is selected among the multiple clothes, the display device 200 may call the virtual clothes material corresponding to "clothes B" and display it through the display window.

The clothing options in the virtual fitting interface can be classified according to the actual wearing position, such as tops, bottoms, shoes, hats, accessories and other categories. Clothing options between different categories support simultaneous selection, but clothing options in the same category do not support multiple selections. For example, FIG. 6b shows an exemplary clothing option classification display effect diagram of some embodiments of the present disclosure. As shown in FIG. For the operation of "pants B" in the bottoms, the display device 200 can simultaneously display images corresponding to the selected top clothing "top A" and bottom clothing "pants B" in the display window. As for "Top A" and "Top B" which are both tops, they cannot be selected at the same time, but the clothing material selected later replaces the clothing material selected first.

Different types of virtual clothing materials can adopt different clothing styles, so the display device 200 needs to call different virtual clothing materials. However, in some embodiments, since there are many kinds of clothes, virtual clothes materials of the same category can be further divided into more detailed categories. For example, the classification of tops can be further divided into categories such as spring clothes, summer clothes, autumn clothes, and winter clothes according to the wearing season; it can also be further divided into categories such as coats and shirts according to the wearing position. Similarly, for different categories that are further divided, the display device 200 also needs to call different virtual clothing materials.

When performing virtual fitting interaction through the display device 200, in addition to selecting the clothes to try on through the above clothing options, the clothes to try on can also be selected through other interactive ways. In some embodiments, the display device 200 determines the clothing to be tried on selected by the user by acquiring information such as the product name and item number of the clothing input by the user. For this type of interaction, the display device 200 can search the clothing material library according to the user information, so as to call the clothing material related to the input information.

In some embodiments, the display device 200 can also identify image information such as display pictures, physical images, and barcodes on hang tags based on image recognition technology to determine the clothing to be tried on. For example, as shown in Figure 6a and Figure 6c, Figure 6c shows a schematic diagram of an interface for identifying clothing in some embodiments of the present disclosure, and the virtual fitting interface may include the option "Identify Clothes", if the option "Identify Clothes" is selected by the user option, the display device 200 can automatically start the image acquisition device 500 to acquire the clothing image, and recognize the clothing image, and call the virtual clothing material according to the image recognition result. Among them, the basis for identifying the clothing image can be calculated by calculating the similarity between the captured clothing image and the preset standard image, and the standard image with the highest similarity is used as the recognition result, and stored in the database. Call the virtual clothing material corresponding to the standard image.

A clothing option in the virtual fitting interface corresponds to a type of virtual fitting material. After determining a clothing option selected by the user through the virtual fitting interface, the clothing parameters can be adjusted through the control options. For example, FIG. 6d shows a schematic diagram of an interface for selecting the color of clothing in some embodiments of the present disclosure. As shown in FIG. 6d, after selecting the "clothing A" option, the display device 200 can display color control options in the virtual fitting interface, so that Show the user the function of selecting any color from the preset multiple color control options, and control the current color of the virtual clothing screen displayed in the display window.

In addition to controlling the color of the selected clothing material, the control options can also be used to control and adjust the display screen of the display window, for example, rotating the display angle, partially zooming in, adjusting brightness, and beautifying functions. The interactive action is adjusted based on the input parameters matched with the corresponding option selected by the user, and the presentation effect of the display window on the display device 200 is controlled.

The outfit recommendation option is used to enable or disable the outfit recommendation function of the virtual fitting application. The clothing recommendation function can automatically display the recommended clothing screen in the display window according to the clothing selected by the user after determining any clothing options selected by the user. The clothing recommendation function can match the clothing materials of unselected categories in the virtual clothing material library according to a specific style algorithm and based on the clothing options selected by the user. The outfit recommendation algorithm can perform unified matching operations based on dimensions such as color, type, and style. For example, in response to the user's operation of selecting black and formal tops, the outfit recommendation algorithm can automatically match black and formal bottoms and shoes, and call the corresponding virtual clothing model, which will be displayed together with the selected top virtual clothing model in the display window.

In addition to the display window, clothing options, control options, and outfit recommendation options, in some embodiments, the virtual fitting interface presented by the display device 200 may also include prompt content and link options. For example, the virtual fitting interface can also include a "save outfit" option. When it is obtained that the user has completed the fitting operation by selecting multiple clothing options, the user can click the "save outfit" option to save all items in the current display window. Showcase patterns. At the same time, the virtual fitting application can also automatically display the purchase link or shopping guide information of the corresponding clothing for the user after responding to the user's click "save outfit", so that the user can purchase the corresponding clothing according to the virtual fitting results.

Obviously, for different types of display devices 200, the prompt content and link options displayed on the virtual fitting interface may be different. For example, FIG. 6e shows a schematic diagram of displaying purchase link effects in some embodiments of the present disclosure, and FIG. 6f shows a schematic diagram of displaying purchase interface in some embodiments of the present disclosure. As shown in FIG. 6e and FIG. 6f, when the computer runs shopping applications, After acquiring that the user selects "Save Outfit", the display device 200 may present a product link corresponding to the virtual clothing pattern displayed in the current display window for the user to select. After obtaining any product link selected by the user, the display device 200 can jump from the virtual fitting interface to the product detail interface. When using the smart fitting mirror in the shopping mall, if it is recognized that the user has selected "Save Outfit", the location of the sales store corresponding to the clothing in the current display window can be displayed on the virtual fitting interface, such as "Clothing A can be sold on the third floor. Area A 306 "purchased by ××", as shown in Figure 6g, which shows a schematic diagram of the effect of the smart fitting mirror displaying the product sales location in some embodiments of the present disclosure.

It can be seen that no matter the virtual fitting application or the shopping application, in order to output the fitting effect screen, it is necessary to obtain the image associated with the user input by the user. Since different applications present different forms of the final fitting effect screen, images associated with the user may be input in different ways. That is, in some embodiments, in order to synthesize a 2D, static fitting image screen, the application prompts the user to display an icon or control for inputting an image associated with the user, responds to the user's operation, opens the file manager, and selects Select a picture file in the save path of the file, and use the picture file as the image input associated with the user. The built-in camera of the display device 200 or the external image acquisition device can also take photos for the user, and use the images obtained by the photos as images associated with the user to input into the application.

However, for 2D static image display, it is impossible to display the fitting effect in real time, and it is also impossible to display the effects of various wearing postures and viewing angles, resulting in a poor final effect presented by this virtual fitting application. For this reason, in some embodiments, the virtual fitting application can also provide a dynamic fitting image screen, so that the dynamic fitting image screen can obtain the information associated with the user input by the user by uploading a video file or recording a video file in real time. image. The input video image can be displayed in a specific area in the application interface to obtain a better composite effect.

In order to obtain a better fitting effect, in some embodiments, the display device 200 can also use augmented reality technology to add a virtual clothing material model to the video screen uploaded by the user and associated with the user, and to The target is tracked, so that the clothing material model can follow the movement changes of the character target in the video. However, this dynamic fitting method is limited by the limitations of AR technology, resulting in poor fusion of clothing material and character targets in the video, and clothing cannot accurately fit the character targets. In addition, there is a large delay in the display effect of the clothes following the movements of the characters, so the interactive experience of the virtual fitting function is poor.

In order to obtain a better virtual fitting effect and interactive experience, some embodiments of the present disclosure provide a virtual fitting method, which can be applied to a virtual fitting system for displaying a fitting effect screen in real time. Wherein, the virtual fitting system includes a display device 200 , a server 400 and an image acquisition device 500 . The image collection device 500 is used to collect images associated with the user in real time, and process image signals to form image data associated with the user. A model reconstruction application can be built in the server 400, and the model reconstruction application can generate a human body model according to the image data associated with the user. The server 400 sends the human body model to the display device 200 . The display device 200 is used to run the virtual fitting application program, and render the human body model to synthesize and display the virtual fitting picture. Specifically, the virtual fitting method includes the following:

The image associated with the user is collected in real time by the image collection device 500 . The image acquisition device 500 may include a camera and a data processing module, wherein the camera may capture images of the user's environment to obtain images associated with the user. The data processing module may perform image signal processing on the image associated with the user to generate image data associated with the user, and input the image data associated with the user into the display device 200 as an image signal.

In some embodiments, the image acquisition device 500 may be a functional module built in the display device 200, and the display device 200 is an integral device. For example, as shown in FIG. 4, the image acquisition device 500 is a camera on the display device 200, which can be under the unified control of the controller 250 in the display device 200, and can directly send the collected images associated with the user to the controller 250. . Obviously, the camera should be set at a specific position on the display device 200 in order to collect images associated with the user. For example, for a display device 200 such as a smart TV, the camera can be set on the top of the smart TV, and the shooting direction of the camera is the same as the light emitting direction of the screen of the smart TV, so that images associated with the user located in front of the screen can be captured.

In some embodiments, the image acquisition device 500 can also be an external device connected to the display device 200, that is, as shown in FIG. High Definition Multimedia Interface, HDMI interface), analog or data high-definition component input interface, composite video broadcast signal (Composite Video Broadcast Signal, CVBS) interface, (Universal Serial Bus (Universal Serial Bus, USB) interface and one of the RGB ports can support a specific data transmission method. Then the image acquisition device 500 can send the collected image associated with the user to the display device 200 through the specific data transmission method supported by the interface after accessing the interface module 240. For example, after being connected to the smart TV, the image acquisition device 500 may communicate with the smart TV through Open Natural Interaction (OpenNI), so as to send the collected image associated with the user to the smart TV.

In order to achieve a better dynamic fitting effect, in addition to the image content, the image data associated with the user sent by the image acquisition device 500 to the display device 200 may also include image recognition results, bone parameters, expression parameters, and gesture recognition results. data etc. Therefore, in some embodiments, the data processing module of the image acquisition device 500 can also have a built-in image processing application. After the camera captures an image associated with the user, the data processing module can run the image processing application to process the image associated with the user. The image is recognized.

For example, the image acquisition device 500 can identify "image-depth" data, 3D human skeleton key point coordinates, portrait head recognition position, portrait target tracking point, etc. in the image associated with the user by running image processing applications with different functions. content. Therefore, when the recognized content is sent to the display device 200 , the display device 200 can be provided with functional supports such as RGBD image data, limb driving, human body tracking, face reconstruction material and expression driving material. Accordingly, these data may, together with user-associated images, constitute user-associated image data.

In order to obtain corresponding data, the image acquisition device 500 needs to have specific hardware support. For example, in order to be able to identify the image depth in the image, the image acquisition device 500 can be a camera group with multiple lenses, and the multiple lenses can detect the same target from different positions, so that by comparing the image capture results and angle differences of multiple lenses , to calculate the depth information of the target.

Apparently, the image acquisition device 500 can also integrate sensor elements with specific functions, so that the images collected by the device 500 associated with the user include data for three-dimensional modeling. For example, in the image acquisition device 500, a laser radar component can be built in, and the laser radar can emit laser light to the shooting area, and detect the reflected laser signal to measure the distance between the target in the shooting area and the image acquisition device 500, and compare the scanning result with the image acquisition device 500. Each frame of image captured by the camera is correlated to generate user-associated image data with image depth.

As shown in FIG. 7 , FIG. 7 shows a schematic flowchart of a virtual fitting method in some embodiments of the present disclosure. After the image acquisition device 500 acquires the image associated with the user and generates the image data associated with the user, it can send the image data associated with the user to the display device 200 . The display device 200 acquires the image data associated with the user as data input for the fitting application. The display device 200 may send an activation command to the image acquisition device 500 after the fitting application is started, control the image acquisition device 500 to start capturing images associated with the user, and perform processing on the images associated with the user to obtain images associated with the user data.

In some embodiments, according to the different image effects presented by the fitting application launched by the display device 200 , the display device 200 may acquire image data associated with the user including different contents from the image acquisition device 500 . For example, when the display device 200 starts a fitting application that can only display 2D static effects, the display device 200 may set the type of image data associated with the user to include only image content in the start instruction sent to the image capture device 500 . When the display device 200 starts a fitting application capable of displaying 3D dynamic effects, the image data associated with the user needs to be specified in the start command, including "image-depth" data, 3D human skeleton key point coordinates and other data content.

Similarly, for different fitting applications, in the image data associated with the user acquired by the display device 200, the image forms may also be different. For example, for a fitting application with static effects, the image data associated with the user in the form of a picture can be obtained from the image acquisition device 500; image data associated with the user.

After the display device 200 acquires the image data associated with the user, it can render the fitting result screen according to the image data associated with the user. In order to present a better clothing fusion effect, the display device 200 may obtain a fitting screen based on a human body model. Therefore, after acquiring the image data associated with the user, the display device 200 can create a human body model according to the image data associated with the user. However, due to the huge amount of data processing during the creation of the human body model and the limitation of the hardware configuration of the display device 200, the quality of the human body model locally created by the display device 200 is rough and the accuracy is poor. Therefore, in some embodiments, after acquiring the image data associated with the user, the display device 200 may send the image data associated with the user to the server 400 for modeling processing.

For this purpose, a model reconstruction application may be built in the server 400 . A model reconstruction application can generate a mannequin based on image data associated with a user. In the server 400, multiple initial human body models may be pre-configured, and multiple initial human body models may be set according to factors such as age and gender. After the display device 200 sends the image data associated with the user to the server 400, the server 400 can first identify the image in the image data associated with the user to determine the age, gender and other information of the person in the image, and then based on these information from Call the appropriate one among the preset multiple initial human models.

After invoking the initial human body model, the server 400 can also read data such as "image-depth" data and 3D human skeleton key point coordinates in the image data associated with the user, and set the initial human body model according to the read data, Modify, adjust, so that the initial model gradually has portrait features in the image data associated with the user. For example, the server 400 may extract the proportional relationship between the user's head width and shoulder width from the image data associated with the user, and adjust the head width and shoulder width in the initial model according to the proportional relationship.

In some embodiments, as shown in FIG. 8 , FIG. 8 shows a software configuration diagram of an exemplary server in some embodiments of the present disclosure. The human body model created by the server 400 may include multiple parts, each part can be parameterized and adjusted independently, and supports replacement through a preset material library, so as to quickly generate the human body model. For example, a variety of hairstyle materials can be pre-stored in the server 400. After the server 400 receives the image data associated with the user, it can match similar models in the hairstyle material library according to the portrait hairstyle and color in the image data associated with the user. , and add the hair model to the mannequin's head as the initial hairstyle.

In the subsequent use, the fitting application has the function of allowing the user to arbitrarily choose to change the hairstyle of the character in the virtual fitting effect screen. After obtaining the user's choice to change a specific hairstyle, the display device 200 can then request the server 400 to rebuild. model, the server 400 can select a new hairstyle material from the hairstyle material library and add it to the human body model to reconstruct the human body model.

In order to create a more detailed human body model, in some embodiments, multiple modeling modules can be built in the server 400, including a head reconstruction module, a body reconstruction module, an expression recognition module, a trial hair module, and the like. Each modeling module can construct a corresponding virtual model through a specific model reconstruction method. For example, head modeling can further include geometric reconstruction and texture reconstruction units, wherein the geometric reconstruction unit can perform point cloud denoising processing, triangulation processing, point cloud smoothing processing, and point cloud fusion processing according to the image data associated with the user , so that the shape of the head model gradually tends to be consistent with the person target in the image data associated with the user. The texture reconstruction unit can perform portrait segmentation processing, skin color migration processing, and skin color fusion processing on the head model, so that the appearance of the head model is gradually consistent with the person target in the image data associated with the user.

Similarly, for the body reconstruction module, models of body parts can also be generated based on specific modeling methods. For example, in the body reconstruction module, a body model can be created through a skinned multi-person linear model (Skinned Multi-Person Linear Model, SMPL). Therefore, the body reconstruction module may include an SMPL model generation unit, a parameterized mapping unit, and a stitching unit. When reconstructing the body model, the server 400 can generate an initial model through the SMPL model generation unit, and then extract human body parameters from the image data associated with the user, so as to perform parameter mapping from the SMPL model to the parameterized model, and obtain an image corresponding to the user. The parametric body model of the data, and finally the body model and the head model are spliced through the splicing unit to obtain the human body model.

In some embodiments, in order to obtain the human body model, when the display device 200 sends the image data associated with the user to the server 400, it can identify the portrait target from the image data associated with the user, and add bones to the portrait target according to the recognition result key points to generate bone parameters. Then send the skeleton parameters to the server 400, so that the server 400 can set the joint point positions of the human body model according to the skeleton parameters. The human body model after adding the bone parameters can change according to the rules corresponding to the bone parameters, so as to simulate a model posture that is more in line with the real state of the character.

In addition, for the expression recognition module, the built-in processing unit can also be used to establish the expression base model and estimate the expression parameters, so as to set specific expression forms for the human body model according to the expressions of the characters in the image data associated with the user. As for the trial hair module, the preset functional units can be used to set the hair matching function and hair penetration processing function, etc. to create a more realistic hair model.

Since the server 400 generates or reconstructs the human body model for the subsequent virtual fitting process, so in some embodiments, in addition to the built-in application for generating the human body model, the server 400 may also have a built-in application for generating the clothing model . The server 400 can perform clothing modeling, clothing material simulation, and clothing deformation processing by running the application, so as to create a clothing model based on clothing image data.

Similar to the human body model, when the server 400 creates the clothing model, the display device 200 can also send the clothing image data to the server 400. The clothing image data can be obtained by taking images of the clothing from multiple angles. Get six views of clothing in the product interface of , namely front view, back view, left view, right view, bottom view and top view. After sending the pictures from different angles to the server 400, the server 400 can automatically model and generate clothes patterns according to these images, and then through rendering processes such as cloth simulation and clothes deformation, a clothes model that conforms to the real effect can be obtained.

After the server 400 generates the human body model, it may send the generated human body model to the display device 200 . After receiving the human body model, the display device 200 renders the human body model to render an image with a fitting effect. During the rendering of the human body model, the display device 200 may add clothing materials to the human body model to synthesize a rendered model.

During the process of synthesizing the rendering model, the display device 200 may also optimize the human body model sent by the server 400 . For example, the display device 200 can perform head synthesis and restoration processing on the human body model based on the Unity engine, including performing head shape processing through the meshfilter tool and meshrender tool; performing texture processing through the texture tool and normalmap tool; and performing head shape processing through the blendshape/morph tool. The face parameter model algorithm is used to process expression parameter information and adjust the hair to fit the head shape.

In some embodiments, in order to show the user the effect of comparative fitting, the display device 200 may also display the image part in the image data associated with the user while sending the image data associated with the user to the server 400 . For example, as shown in FIG. 9 , FIG. 9 shows a schematic diagram of a fitting application interface of some embodiments of the present disclosure. The program interface of a fitting application program can include a fitting window and an original window. The rendering results of the human body model and clothing materials can be displayed in the fitting window, and the image data associated with the user can be displayed in the original window. The image part is the image frame captured by the image acquisition device 500 in real time.

After the display device 200 synthesizes the rendered model, it can also extract action parameters from the image data associated with the user, and adjust the model pose of the rendered model according to the action parameters to render a fitting picture. That is, the display device 200 may determine the user's action according to the image data associated with the user, and control the rendering model to follow the user's action corresponding to the image data associated with the user. Among them, the actions that can be presented in the fitting screen include body actions, head actions, facial expressions, gesture actions, and clothing actions. The motion parameters can be calculated and obtained by the image acquisition device 500 through head detection, 3D human skeleton key point detection, facial expression detection, and gesture recognition detection.

The display device 200 can detect the user's action by comparing multiple frames of images in the image data associated with the user. That is, in some embodiments, in the step of adjusting the model pose of the rendered model according to the action parameters, the display device 200 can traverse the skeleton key points of each frame image in the image data associated with the user; Skeleton key point position to obtain the movement distance of each bone key point to move the joint point position of the human body model according to the movement distance. Through the coordinated adjustment of multiple skeleton key points, the display device 200 can adjust the pose of the person in the rendered model to follow the image content of each frame of the image data associated with the user to the same action, realizing the action following effect.

In order to achieve the action following effect, the display device 200 can make the human body model produce a reasonable running mode by skinning, that is, adding bones to the model. For example, after obtaining the human body model, the display device 200 can skin the human body model and the clothing model and provide a texture for the cloth simulation algorithm for skinning, and use the blendshape/morph tool to process the skeleton skinning animation for the model.

It can be seen that, in the above-mentioned embodiment, the virtual fitting method can collect image data associated with the user through the image acquisition device 500 in real time, and generate a human body model through the server 400 according to the image data associated with the user, and display a human body model on the display device 200. Add clothing materials to the model to form a rendered model. At the same time, the display device 200 can also acquire user actions according to the image data associated with the user, and adjust the model posture of the rendering model in real time according to the user actions, so as to realize avatar driving. The method can realize the effect of 3D, dynamic, real-time and multi-angle display of the fitting screen through the human body model. Moreover, the server 400 executes model building and reconstruction, which can share the data processing load of the display device 200 and is conducive to generating a more detailed and realistic character model.

Since the display device 200 is deployed locally and the server 400 is deployed in the cloud, the data communication between the display device 200 and the server 400 will be affected by factors such as network delay, resulting in that after the image data associated with the user is sent to the server 400, There may be a delay in receiving the manikin. The waiting time for displaying the fitting effect is prolonged. Therefore, in the process of presenting the final fitting screen, the number of data interactions between the display device 200 and the server 400 should be minimized to reduce delay. For this, in some embodiments, the display device 200 may monitor the portrait target after obtaining the human body model for the first time, so that the human body model obtained for the first time is always used when there is no obvious characteristic change. That is, as shown in FIG. 10 , FIG. 10 shows a schematic diagram of a key frame-based action driving process in some embodiments of the present disclosure. After the display device 200 sends the image data associated with the user to the server 400, FIG. 10 includes at least the following steps:

S1001. Extract an initial key frame image from image data associated with a user.

S1002. Extract image depth parameters from the initial key frame image, and record the initial key frame image.

S1003. Send both the initial key frame and the image depth parameter to the server 400.

The server 400 sets the human body model parameters according to the image depth parameters, and sends the human body model to the display device 200 . After the display device 200 obtains the human body model, the display device 200 may execute S1004 in subsequent applications.

S1004. Extract real-time key frame images from the image data associated with the user.

Wherein, the key frame may be a frame of image corresponding to a specific time point. For example, the display device 200 may extract a frame of image from the image data associated with the user every 1 second as a key frame. The key may also be a frame of image obtained at intervals of a certain number of frames. For example, the display device 200 may extract a frame of image from image data associated with the user at an interval of 20 frames as a key frame. The key frame may also be a frame of image with a distinctive portrait target obtained by performing image recognition on the image frame. For example, by inputting the image data associated with the user into the portrait recognition model frame by frame, when it is determined by the model recognition that the image frame contains a portrait, it will be marked as a key frame.

S1005. Comparing the initial key frame image and the real-time key frame image to obtain image similarity.

Wherein, the initial key frame is an image frame used to extract image depth parameters to generate a human body model. The real-time key frame is the key frame extracted by the display device 200 in real time from the image data associated with the user. The image similarity can be determined by calculating the histograms of the two images separately, and then calculating the normalized correlation coefficient of the two histograms, such as Bhattacharyachian distance, histogram intersection distance and other data, to determine the similarity of the two images. When the user is in a static state, the content of the two key frame images is the same, but when the user is in an action state, the content of the two key frame images is different, and the greater the range of motion, the lower the similarity of the content of the two frame images , so the user's action state can be detected by image similarity.

S1006. Determine whether the image similarity is greater than a preset similarity threshold, if so, execute S1007, otherwise execute S1008.

After comparing the initial key frame and real-time key frame images to obtain the image similarity, the display device 200 may compare the image similarity with a preset similarity threshold.

S1007. Extracting action parameters.

If the image similarity is greater than or equal to the preset similarity threshold, that is, the actual motion range of the user in the two key frames is small, or in a static state, there is no need to re-model at this time, and the original human body model can still be used to obtain better results. The display effect of the fitting screen, that is, the display device 200 can execute the step of extracting motion parameters from the image data associated with the user, so as to drive the human body model to generate motions according to the motion parameters.

S1008. Extract image depth parameters from real-time key frame images.

If the image similarity is less than the preset similarity threshold, that is, the content in the two key frames differs greatly, at this time, in order to obtain a more realistic model effect, the display device 200 can extract the image depth parameter from the real-time key frame image, and The extracted image depth parameters are sent to the server 400, so that the server 400 reconstructs the human body model according to the image depth parameters.

As the image acquisition device 500 continuously acquires images associated with the user, the screen content included in the image may gradually vary with the user's actions, that is, the image similarity between the initial key frame and the subsequently extracted key frame gradually decreases. Therefore, In order to maintain the consistency of the display device 200 driving the model action, reduce the number of times of remodeling.

S1009. Replace the recorded initial key frame image with the real-time key frame image.

In some embodiments, the display device 200 may record the initial key frame image after extracting the image depth parameter from the initial key frame image. And after the step of extracting motion parameters from the image data associated with the user is performed, or after the step of sending the image depth parameters to the server, the real-time key frame images are used to replace the recorded initial key frame images.

For example, after a fitting application is started, the display device 200 may first extract an initial key frame T0 from image data associated with the user, and send the initial key frame T0 to the server 400 to generate a human body model. Moreover, when the display device 200 subsequently renders the fitting picture of the mannequin, it can continuously acquire the real-time key frame, that is, T1. Then the display device 200 can compare the initial key frame T0 and the real-time key frame T1 after acquiring the real-time key frame T1, and calculate the similarity S01 of the two frames of images. When the similarity S01 is greater than or equal to the similarity threshold S, then start Action parameters are extracted from the key frame T1 or the image data associated with the user to drive the human body model to generate actions and follow the changes of the user's actions. After extracting the motion parameters, the display device 200 may use the real-time key frame T1 to replace the initial key frame T0 as the initial key frame in the subsequent motion judgment process. That is, when the real-time key frame T2 is obtained, the similarity between the key frame T1 and the key frame T2 can be compared to continue tracking user actions or reconstructing the human body model.

It can be seen that, the above embodiment can continuously analyze the image frame associated with the user collected by the image collection device 500 through the initial key frame and the real-time key frame, so as to track the actual action of the user. Moreover, the display device 200 can determine whether it is necessary to reconstruct the human body model by comparing the image similarity between two adjacent key frames, so that the human body model can reduce the number of reconstruction models on the premise that it can be synchronized in time, and improve the fitting class. The responsiveness of the application.

In some embodiments, while the fitting application provides real-time dynamic fitting functions, it can also provide users with outfit recommendations, as shown in Figure 11, which shows the process of matching associated clothes in some embodiments of the present disclosure schematic diagram. During the process of adding clothing material to the human body model, the display device 200 may acquire a selection instruction input by the user for selecting clothing. For example, in a fitting application interface, multiple clothing options can be set, and any option in the interface that is selected can be determined by obtaining user interaction operations, so as to input a selection instruction. It can also be recognized that the user selects multiple target clothes among multiple clothes options, that is, at least one target clothes is specified in the selection command. For example, the selection instruction may be that the user can simultaneously select upload and download to input the selection instruction.

After the display device 200 receives the user's input selection instruction, the display device 200 may respond to the selection instruction and extract the target clothing material from the clothing material library. Because clothing can be divided into multiple categories, such as tops, bottoms, shoes, hats, bags, etc. These types can be matched with each other to form the final dressing effect. Therefore, when the selection instruction indicates that not all types are selected, the fitting application can automatically match other suitable types of clothing according to the characteristics of the selected clothing, so as to present a better fitting effect.

Therefore, while extracting the target clothing material, the display device 200 can match the associated clothing material in the clothing material library according to the target clothing material according to the preset dressing recommendation rules. Among them, the preset recommendation rules for outfits can be comprehensively set based on categories such as color, purpose, style, and applicable age. For example, when the user chooses blue and white tops with a small and fresh style, related clothing materials such as blue and white bottoms and shoes can be recommended to the user according to the preset recommendation rules for outfitting.

After obtaining the target clothing material and the associated clothing material, the display device 200 may add the target clothing material and the associated clothing material to the human body model, so as to form a rendering model with the effect of dressing in the opinion. Obviously, in order to recommend more types of dressing styles to the user, the display device 200 can pre-set multiple dressing recommendation rules, and each rule can match the appropriate associated clothing under the rule according to the target clothing material selected by the user. footage and preview it through multiple windows for enhanced display.

Due to the high complexity of the user's actual expression, the amount of parameter data that needs to be modified on the human body model under different expressions is huge. As a result, fitting applications need to process a large amount of data when performing expression tracking to modify the face in the human body model. feature. This will slow down the response speed of the application. Therefore, in some embodiments, the display device 200 can quickly implement expression switching in the form of a preset standard expression template. That is, as shown in FIG. 12 , FIG. 12 shows a schematic diagram of an expression matching process in some embodiments of the present disclosure. When adjusting the model pose of the rendering model according to the action parameters, the display device 200 can identify the head area in the image data associated with the user, and detect the user's expression in the head area, and then according to the expression type of the user's expression, Match the expression model of the same type as the expression in the preset expression library, thereby replacing the facial area in the rendered model with the expression model.

For example, the display device 200 may identify the head area in the image associated with the user by detecting the target shape in the image data associated with the user and the screen layout characteristics in the shape. From the head area, use the expression recognition model to recognize the current user's expression. Among them, the facial expression recognition model can be obtained by training an artificial intelligence model with sample data. That is, by inputting a large amount of sample data with expression labels into the initial model, and setting the model output result as the classification probability of the image belonging to a specific expression category, and then calculating the error between the classification probability and the expression label, and adjusting according to the error backpropagation Model parameters, so that the output result of the model is gradually the same as the label result, and an expression recognition model is obtained.

After the expression recognition model outputs the classification probability of the current image for a certain expression, the expression with the highest classification probability can be used as the user's expression, such as a smile. Then match the same type of expression model in the preset database according to the recognized user's expression, that is, the standard smile model, and use the standard smile model to replace the facial area in the rendering model, so that the rendering model can display smiling expressions.

It can be seen that the above embodiment can quickly replace the facial area in the rendering model by matching the standard expression model, so that the display device 200 does not need to modify the facial parameters of the model to reduce the amount of data processing. In addition, better facial expression tracking timeliness can be obtained, and the display effect can be improved.

Based on the above virtual fitting method, some embodiments of the present disclosure further provide a display device 200 including: a display 260 , a camera, a communicator 220 and a controller 250 . Wherein, the display 260 is configured to display images and/or user interfaces; the camera is configured to collect image data in real time, and the image data includes images associated with the user; the communicator 220 is configured to establish a communication connection with the server 400, and the server 400 has a built-in model a reconstruction application for generating a mannequin from image data associated with a user; the controller 250 is configured to perform the following procedural steps:

Get the image data associated with the user;

sending the image data associated with the user to the server, so that the server generates and sends the mannequin according to the image data associated with the user;

Add clothing material to the mannequin to synthesize the rendered model;

Extract action parameters from the image data associated with the user, and adjust the model pose of the rendering model according to the action parameters to render a fitting picture.

As shown in FIG. 13 , FIG. 13 shows an exemplary virtual fitting system data interaction sequence diagram of some embodiments of the present disclosure. In some embodiments, the present disclosure also provides a virtual fitting system, including: a display device 200, an image acquisition device 500, and a server 400; wherein, the image acquisition device 500 is connected to the display device 200, and the display device 200 establishes communication with the server 400 connect.

S1300. The image acquisition device 500 receives an image associated with the user from the user.

S1301. The image acquisition apparatus 500 performs image signal processing on an image associated with a user, so as to generate image data associated with the user.

S1302. The image acquisition apparatus 500 sends the image data associated with the user to the display device 200.

S1303. The display device 200 sends the image data associated with the user to the server 400.

S1304. The server 400 runs a model reconstruction application; generates a human body model according to the image data associated with the user.

S1305. The server 400 sends the human body model to the display device 200 .

S1306. The display device 200 adds clothing materials to the human body model to synthesize a rendered model; and extracts action parameters from image data associated with the user, and adjusts the model pose of the rendered model according to the action parameters to render a fitting picture.

S1307. The display device 200 sends a fitting screen to the device associated with the user.

It can be seen that, in the above embodiments, the entire virtual fitting system may include: a cloud server 400 , a local display device 200 and an image acquisition device 500 . Among them, the cloud server 400 can be responsible for human body reconstruction, including basic algorithm modules such as head reconstruction, body reconstruction, expression recognition, hair try-on, etc., and provides functional support for the display device 200 such as human body model, expression driving parameters, and hair try-on. The server 400 may rely on the data collected by the image collection device 500 as input, and output the modeling processing results to the display device 200 to support applications.

The image acquisition device 500 is responsible for providing acquisition data, including image signal processing (Image Signal Processing, ISP) debugging, RGBD stream, key points of 3D human skeleton, human head detection, multi-target tracking data, etc., so as to communicate and transmit through OpenNI, Provide the display device 200 with functional support for RGBD image data, limb driving, human body tracking, face reconstruction material, and expression driving material. The display device 200 can be responsible for functions such as rendering of the human body model, display and rendering of clothing materials, motion driving, and local parameter adjustment.

It can be seen that the virtual fitting system provided by the present disclosure can collect image data associated with the user in real time during use, and send the image data associated with the user to the server 400 to generate a human body model. Then add clothing materials to the human body model to synthesize the rendering model, and extract the character's actions in real time, so as to adjust the model pose of the rendering model according to the action parameters to form a fitting screen. The virtual fitting system can realize the dynamic 3D virtual fitting function, and display the user's actions in real time through the rendering model, so as to achieve the effect of clothes moving with people and recommended outfits, and solve the problem that the traditional virtual fitting method cannot display the fitting effect in real time.

Similar parts between the embodiments provided in the present disclosure can be referred to each other, and the specific implementation manners provided above are only a few examples under the general concept of the present disclosure, and do not constitute a limitation of the protection scope of the present disclosure. For those skilled in the art, any other implementations extended according to the solutions of the present disclosure on the premise of not paying creative efforts belong to the protection scope of the present disclosure.

Claims

A display device comprising:

a display configured to display images and/or a user interface;

a camera configured to collect image data in real time, the image data including images associated with the user;

The communicator is configured to establish a communication connection with a server, and the server has a built-in model reconstruction application for generating a human body model according to the image data associated with the user;

a controller, coupled to said display, camera and communicator, configured to:

Obtain the image data associated with the user;

sending the image data associated with the user to the server, so that the server generates a mannequin according to the image data associated with the user and sends the mannequin to the controller;

Adding clothing materials to the human body model to synthesize the rendered model;

Extracting action parameters from the image data associated with the user, and adjusting the model pose of the rendering model according to the action parameters, so as to render a fitting picture.
The display device according to claim 1, the controller is further configured to:

extracting an initial keyframe image from said image data associated with the user;

extracting image depth parameters in the initial key frame image;

Sending the image depth parameter to the server, so that the server sets the human body model parameter according to the image depth parameter.
The display device according to claim 2, the controller is further configured to:

extracting real-time keyframe images from the image data associated with the user;

comparing the initial key frame image with the real-time key frame image to obtain image similarity;

If the image similarity is greater than or equal to a preset similarity threshold, extracting action parameters from the image data associated with the user;

If the image similarity is less than the preset similarity threshold, extracting an image depth parameter from the real-time key frame image;

Sending the image depth parameters to the server, so that the server reconstructs a human body model according to the image depth parameters.
The display device according to claim 3, the controller is further configured to:

After extracting the image depth parameters in the initial key frame image, recording the initial key frame image;

After extracting the action parameters from the image data associated with the user, or after sending the image depth parameters to the server, the controller is further configured to:

The recorded initial key frame image is replaced with the real-time key frame image.
The display device according to claim 1, the controller is further configured to:

identifying a portrait subject from said image data associated with the user;

Add skeletal key points for the portrait target to generate skeletal parameters;

Sending the skeleton parameters to the server, so that the server sets the joint point positions of the human body model according to the skeleton parameters.
The display device according to claim 1, the controller is further configured to:

Obtaining a selection instruction input by the user for selecting clothing, where at least one target clothing is specified in the selection instruction;

Responding to the selection instruction, extracting the target clothing material from the clothing material library;

Matching associated clothing materials in the clothing material library according to the target clothing materials according to the preset recommendation rules for wearing;

Adding the target clothing material and the associated clothing material to the mannequin.
The display device according to claim 1, the controller is further configured to:

Traversing the skeletal key points of each frame of image in the image data associated with the user;

Compare the position of the bone key points in two adjacent frames of images to obtain the movement distance of each bone key point;

Move the joint point positions of the human body model according to the moving distance.
The display device according to claim 1, the controller is further configured to:

identifying a head region in said image data associated with the user;

detecting a user's expression in said head region;

According to the expression type to which the user's expression belongs, match an expression model identical to the expression type in a preset expression library;

The facial region in the rendered model is replaced with the expression model.
A virtual fitting system, comprising: a display device, an image acquisition device, and a server; wherein the image acquisition device is connected to the display device, and the display device establishes a communication connection with the server;

The image acquisition device is configured to collect images associated with the user in real time, and perform image signal processing on the image associated with the user to generate image data associated with the user; the image acquisition device is also configured to provide The display device sends the image data associated with the user;

The display device is configured to acquire the image data associated with the user, and send the image data associated with the user to the server;

A built-in model reconstruction application in the server, the server is configured to receive the image data associated with the user, and run the model reconstruction application; generate a human body model according to the image data associated with the user, and the server also configured to send the mannequin to the display device;

The display device is further configured to add clothing materials to the human body model to synthesize a rendered model; and extract motion parameters from the image data associated with the user, and adjust the rendering model according to the motion parameters. Model pose to render the fitting screen.
A virtual fitting method, applied to a virtual fitting system, the virtual fitting system includes: a display device, an image acquisition device, and a server; wherein, the image acquisition device is connected to the display device, and the display device is connected to the display device Said server establishes communication connection; Described virtual fitting method comprises:

The image collection device collects images associated with the user in real time, and performs image signal processing on the image associated with the user to generate image data associated with the user;

The display device acquires the image data associated with the user, and sends the image data associated with the user to the server;

The server receives the image data associated with the user, generates a human body model according to the image data associated with the user, and sends the human body model to the display device;

The display device adds clothing material to the human body model to synthesize a rendered model;

The display device extracts action parameters from the image data associated with the user, and adjusts the model pose of the rendering model according to the action parameters, so as to render a fitting picture.