CN117934690A

CN117934690A - Household soft management method, device, equipment and storage medium

Info

Publication number: CN117934690A
Application number: CN202410342672.4A
Authority: CN
Inventors: 周志胜
Original assignee: All House Premium Technology Shenzhen Co ltd
Current assignee: All House Premium Technology Shenzhen Co ltd
Priority date: 2024-03-25
Filing date: 2024-03-25
Publication date: 2024-04-26
Anticipated expiration: 2044-03-25
Also published as: CN117934690B

Abstract

The invention provides a household soft management method, a device, equipment and a storage medium, wherein the method comprises the following steps: and generating and coupling pseudo point cloud data to form a scene model by acquiring and deeply labeling the multi-angle home images shot by the user. And distinguishing the soft package types which exist and do not exist in the image through visual identification, dividing the image area according to the soft package types, and inputting a soft package recommendation model to obtain a recommended soft package combination by combining the user data and the divided image area. And selecting a corresponding soft package component from the soft package component library for display, and allowing a user to click for secondary rendering and display of the scene model. According to the method, a personalized home decoration scheme is provided for a user through deep learning and a 3D rendering technology, a proper soft package collocation is recommended through identifying a home environment and user preferences, and the user is allowed to preview and adjust the decoration scheme through an interactive interface so as to achieve the best visual effect and satisfaction degree, and the conversion rate of soft package products is improved.

Description

Household soft management method, device, equipment and storage medium

Technical Field

The invention relates to the field of artificial intelligence, in particular to a household soft management method, a device, equipment and a storage medium.

Background

At present, the mode of selecting the home soft package is mainly selected through off-line physical shops, home decoration magazines, on-line home decoration platforms and the like. The user can obtain inspiration and select soft goods by visiting a physical store or browsing a magazine or a website in the field. However, conventional approaches often fail to provide personalized soft-pack recommendations for specific home environments and user preferences, resulting in a selected soft-pack product that is not necessarily the soft-pack product desired by the user, resulting in a lower conversion rate of the soft-pack product.

Disclosure of Invention

The invention mainly aims to solve the technical problem of low conversion rate of soft package selection of the existing user.

The first aspect of the invention provides a household soft management method, which comprises the following steps:

Acquiring a plurality of home images and target user data of multiple angles under a target home environment shot by user equipment, and performing depth annotation on the home images to obtain depth information of each home image;

Generating pseudo point cloud data corresponding to each home image based on the depth information, and coupling the pseudo point cloud data based on shooting angles corresponding to the plurality of home images to obtain total pseudo point cloud data;

Performing primary model rendering according to the total pseudo point cloud data and the plurality of home images to obtain a scene model in the target home environment;

Performing visual identification on the plurality of home images, and determining a first soft package class and a second soft package class in the plurality of home images, wherein the first soft package class is a soft package class existing in the plurality of home images, and the second soft package class is a soft package class not existing in the plurality of home images;

Performing image division on the plurality of home images based on the first soft package class and the second soft package class to obtain a first image area, a second image area and a third image area of the plurality of home images;

inputting the target user data, a first image area and a third image area of the plurality of home images into a soft package recommendation model corresponding to the first soft package class to obtain a first recommendation soft package combination corresponding to the first soft package class;

inputting the target user data, a second image area and a third image area of the plurality of home images into a soft package recommendation model corresponding to the second soft package class to obtain a second recommendation soft package combination corresponding to the second soft package class;

Acquiring a plurality of first soft package components and a plurality of second soft package components corresponding to the first recommended soft package combination and the second recommended soft package combination from a preset soft package component library, and displaying the first soft package components and the second soft package components on a user interface of the user equipment;

And responding to the clicking operation of any one of the first soft package components and the second soft package components, determining a corresponding target soft package component, performing secondary model rendering on the scene model according to the target soft package component, and displaying the scene model after the secondary model rendering to the user equipment.

Optionally, in a first implementation manner of the first aspect of the present invention, the obtaining multiple home images and target user data of multiple angles under a target home environment shot by a user device, and performing depth labeling on the multiple home images, obtaining depth information of each home image includes:

Acquiring a plurality of home images and target user data of multiple angles under a target home environment shot by user equipment, and inputting the home images into a preset depth estimation model, wherein the depth estimation model comprises an encoder, a convolution layer and a decoder;

Extracting first feature images of a plurality of input home images through the encoder, and carrying out convolution processing on the first feature images through the convolution layer to obtain image features of the first feature images;

Compressing the image features through the convolution layer to obtain feature vectors, carrying out convolution processing on the feature vectors and processing through a preset activation function to obtain weight vectors;

performing pixel-by-pixel dot product on the image features and the weight vectors to obtain weighted features, and integrating the weighted features and the image features to obtain a second feature map of the plurality of home images;

and carrying out depth estimation on the second feature map through the decoder to obtain the depth information of each home image.

Optionally, in a second implementation manner of the first aspect of the present invention, the visually identifying the plurality of home images, and determining the first soft package class and the second soft package class in the plurality of home images includes:

Performing image preprocessing on the plurality of home images to obtain preprocessed images, and inputting the preprocessed images into a preset image detection model, wherein the image detection model comprises an input layer, a feature extraction layer, a feature pyramid network, a feature fusion layer, a multitasking detection layer and an output layer;

Mapping the preprocessed image to a high-dimensional space through the input layer to obtain initial characteristic information in the high-dimensional space;

Performing feature extraction on the initial feature information through the feature extraction layer to obtain convolution features of different layers;

Generating initial fusion features corresponding to convolution features of different levels through the feature pyramid network, and carrying out feature fusion on the initial fusion features of different levels through the feature fusion layer to obtain secondary fusion features;

The multitasking detection layer and the output layer are used for detecting the first soft package class in the multiple home images according to the secondary fusion characteristics;

And calculating a second soft package class in the plurality of home images based on the first soft package class and a preset total soft package class.

Optionally, in a third implementation manner of the first aspect of the present invention, the performing image division on the plurality of home images based on the first soft package class and the second soft package class to obtain a first image area, a second image area, and a third image area of the plurality of home images includes:

performing primary image division on the plurality of home images based on the first soft package class to obtain a first image area and a to-be-secondarily divided area;

Inputting the secondary division area to be subjected to preset generation countermeasure network, generating a generation image containing the second soft package class, and determining the position to be placed of the second soft package class based on the generation image;

and carrying out secondary image division on the to-be-divided area based on the to-be-placed position to obtain a second image area and a third image area.

Optionally, in a fourth implementation manner of the first aspect of the present invention, inputting the to-be-secondarily divided area into a preset generating countermeasure network, generating a generated image including the second soft package class, and determining a to-be-placed position of the second soft package class based on the generated image includes:

Inputting the region to be secondarily divided into a preset generation countermeasure network to generate a generated image containing the second soft goods;

determining the initial position of the second soft package class in the corresponding home image based on the generated image;

Performing position adjustment on the initial position to generate a group of candidate positions as a population, wherein each body in the population corresponds to one candidate position;

Calculating the fitness value of each individual in the population according to the spatial layout data of the scene model and a preset fitness function, and screening parent individuals from the population according to the fitness value;

Respectively performing cross operation and mutation operation on the parent individuals to obtain new individuals of the population, and returning to the step of calculating the fitness value of each individual in the population according to the spatial layout data of the scene model and a preset fitness function until a preset stopping condition is met;

And taking the candidate position corresponding to the individual with the highest fitness value as the position to be placed of the second soft package class.

Optionally, in a fifth implementation manner of the first aspect of the present invention, the performing position adjustment on the initial position generates a set of candidate positions as a population, where each body in the population corresponds to one candidate position, and the method includes:

Performing position adjustment on the initial position to obtain a plurality of candidate positions of the second soft package class;

Performing physical constraint simulation on the second soft package class in the scene model according to the plurality of candidate positions to obtain a constraint result;

Screening the candidate positions according to the constraint result, and taking the screened candidate positions as a group of candidate positions;

And taking the group of candidate positions as a population, wherein each body in the population corresponds to one candidate position.

Optionally, in a sixth implementation manner of the first aspect of the present invention, the inputting the target user data, the first image area and the third image area of the plurality of home images into the soft package recommendation model corresponding to the first soft package class, and obtaining the first recommended soft package combination corresponding to the first soft package class includes:

inputting the target user data, a first image area and a third image area of the plurality of home images into a soft package recommendation model corresponding to the first soft package product, wherein the soft package recommendation model comprises an input layer, an attention mechanism layer, a feature fusion layer, a multi-task detection layer and an output layer;

performing data preprocessing and data feature extraction on the target user data through the input layer to obtain data features, and performing image feature extraction on a first image area and a third image area of the plurality of home images to obtain a first image feature and a second image feature;

Calculating attention weight values of the data feature, the first image feature and the second image feature through the attention mechanism layer respectively;

the feature fusion layer carries out weighted fusion on the corresponding data feature, the first image feature and the second image feature according to the attention weight value to obtain a fusion vector;

Performing multi-task detection according to the fusion vector through the multi-task detection layer to obtain various soft package recommendation information;

and selecting a plurality of first recommended soft packages from a candidate soft package library corresponding to the first soft package type according to the plurality of soft package recommendation information through the output layer, and forming a first recommended soft package combination according to the plurality of first recommended soft packages.

A second aspect of the present invention provides a home soft management device, comprising:

The acquisition module is used for acquiring a plurality of home images and target user data of multiple angles under a target home environment shot by the user equipment, and carrying out depth annotation on the plurality of home images to obtain depth information of each home image;

The pseudo point cloud generation module is used for generating pseudo point cloud data corresponding to each household image based on the depth information, and coupling the pseudo point cloud data based on shooting angles corresponding to the household images to obtain total pseudo point cloud data;

the model generation module is used for performing primary model rendering according to the total pseudo point cloud data and the plurality of home images to obtain a scene model in the target home environment;

The visual identification module is used for carrying out visual identification on the plurality of home images and determining a first soft package class and a second soft package class in the plurality of home images, wherein the first soft package class is a soft package class existing in the plurality of home images, and the second soft package class is a soft package class not existing in the plurality of home images;

The image dividing module is used for dividing the images of the plurality of home images based on the first soft package class and the second soft package class to obtain a first image area, a second image area and a third image area of the plurality of home images;

The first soft package recommending module is used for inputting the target user data, the first image area and the third image area of the plurality of home images into a soft package recommending model corresponding to the first soft package class to obtain a first recommended soft package combination corresponding to the first soft package class;

The second soft package recommending module is used for inputting the target user data, the second image area and the third image area of the plurality of home images into a soft package recommending model corresponding to the second soft package class to obtain a second recommended soft package combination corresponding to the second soft package class;

the component display module is used for acquiring a plurality of first soft package components and a plurality of second soft package components corresponding to the first recommended soft package combination and the second recommended soft package combination from a preset soft package component library and displaying the first soft package components and the second soft package components on a user interface of the user equipment;

and the model display module is used for responding to the clicking operation of any one of the first soft package components and the second soft package components, determining a corresponding target soft package component, performing secondary model rendering on the scene model according to the target soft package component, and displaying the scene model after the secondary model rendering to the user equipment.

A third aspect of the present invention provides a home soft management device, comprising: a memory and at least one processor, the memory having instructions stored therein, the memory and the at least one processor being interconnected by a line; the at least one processor invokes the instructions in the memory to cause the home soft management device to perform the steps of the home soft management method described above.

A fourth aspect of the present invention provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the steps of the home soft management method described above.

According to the home soft management method, device, equipment and storage medium, the pseudo point cloud data are generated and coupled to form the scene model through obtaining and deeply labeling the multi-angle home images shot by the user. And distinguishing the soft package types which exist and do not exist in the image through visual identification, dividing the image area according to the soft package types, and inputting a soft package recommendation model to obtain a recommended soft package combination by combining the user data and the divided image area. And selecting a corresponding soft package component from the soft package component library for display, and allowing a user to click for secondary rendering and display of the scene model. According to the method, a personalized home decoration scheme is provided for a user through deep learning and a 3D rendering technology, a proper soft package collocation is recommended through identifying a home environment and user preferences, and the user is allowed to preview and adjust the decoration scheme through an interactive interface so as to achieve the best visual effect and satisfaction degree, and the conversion rate of soft package products is improved.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

FIG. 1 is a schematic view of a first embodiment of a home soft management method according to an embodiment of the present invention;

FIG. 2 is a schematic view of an embodiment of a home soft management device according to an embodiment of the present invention;

Fig. 3 is a schematic view of an embodiment of a home soft management device according to the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terms "comprising" and "having" and any variations thereof, as used in the embodiments of the present invention, are intended to cover non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed or inherent to such process, method, article, or apparatus but may optionally include other steps or elements not listed or inherent to such process, method, article, or apparatus.

For the sake of understanding the present embodiment, a detailed description is first given of a home soft management method disclosed in the present embodiment. As shown in fig. 1, the method comprises the following steps:

101. Acquiring multiple home images and target user data of multiple angles under a target home environment shot by user equipment, and performing depth annotation on the multiple home images to obtain depth information of each home image;

In one embodiment of the present invention, the obtaining multiple home images and target user data of multiple angles in a target home environment captured by a user device, and performing depth labeling on the multiple home images, where obtaining depth information of each home image includes: acquiring a plurality of home images and target user data of multiple angles under a target home environment shot by user equipment, and inputting the home images into a preset depth estimation model, wherein the depth estimation model comprises an encoder, a convolution layer and a decoder; extracting first feature images of a plurality of input home images through the encoder, and carrying out convolution processing on the first feature images through the convolution layer to obtain image features of the first feature images; compressing the image features through the convolution layer to obtain feature vectors, carrying out convolution processing on the feature vectors and processing through a preset activation function to obtain weight vectors; performing pixel-by-pixel dot product on the image features and the weight vectors to obtain weighted features, and integrating the weighted features and the image features to obtain a second feature map of the plurality of home images; and carrying out depth estimation on the second feature map through the decoder to obtain the depth information of each home image.

Specifically, the user equipment may acquire multiple home images at multiple angles in the target home environment by carrying multiple cameras or by taking multiple photos at different angles by the user. Therefore, more comprehensive household environment information can be obtained from different perspectives, and more accurate data support is provided for subsequent depth labeling and depth estimation. In the depth estimation model described, the encoder plays a key role. The encoder firstly performs feature extraction on a plurality of input home images, and gradually extracts feature information of the images by performing convolution processing on each image, and the information is assembled into a first feature image. This process is similar to compressing and abstracting image information to better express the content characteristics of the image. The convolution layer performs sliding calculation on the input image through a convolution kernel (filter), so as to extract characteristic information in the image. The convolution operation multiplies the input image and the convolution kernel point by point, and adds the obtained product values to form an output feature map. The output feature map is the image feature obtained after convolution processing. In the convolution operation process, the type and the number of the extracted features can be controlled by adjusting the size and the number of the convolution kernels, so that abstract expression of the image features is realized. Then, after the image features are obtained, the image features are typically compressed by pooling (pooling) or the like to reduce the dimensionality and complexity of the data while retaining the most important information. This may make the subsequent processing more efficient while reducing the risk of overfitting. When compressed image features are obtained, weight vectors need to be calculated next. This is typically achieved by convolving the feature vectors and applying a preset activation function. In the process, important information in the image characteristics can be better captured through learning and weight adjustment, so that a more accurate basis is provided for subsequent depth estimation. Finally, pixel-by-pixel dot products are performed on the image features and the weight vectors, namely, the values of the corresponding positions of the image features and the weight vectors are multiplied, and the results are added, so that the weight features are obtained. The operation can effectively combine the image characteristics and the weight information, so that each characteristic can be correctly weighted, important characteristics of the image can be better reflected, and more accurate data support is provided for subsequent depth estimation. In deep learning models, the weighted features and image features are typically integrated using an attention mechanism (attention mechanism) to obtain a more comprehensive feature representation. The attention mechanism may help the network focus on important features, thereby improving the performance of the model. By performing a weighted addition or stitching operation on the weighted features and the image features, a richer, more representative feature representation, i.e. the second feature map, can be obtained. The decoder, which is a function of restoring and reconstructing the input data in the deep learning model, generally corresponds to the encoder for mapping the extracted features back into the original data space. When the depth estimation is performed on the second feature map, the decoder may reverse the process of operating the encoder, and restore the feature map to depth information through deconvolution or the like.

102. Generating pseudo point cloud data corresponding to each home image based on the depth information, and coupling the pseudo point cloud data based on shooting angles corresponding to a plurality of home images to obtain total pseudo point cloud data;

In one embodiment of the invention, for each pixel point in the depth image, it may be converted into three-dimensional coordinates in the camera coordinate system based on the internal reference matrix and the depth values. And converting the three-dimensional coordinates in the camera coordinate system into three-dimensional coordinates in the world coordinate system, so that the point cloud data in the real world corresponding to the depth information is obtained. And then, coupling the pseudo point cloud data according to shooting angles corresponding to the plurality of home images, and for each home image, firstly, rotating and translating the corresponding pseudo point cloud data according to the shooting angles, and converting the pseudo point cloud data into a common coordinate system. The pseudo point cloud data from different angle shots are registered under a common coordinate system, i.e. they are aligned under the same coordinate system to ensure that their positions and directions in space are consistent. And the registered pseudo point cloud data are fused to obtain total pseudo point cloud data, wherein the total pseudo point cloud data comprise comprehensive information of a plurality of home images.

Specifically, in the process of coupling the pseudo point clouds according to the shooting angles, the coordinate transformation and registration of the pseudo point cloud data of different angles are mainly involved. The spatial relationship among different shooting angles needs to be considered, so that the coupled total pseudo point cloud data can completely and accurately reflect the overall structure and characteristics of the household environment. The pseudo point cloud data under each view angle can be converted into the same coordinate system by calculating a rotation matrix and a translation vector between different view angles. And carrying out fusion processing on the registered pseudo point cloud data, and combining information of a plurality of view angles into total pseudo point cloud data by adopting weighted average or other fusion strategies. : and finally, verifying whether the coupled total pseudo point cloud data completely and accurately reflects the structure and characteristics of the whole home environment or not, and ensuring that the generated data can be effectively applied to subsequent tasks and analysis.

103. Performing primary model rendering according to the total pseudo point cloud data and the plurality of home images to obtain a scene model in the target home environment;

In one embodiment of the present invention, when rendering is performed, first, the total pseudo point cloud data is preprocessed, including operations of removing noise, filling up missing values, and the like, so as to ensure quality and integrity of the pseudo point cloud data. Meanwhile, camera parameters such as an internal reference matrix, distortion coefficients and the like of a plurality of home images are required to be acquired. Based on the total pseudo point cloud data, a scene model in the target home environment can be created. This typically involves converting the pseudo-point cloud data into a visualized three-dimensional model using three-dimensional modeling software or a graphics library. In creating a scene model, it is necessary to map pseudo-point cloud data to appropriate locations and add a realistic look to the model according to the attributes of the point cloud (e.g., color, texture, etc.). In order to increase the realism of the scene model, texture information in the plurality of home images may be mapped onto the scene model. Specifically, by projecting the home image onto the surface of the scene model, the texture information of the image is combined with the geometry of the model, thereby making the appearance of the model more realistic and fine. In the rendering process, the influence of illumination on the scene model needs to be considered. By setting proper light source attributes and environment illumination parameters, rendering results can be more real. This includes determining the location, intensity, color, etc. of the light source and adjusting the material properties to reflect and scatter light to produce the appropriate shadow effect. And finally, rendering the scene model according to the set rendering parameters, and outputting a rendering result as an image or video. In the rendering process, a scene model is rendered from a corresponding view angle according to camera parameters and view angle settings, and a virtual scene image similar to an actual home environment is obtained.

104. Performing visual identification on the plurality of home images, and determining a first soft package class and a second soft package class in the plurality of home images;

In one embodiment of the present invention, the first soft package class is a soft package class existing in the plurality of home images, and the second soft package class is a soft package class not existing in the plurality of home images; the visual identification of the plurality of home images is carried out, and the determination of the first soft package class and the second soft package class in the plurality of home images comprises the following steps: performing image preprocessing on the plurality of home images to obtain preprocessed images, and inputting the preprocessed images into a preset image detection model, wherein the image detection model comprises an input layer, a feature extraction layer, a feature pyramid network, a feature fusion layer, a multitasking detection layer and an output layer; mapping the preprocessed image to a high-dimensional space through the input layer to obtain initial characteristic information in the high-dimensional space; performing feature extraction on the initial feature information through the feature extraction layer to obtain convolution features of different layers; generating initial fusion features corresponding to convolution features of different levels through the feature pyramid network, and carrying out feature fusion on the initial fusion features of different levels through the feature fusion layer to obtain secondary fusion features; the multitasking detection layer and the output layer are used for detecting the first soft package class in the multiple home images according to the secondary fusion characteristics; and calculating a second soft package class in the plurality of home images based on the first soft package class and a preset total soft package class.

Specifically, in the input layer, first, the input layer receives the image data after preprocessing, which is generally an image subjected to normalization, resizing, and the like. The input layer maps the entire image into a high-dimensional vector by taking the value of each pixel point as one dimension of the feature vector. In this way, the information of color, brightness, etc. of each pixel is encoded into one component of the vector, thereby forming a representation of the image in a high-dimensional space. By this mapping, the structure and features of the image in the original two-dimensional space are transformed into a high-dimensional space, each dimension of which represents a different feature in the image. Therefore, the representation of the image in the high-dimensional space contains more abundant and abstract feature information, which facilitates subsequent feature extraction and pattern recognition. In high-dimensional space, this initial feature information may be passed to subsequent layers, such as a feature extraction layer, for further extraction and analysis of features in the image, thereby enabling identification and classification of the soft goods class. In the feature extraction layer, it uses convolution operations to process the initial feature information. The convolution operation adds up each convolution kernel by multiplying it with a portion of the input feature map by sliding the convolution kernels (filters) over the image, resulting in an output feature map. This process may capture local features such as edges, textures, etc. After the convolution operation, the output is typically non-linearly transformed using an activation function (e.g., reLU) to introduce non-linear characteristics and enhance the expressive power of the network. In some cases, the feature extraction layer may include pooling operations, such as maximum pooling or average pooling, to reduce the size of the feature map and preserve important information. The feature extraction layer is typically composed of multiple convolution layers and activation functions, each of which can extract features at different levels of abstraction. Shallow convolution layers focus more on low-level features (e.g., edges, colors), while deep convolution layers gradually shift to higher-level semantic features (e.g., object shape, texture). After the feature extraction layer processing, convolution feature diagrams of different layers are obtained, and each feature diagram corresponds to a specific abstract feature representation. These feature maps will be passed to the next hierarchy, such as a feature pyramid network or feature fusion layer, for further information extraction and processing. The feature pyramid network enables the network to sense the size and position information of objects on different levels by constructing a multi-scale feature map. The feature map of each level contains different degrees of semantic information. In the feature pyramid network, convolution features of different levels are fused together to form a primary fusion feature. The fusion features retain multi-scale information, and are beneficial to improving the detection performance of the model on objects with different scales. The feature fusion layer receives primary fusion features from different layers, and obtains more comprehensive and global feature representation, namely secondary fusion features, by fusing and integrating the feature information. The obtained secondary fusion features contain fusion of information from different layers and different scales through the processing of the feature fusion layer, have stronger semantic expression capability and global information, and are beneficial to subsequent detection tasks. The multi-task detection layer can simultaneously perform a plurality of target detection tasks, including detection of different soft goods in the home image, positioning and the like. The layer can perform target detection and classification based on the input secondary fusion characteristics, and various objects in the image are identified. The output layer receives the information from the multi-task detection layer, and outputs a final prediction result including the identification result of the first soft package class according to the secondary fusion characteristic and the detection result.

105. Performing image division on a plurality of home images based on the first soft package class and the second soft package class to obtain a first image area, a second image area and a third image area of the plurality of home images;

In one embodiment of the present invention, the image dividing the plurality of home images based on the first soft package class and the second soft package class, to obtain a first image area, a second image area, and a third image area of the plurality of home images includes: performing primary image division on the plurality of home images based on the first soft package class to obtain a first image area and a to-be-secondarily divided area; inputting the secondary division area to be subjected to preset generation countermeasure network, generating a generation image containing the second soft package class, and determining the position to be placed of the second soft package class based on the generation image; and carrying out secondary image division on the to-be-divided area based on the to-be-placed position to obtain a second image area and a third image area.

Specifically, first, a plurality of home images are subjected to primary image division based on a first soft package class, and a first image area and an area to be secondarily divided are obtained. The first image area may contain objects of a first soft goods class, while the area to be subdivided requires further processing to determine the location of the second soft goods class. After the secondary divided area is input into a preset generation countermeasure network, the generation countermeasure network outputs a generated image containing the second soft package class. And determining the position to be placed of the second soft package according to the generated image, and then secondarily dividing the area to be secondarily divided based on the position to obtain a second image area and a third image area. Wherein the generation countermeasure network is a framework composed of a generator and a discriminator, wherein the generator is responsible for generating a realistic image, and the discriminator is responsible for distinguishing the real image from the generated image. The area to be sub-divided is fed as input into a generator section that generates the countermeasure network. The generator tries to generate an image containing the second soft goods class by learning the distribution of the real image data and outputs it. The generated image contains the second soft goods, and the position information of the second soft goods in the image can be determined by analyzing or processing the generated image.

Further, inputting the to-be-secondarily divided area into a preset generation countermeasure network, generating a generated image including the second soft package class, and determining the to-be-placed position of the second soft package class based on the generated image includes: inputting the region to be secondarily divided into a preset generation countermeasure network to generate a generated image containing the second soft goods; determining the initial position of the second soft package class in the corresponding home image based on the generated image; performing position adjustment on the initial position to generate a group of candidate positions as a population, wherein each body in the population corresponds to one candidate position; calculating the fitness value of each individual in the population according to the spatial layout data of the scene model and a preset fitness function, and screening parent individuals from the population according to the fitness value; respectively performing cross operation and mutation operation on the parent individuals to obtain new individuals of the population, and returning to the step of calculating the fitness value of each individual in the population according to the spatial layout data of the scene model and a preset fitness function until a preset stopping condition is met; and taking the candidate position corresponding to the individual with the highest fitness value as the position to be placed of the second soft package class.

Specifically, the purpose of adjusting the initial position is to optimize the placement position of the second soft package product in the home image, so as to meet the design or layout requirements. Through the adjustment position, the placing effect and the overall visual perception of soft goods can be improved. The set of candidate positions is generated in order to find the best placement position in the spatial layout. Thus, the most suitable position can be selected from a plurality of possible positions so as to improve the placing effect and the space aesthetic feeling of soft goods.

Specifically, the spatial layout data of the scene model describes information such as layout, furniture placement rules, spatial limitation and the like of the home scene. The fitness function is used to evaluate the fitness of each individual (i.e., candidate location) in a given scene. The fitness function may be defined in terms of design requirements and space layout data, and typically includes factors related to design goals, such as aesthetics, space utilization efficiency, and the like. In genetic algorithms, crossover refers to the process of producing offspring individuals from two parent individuals. Certain characteristics of the parent individuals are typically selected for combination to generate new individuals. The mutation operation is to increase the diversity of the population, and is helpful to avoid sinking into the locally optimal solution. In genetic algorithms, new individuals are introduced by making small random changes to certain characteristics of the individual. By continuously performing crossover operation, mutation operation and fitness evaluation, new individuals are generated and individuals with high fitness are screened until preset stopping conditions (such as maximum iteration times or satisfactory solutions) are met. And finally, selecting a candidate position corresponding to the individual with the highest fitness value as a final position to be placed of the second soft package class.

Further, the position adjustment is performed on the initial position, a group of candidate positions is generated as a population, and each body in the population corresponds to one candidate position, and the method includes: performing position adjustment on the initial position to obtain a plurality of candidate positions of the second soft package class; performing physical constraint simulation on the second soft package class in the scene model according to the plurality of candidate positions to obtain a constraint result; screening the candidate positions according to the constraint result, and taking the screened candidate positions as a group of candidate positions; and taking the group of candidate positions as a population, wherein each body in the population corresponds to one candidate position.

Specifically, performing position adjustment on the initial position to obtain a plurality of candidate positions of the second soft package class; and performing physical constraint simulation on the second soft package class in the scene model according to the candidate positions to obtain a constraint result. In the physical constraint simulation, the actual physical characteristics and limitations of the home scene, such as gravity, collision detection, space occupation and the like, are considered, so that the soft goods meet the actual feasibility in the layout process. And screening the candidate positions according to the constraint result, and taking the screened candidate positions as a group of candidate positions. And (3) removing candidate positions which do not accord with physical constraint through a constraint simulation result, so as to obtain reasonable candidate positions after physical constraint screening. And finally, taking the group of candidate positions as a population, wherein each body in the population corresponds to one candidate position. Thus, based on the screening result of the physical constraint simulation, a candidate position population verified by the physical constraint is formed, and a more reasonable and feasible initial solution space is provided for the operation of a subsequent genetic algorithm.

106. Inputting target user data, a first image area and a third image area of a plurality of home images into a soft package recommendation model corresponding to a first soft package class to obtain a first recommended soft package combination corresponding to the first soft package class;

In one embodiment of the present invention, the inputting the target user data, the first image area and the third image area of the plurality of home images into the soft package recommendation model corresponding to the first soft package category, and obtaining the first recommended soft package combination corresponding to the first soft package category includes: inputting the target user data, a first image area and a third image area of the plurality of home images into a soft package recommendation model corresponding to the first soft package product, wherein the soft package recommendation model comprises an input layer, an attention mechanism layer, a feature fusion layer, a multi-task detection layer and an output layer; performing data preprocessing and data feature extraction on the target user data through the input layer to obtain data features, and performing image feature extraction on a first image area and a third image area of the plurality of home images to obtain a first image feature and a second image feature; calculating attention weight values of the data feature, the first image feature and the second image feature through the attention mechanism layer respectively; the feature fusion layer carries out weighted fusion on the corresponding data feature, the first image feature and the second image feature according to the attention weight value to obtain a fusion vector; performing multi-task detection according to the fusion vector through the multi-task detection layer to obtain various soft package recommendation information; and selecting a plurality of first recommended soft packages from a candidate soft package library corresponding to the first soft package type according to the plurality of soft package recommendation information through the output layer, and forming a first recommended soft package combination according to the plurality of first recommended soft packages.

Specifically, in the input layer, data preprocessing and data feature extraction are performed on target user data to obtain data features. And simultaneously, extracting image features of a first image area and a third image area of the plurality of home images to obtain a first image feature and a second image feature. In the attention mechanism layer, attention weight values for the data feature, the first image feature, and the second image feature are calculated. By learning the attention weight value, the model can automatically focus on feature information related to the soft-pack recommendation. And in the feature fusion layer, corresponding data features, the first image features and the second image features are subjected to weighted fusion according to the attention weight value, so that fusion vectors are obtained. Thus, the useful information of different characteristics can be combined to improve the accuracy and effect of soft package recommendation. In the multi-task detection layer, multi-task detection is carried out based on the fusion vector, and multiple soft package recommendation information is obtained. The multi-task detection can process different soft package recommendation tasks, such as color recommendation, style recommendation and the like, simultaneously so as to meet the diversified requirements of users. And selecting a plurality of first recommended soft packages from the candidate soft package libraries corresponding to the first soft package types according to the plurality of soft package recommendation information in the output layer. And forming a first recommended soft package combination by combining a plurality of first recommended soft packages, and providing a diversified and personalized soft package recommendation scheme for the user.

107. Inputting target user data, a second image area and a third image area of the multiple home images into a soft package recommendation model corresponding to a second soft package class to obtain a second recommendation soft package combination corresponding to the second soft package class;

In an embodiment of the present invention, the soft package recommendation model corresponding to the second soft package category uses the same model structure, which is not described herein again.

108. Acquiring a plurality of first soft package components and a plurality of second soft package components corresponding to the first recommended soft package combination and the second recommended soft package combination from a preset soft package component library, and displaying the first soft package components and the second soft package components on a user interface of user equipment;

In one embodiment of the present invention, a plurality of first soft-package components and a plurality of second soft-package components corresponding to the first recommended soft-package combination and the second recommended soft-package combination are obtained from a preset soft-package component library, and displayed on a user interface of the user device. After the soft-good recommendation model generates the first recommended soft-good combination and the second recommended soft-good combination, the system will match in a pre-prepared library of soft-good components based on these combinations. The library contains various soft-packaged components such as sofas, tables, lamps, etc. The system will select the soft-packaged components that match the recommended combination based on the recommended results. The system then extracts the first plurality of soft-packaged components and the second plurality of soft-packaged components from the library and displays them on a user interface of the user device. The user can then view and select different soft-packaged components through the interface for further customization and adjustment according to personal preferences and needs. By displaying the recommended soft-packaged components on the user interface, the user can intuitively see and compare the appearance, style, and applicability of the different components. The method provides a convenient and quick way for users to select favorite soft package components, and matches and combines the favorite soft package components according to own requirements so as to create ideal home decoration effects.

109. And responding to the clicking operation of any one of the first soft package components and the second soft package components, determining a corresponding target soft package component, performing secondary model rendering on the scene model according to the target soft package component, and displaying the scene model after the secondary model rendering to the user equipment.

In one embodiment of the present invention, when a user performs a clicking operation for any of the plurality of first and second soft-packaged components, the system determines a corresponding target soft-packaged component according to the soft-packaged component selected by the user. In this way, a user may specifically select a particular one of the soft-packaged components for viewing and adjustment. Once the target soft-packaged component is determined, the system performs a secondary model rendering on the scene model. This means that the system will render the original scene model again based on the selection and parameter settings of the target soft-packaged components to reflect the user's selection and changes to the soft-packaged components. In this way, the user can see the effect and adaptability of the selected soft-packaged component in the scene in real time. After the secondary model rendering, the system displays the updated scene model in a user interface on the user device. The user can observe and evaluate the new scene model through the interface so as to better understand the collocation effect of the selected soft-packaged components and the whole scene. This allows the user to intuitively see the impact of the appearance, size, color, etc. characteristics of the soft-packaged component on the overall scene, and make fine adjustments and modifications as needed. By displaying the secondary model rendered scene model on the user device, the user can get more accurate, real-time visual feedback, thereby helping them make more intelligent soft-packaged component selections and decisions. The interaction mode provides an intuitive and flexible mode, so that a user can better participate in the soft-package collocation process to meet personalized requirements and aesthetic requirements.

In this embodiment, the pseudo point cloud data is generated and coupled to form the scene model by acquiring and deeply labeling the multi-angle home image shot by the user. And distinguishing the soft package types which exist and do not exist in the image through visual identification, dividing the image area according to the soft package types, and inputting a soft package recommendation model to obtain a recommended soft package combination by combining the user data and the divided image area. And selecting a corresponding soft package component from the soft package component library for display, and allowing a user to click for secondary rendering and display of the scene model. According to the method, a personalized home decoration scheme is provided for a user through deep learning and a 3D rendering technology, a proper soft package collocation is recommended through identifying a home environment and user preferences, and the user is allowed to preview and adjust the decoration scheme through an interactive interface so as to achieve the best visual effect and satisfaction degree, and the conversion rate of soft package products is improved.

The method for managing household soft-management in the embodiment of the present invention is described above, and the following describes a household soft-management device in the embodiment of the present invention, referring to fig. 2, and one embodiment of the household soft-management device in the embodiment of the present invention includes:

the acquiring module 201 is configured to acquire multiple home images and target user data of multiple angles in a target home environment shot by a user device, and perform depth labeling on the multiple home images to obtain depth information of each home image;

The pseudo point cloud generating module 202 is configured to generate pseudo point cloud data corresponding to each home image based on the depth information, and couple the pseudo point cloud data based on shooting angles corresponding to the plurality of home images to obtain total pseudo point cloud data;

the model generation module 203 is configured to perform model rendering for one time according to the total pseudo point cloud data and the plurality of home images, so as to obtain a scene model in the target home environment;

the visual identification module 204 is configured to perform visual identification on the plurality of home images, and determine a first soft package class and a second soft package class in the plurality of home images, where the first soft package class is a soft package class existing in the plurality of home images, and the second soft package class is a soft package class not existing in the plurality of home images;

The image dividing module 205 is configured to perform image division on the plurality of home images based on the first soft package class and the second soft package class, so as to obtain a first image area, a second image area, and a third image area of the plurality of home images;

a first soft package recommendation module 206, configured to input the target user data, the first image area and the third image area of the plurality of home images into a soft package recommendation model corresponding to the first soft package category, to obtain a first recommended soft package combination corresponding to the first soft package category;

a second soft package recommendation module 207, configured to input the target user data, the second image area and the third image area of the plurality of home images into a soft package recommendation model corresponding to the second soft package category, to obtain a second recommended soft package combination corresponding to the second soft package category;

the component display module 208 is configured to obtain, from a preset soft package component library, a plurality of first soft package components and a plurality of second soft package components corresponding to the first recommended soft package combination and the second recommended soft package combination, and display the first soft package components and the second soft package components on a user interface of the user equipment;

the model display module 209 is configured to determine a corresponding target soft package component in response to a clicking operation for any one of the plurality of first soft package components and the plurality of second soft package components, perform secondary model rendering on the scene model according to the target soft package component, and display the scene model after the secondary model rendering to the user equipment.

In the embodiment of the invention, the home soft management device runs the home soft management method, and generates and couples pseudo point cloud data to form a scene model by acquiring and deeply labeling multi-angle home images shot by a user. And distinguishing the soft package types which exist and do not exist in the image through visual identification, dividing the image area according to the soft package types, and inputting a soft package recommendation model to obtain a recommended soft package combination by combining the user data and the divided image area. And selecting a corresponding soft package component from the soft package component library for display, and allowing a user to click for secondary rendering and display of the scene model. According to the method, a personalized home decoration scheme is provided for a user through deep learning and a 3D rendering technology, a proper soft package collocation is recommended through identifying a home environment and user preferences, and the user is allowed to preview and adjust the decoration scheme through an interactive interface so as to achieve the best visual effect and satisfaction degree, and the conversion rate of soft package products is improved.

The above fig. 2 describes the middle-home soft management device in the embodiment of the present invention in detail from the point of view of modularized functional entities, and the following describes the middle-home soft management device in the embodiment of the present invention in detail from the point of view of hardware processing.

Fig. 3 is a schematic structural diagram of a home soft management device according to an embodiment of the present invention, where the home soft management device 300 may have a relatively large difference due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (e.g., one or more processors) and a memory 320, and one or more storage mediums 330 (e.g., one or more mass storage devices) storing application programs 333 or data 332. Wherein memory 320 and storage medium 330 may be transitory or persistent storage. The program stored on the storage medium 330 may include one or more modules (not shown), each of which may include a series of instruction operations for the home soft management device 300. Still further, the processor 310 may be configured to communicate with the storage medium 330 and execute a series of instruction operations in the storage medium 330 on the home soft management device 300 to implement the steps of the home soft management method described above.

The home soft management device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input/output interfaces 360, and/or one or more operating systems 331, such as Windows Serve, mac OS X, unix, linux, freeBSD, and the like. It will be appreciated by those skilled in the art that the structure of the home soft management device shown in fig. 3 does not constitute a limitation of the home soft management device provided by the present invention, and may include more or less components than those illustrated, or may combine certain components, or may be arranged in different components.

The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, or may be a volatile computer readable storage medium, where instructions are stored in the computer readable storage medium, which when executed on a computer, cause the computer to perform the steps of the home soft management method.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system or apparatus and unit described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. The household soft management method is characterized by comprising the following steps of:

2. The home soft management method according to claim 1, wherein the obtaining multiple home images and target user data of multiple angles under a target home environment shot by the user equipment, and performing depth labeling on the multiple home images, and obtaining depth information of each home image comprises:

3. The home soft management method of claim 1, wherein the visually identifying the plurality of home images, determining a first class of soft goods and a second class of soft goods in the plurality of home images comprises:

4. The method of claim 1, wherein the performing image division on the plurality of home images based on the first soft package class and the second soft package class to obtain a first image area, a second image area, and a third image area of the plurality of home images comprises:

5. The home soft management method of claim 4, wherein inputting the region to be secondarily divided into a preset generation countermeasure network, generating a generated image including the second soft goods class, and determining a position to be placed of the second soft goods class based on the generated image comprises:

6. The home soft management method of claim 5, wherein the performing the position adjustment on the initial position generates a set of candidate positions as a population, each body in the population corresponding to one candidate position, and the step of:

7. The method for managing household soft goods according to claim 6, wherein the inputting the target user data, the first image area and the third image area of the plurality of household images into the soft goods recommendation model corresponding to the first soft goods class, and obtaining the first recommended soft goods combination corresponding to the first soft goods class comprises:

8. A home soft management device, characterized in that it comprises:

9. A home soft management device, characterized in that it comprises: a memory and at least one processor, the memory having instructions stored therein;

The at least one processor invokes the instructions in the memory to cause the home soft management device to perform the steps of the home soft management method according to any of claims 1-7.

10. A computer readable storage medium having instructions stored thereon, which instructions, when executed by a processor, implement the steps of the home soft management method of any of claims 1-7.