CN117730309A

CN117730309A - Model determination method, layout generation method, device, medium and chip

Info

Publication number: CN117730309A
Application number: CN202280004486.9A
Authority: CN
Inventors: 刘坤
Original assignee: Beijing Xiaomi Mobile Software Co Ltd; Beijing Xiaomi Pinecone Electronic Co Ltd; Xiaomi Technology Wuhan Co Ltd
Current assignee: Beijing Xiaomi Mobile Software Co Ltd; Beijing Xiaomi Pinecone Electronic Co Ltd; Xiaomi Technology Wuhan Co Ltd
Priority date: 2022-05-31
Filing date: 2022-05-31
Publication date: 2024-03-19
Also published as: WO2023230907A1

Abstract

The method for determining the model, the method for generating the layout, the device, the medium and the chip comprise the following steps: acquiring a basic layout (11) for layout generation; inputting the basic layout into a generator to obtain candidate layouts corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates candidate layouts (12) based on the extracted features; determining a generation loss corresponding to a generator and a discrimination loss of a discriminator according to the candidate layout, the sample layout and the discriminator (13); a model formed by the generator and the arbiter is trained on the basis of the generation loss and the discrimination loss, and the generator in the trained model is determined as a layout generation model (14).

Description

Model determination method, layout generation method, device, medium and chip

Technical Field

The present disclosure relates to the field of image processing, and in particular, to a method for determining a model, a method for generating a layout, a device, a medium, and a chip.

Background

With the development of computer technology and networks, the planar design in the web page is used as an interactive object directly facing the user, and has very direct influence on the use experience of the user. When laying out a planar design in a page, such as a poster or a mobile phone promotion page, certain fixed elements, such as a front view, a text in the page, and the like, are usually required to be contained. In the related art, usually, a designer is required to manually perform related design, or mirror multiplexing can be performed based on the existing poster, in the scheme, the technical requirement on the designer is high, and a large amount of work is required to be put into the scheme, and when multiplexing is performed based on the existing poster, the generation of a new layout has a limitation, is highly dependent on a manual strategy, and has low efficiency.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides a method of determining a model, a layout generating method, an apparatus, a medium, and a chip.

According to a first aspect of an embodiment of the present disclosure, there is provided a method for determining a layout generation model, including:

acquiring a basic layout for layout generation;

inputting the basic layout into a generator to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

determining the corresponding generation loss of the generator and the discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

training models formed by the generator and the discriminant according to the generation loss and the discriminant loss, and determining a generator in the trained models as a layout generation model.

Optionally, the sample layout is determined by:

acquiring annotation layout information, wherein each annotation layout information comprises annotation category and position information of each element in the annotation layout;

And determining element categories corresponding to the annotation categories according to the annotation layout information, and determining the element categories corresponding to each element in the annotation layout and the layout corresponding to the position information as the sample layout.

Optionally, the determining the generation loss corresponding to the generator and the discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator includes:

determining judging labels respectively corresponding to the candidate layout and the sample layout according to the candidate layout, the sample layout and the discriminator;

determining the generation loss according to the judgment label corresponding to the candidate layout and the labeling label corresponding to the candidate layout;

and determining the discrimination loss according to the discrimination labels respectively corresponding to the candidate layout and the sample layout and the labeling labels respectively corresponding to the candidate layout and the sample layout.

Optionally, the determining, according to the candidate layout, the sample layout and the arbiter, a decision tag corresponding to the candidate layout and the sample layout respectively includes:

generating a candidate layout image according to the layout information corresponding to the candidate layout, and generating a sample layout image according to the layout information of the sample layout;

Inputting the candidate layout image into the discriminator, extracting features of the candidate layout image by the discriminator, and determining a judging label of the candidate layout according to the extracted features;

inputting the sample layout image into the discriminator, extracting features of the sample layout image by the discriminator, and determining a judging label of the sample layout according to the extracted features.

Optionally, the generating a candidate layout image according to the layout information corresponding to the candidate layout includes:

inputting the layout information corresponding to the candidate layout into an encoder of the discriminator to obtain coding features corresponding to the layout information corresponding to the candidate layout;

extracting the characteristics of the coding characteristics based on an attention mechanism to obtain layout space characteristics of layout information corresponding to the candidate layout;

and performing image rendering according to the layout space features to obtain the candidate layout images.

According to a second aspect of embodiments of the present disclosure, there is provided a layout generating method, the method including:

acquiring target element information in a layout to be generated;

inquiring a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model in the first aspect;

And determining a target layout corresponding to the target element information according to the template layout.

Optionally, the plurality of layouts in the layout library is determined by:

determining candidate element information for layout generation, wherein the candidate element information comprises element categories and the number of elements under each element category;

inputting the candidate element information into the layout generation model to obtain a plurality of generated layouts corresponding to the candidate element information;

and adjusting the generated layout according to a preset layout adjustment rule, and storing the adjusted layout and the candidate element information in a layout library in an associated manner.

According to a third aspect of the embodiments of the present disclosure, there is provided a determination apparatus of a layout generation model, including:

a first acquisition module configured to acquire a base layout for layout generation;

a generation module configured to input the basic layout into a generator to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

A first determining module configured to determine a generation loss corresponding to the generator and a discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

a training module configured to train a model formed by the generator and the discriminant according to the generation loss and the discriminant loss, and determine a generator of the trained model as a layout generation model.

According to a fourth aspect of embodiments of the present disclosure, there is provided a layout generating apparatus, the apparatus including:

a second acquisition module configured to acquire target element information in a layout to be generated;

the query module is configured to query a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model in any one of the first aspect;

and the second determining module is configured to determine a target layout corresponding to the target element information according to the template layout.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a determining apparatus of a layout generation model, including:

A processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring a basic layout for layout generation;

inputting the basic layout into a generator to obtain candidate layouts corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

According to a sixth aspect of the embodiments of the present disclosure, there is provided a layout generating apparatus, including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring target element information in a layout to be generated;

inquiring a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model in any one of the first aspect;

According to a seventh aspect of embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the method of any of the first or second aspects.

According to an eighth aspect of embodiments of the present disclosure, there is provided a chip comprising a processor and an interface; the processor is configured to read instructions to perform the method of any of the first or second aspects.

The technical scheme provided by the embodiment of the disclosure can comprise the following beneficial effects:

according to the technical scheme, the layout containing the diversity of the various elements can be automatically generated based on the layout generation model, and in the process of generating the layout, the corresponding candidate layout is generated by extracting the semantics and the spatial features of the basic layout, so that the spatial relation among the elements in the generated layout can be improved, the use requirement of planar design is attached, the accuracy of generating the layout is improved, the accuracy of the generator can be further improved through the countermeasure training between the generator and the discriminator, the effectiveness of the layout generated based on the layout generation model is ensured, the workload of a designer is greatly reduced, and the diversity and the efficiency of generating the layout are improved by providing reliable data support for the planar design for the designer.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow chart illustrating a method of determining a layout generation model according to an exemplary embodiment.

FIG. 2 is a flowchart illustrating an exemplary implementation of determining a generation penalty for a generator and a discrimination penalty for a discriminator based on candidate layouts, sample layouts, and discriminators, according to an exemplary embodiment.

FIG. 3 is a flowchart illustrating an exemplary implementation of determining decision tags for candidate layouts and sample layouts, respectively, based on the candidate layouts, sample layouts, and a arbiter, according to an example embodiment.

Fig. 4 is a flowchart illustrating a layout generation method according to an example embodiment.

Fig. 5 is a block diagram illustrating a determination apparatus of a layout generation model according to an exemplary embodiment.

Fig. 6 is a block diagram of a layout generation model determination device or layout generation device, according to an example embodiment.

Fig. 7 is a block diagram of a layout generation model determination device or layout generation device, according to an example embodiment.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.

It should be noted that, all actions for acquiring signals, information or data in the present application are performed under the condition of conforming to the corresponding data protection rule policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.

FIG. 1 is a flow chart illustrating a method of determining a layout generation model, as shown in FIG. 1, according to an exemplary embodiment, which may include:

in step 11, a base layout for layout generation is acquired.

Wherein, the number range of the elements contained in the layout can be preset based on the number of the elements in the common design in the practical application scene, for example, the number range of the elements in the layout can be preset to be [4,9 ]I.e. the number of elements in the basic layout may be 4-9. If the number of elements in one sample is 4, 4 element scores can be determined based on the initial probability parameters and geometric parametersThe element category and position information corresponding thereto can be expressed as { (p) ₁ ,θ ₁ ),(p ₂ ,θ ₂ ),(p ₃ ,θ ₃ ),(p ₄ ,θ ₄ ) }. Wherein the probability parameter p _i For representing the probability that the i-th element belongs to a certain class, the geometric parameter theta _i The coordinates of the target point for representing the i-th element may be, for example, two points corresponding to the upper left corner and the lower right corner, or two points corresponding to the lower left corner and the upper right corner, which is not limited by the present disclosure.

In the initial case, the probability parameters may be sampled from a uniform distribution, and the geometric parameters may be sampled from a gaussian distribution, so that element types and position information corresponding to each element in the layout may be determined to obtain the basic layout. The method comprises the steps of sampling for a plurality of times based on the number of elements, probability parameters and geometric parameters in a preset layout, and obtaining a plurality of basic layouts so as to facilitate subsequent training.

In step 12, a base layout is input to a generator, which performs feature extraction on the base layout based on a multi-headed attention mechanism, and generates a candidate layout based on the extracted features, to obtain the candidate layout corresponding to the base layout.

Wherein the generator may be comprised of an encoder and a decoder. In this step, the obtained basic layout may be input to a generator, that is, probability parameters and geometric parameters corresponding to the basic layout may be input to an encoder, and feature extraction may be performed on the basic layout in the encoder based on a multi-head attention mechanism, so that the extracted features may include semantic and spatial relationships between elements in the basic layout. In feature extraction based on a multi-headed attention mechanism, multiple sets of different linear projections (linear projections) learned independently can be used to transform queries, keys and values. The multiple sets of transformed queries, keys and values will then be fed in parallel into the attention pool so that the multiple attention pool outputs can be stitched together and transformed by another linear projection that can be learned to produce the final output, the extracted features, to accurately characterize the global semantic and spatial relationships between the individual elements.

And then, the extracted features can be decoded by a decoder to obtain candidate layouts, and the semantic and spatial relations of each element in the basic layout are extracted from the extracted features, so that the basic layout can be finely adjusted from the dimension of the element to obtain the candidate layouts, and the accuracy of the candidate layouts and the matching degree between the candidate layouts and the elements are improved.

In step 13, the generation loss corresponding to the generator and the discrimination loss of the discriminator are determined according to the candidate layout, the sample layout and the discriminator.

The sample layout may be a sample obtained by collecting and labeling a planar layout of an actual application. In this embodiment, training can be performed by the countermeasures between the generator and the arbiter, so that the generation loss and the discrimination loss can be calculated separately to perform training separately.

In step 14, the model formed by the generator and the discriminant is trained based on the generation loss and the discriminant loss, and the generator in the trained model is determined as the layout generation model.

When the generation loss and the discrimination loss meet the convergence condition, the model training is determined to be completed, if the generation loss or the discrimination loss does not meet the convergence condition, parameters of the generator can be adjusted based on the generation loss, so that probability parameters and geometric parameters when the basic layout is generated are adjusted, and when the basic layout is required to be generated again in the subsequent training process, the basic layout is generated based on the adjusted probability parameters and geometric parameters, so that the generation randomness of the basic layout is reduced, and the consistency between the basic layout and the real layout is improved. Accordingly, parameters of the discriminant can be adjusted based on the discriminant loss to improve the discriminant accuracy of the discriminant, so that parameter support is provided for further improving the accuracy judgment of the layout generated by the generator, and the accuracy and rationality of the candidate layout generated by the generator are improved.

As an example, when the parameters of the generator and the arbiter are adjusted, the adjustment may be performed by a gradient descent method, so that the output loss of the model may be reduced. The convergence condition may be that the generation loss and the discrimination loss are both within a preset loss range, or that the training frequency of the generator reaches a preset frequency threshold. After model training is completed, an accurate layout can be generated based on the generator, so that the generator can be directly determined as a layout generation model to simplify deployment and application of the layout generation model.

According to the technical scheme, the layout containing the diversity of the multiple elements can be automatically generated based on the layout generation model, in the process of generating the layout, the corresponding candidate layout is generated by extracting the semantics and the spatial features of the basic layout, so that the spatial relation among the elements in the generated layout can be improved, the use requirement of planar design is attached, the accuracy of the layout generation is improved, the accuracy of the generator can be further improved through the countermeasure training between the generator and the discriminator, the effectiveness of the layout generated based on the layout generation model is ensured, the workload of a designer is greatly reduced, and reliable data support is provided for the designer to perform planar design, so that the diversity and the efficiency of the layout generation are improved.

In one possible embodiment, the sample layout mentioned in step 13 may be determined by:

and obtaining annotation layout information, wherein each piece of annotation layout information comprises annotation category and position information of each element in the annotation layout.

Wherein a plurality of already applied planar layouts can be obtained in advance by the page acquisition technique in the related art. The planar layout is then displayed to a user for labeling by the user to obtain labeling layout information. For example, when a user performs labeling, the user may label the region based on a labeling frame, for example, the region of the main view may be divided by the labeling frame, and the corresponding element category is labeled by the user, that is, the labeling category is obtained as the main view category, meanwhile, the labeling frame region defined by the user may be determined as the main view region, and position information may be obtained by extracting position points from the main view region, for example, coordinates of points in the upper left corner and the lower right corner corresponding to the main view region may be obtained as the corresponding position information. Thus, a plurality of label layout information can be obtained in the above manner.

In order to ensure the universality of the generated layout, a plurality of element categories in the layout can be preset, for example, the universal layout can be counted to obtain the element categories, and the element categories can comprise a plurality of categories such as brand names, documents, disclaimers, main views, logo, buttons and the like. Further, to facilitate labeling of the layout, a sub-category corresponding to each element category may be determined, and for example, the document category may further include sub-categories such as selling points, operations, performance parameters, etc. to facilitate labeling of the categories by the user. The sub-categories contained in each element category can be stored in a corresponding relation mode, so that the element category corresponding to the labeling category can be conveniently and rapidly determined based on the corresponding relation.

As an example, if the labeling category is an element category, the labeling category may be directly used as the element category, and if the labeling category is a sub-category, the element category corresponding to the sub-category may be determined as the element category corresponding to the labeling category. For example, for an element in a layout, if a user performs labeling with a corresponding labeling category as a performance parameter, the labeling category can be determined to be a sub-category through the corresponding relationship, further, based on the corresponding relationship, the element category to which the sub-category belongs can be determined to be a document, and further, the element category of the element can be determined to be a document, so that the sample layout is obtained in combination with corresponding position information.

Therefore, through the technical scheme, the convenience of acquiring the labeling layout information can be ensured, the diversity and the universality of the obtained sample layout can be ensured, accurate and effective data support is provided for subsequent training of the layout generation model, and the training efficiency of the layout generation model is further improved.

In one possible embodiment, an exemplary implementation of determining the generation loss corresponding to the generator and the discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator in step 13 is as follows, as shown in fig. 2, and this step may include:

in step 21, according to the candidate layout, the sample layout and the discriminator, determination tags corresponding to the candidate layout and the sample layout, respectively, are determined.

Wherein the decision tag is used to characterize the rationality of the layout of the input. The arbiter may be a binary classifier, for example, an output of 1 indicates that the layout of the input is a reasonable layout, i.e., the decision tag is true, and an output of 0 indicates that the layout of the input is an unreasonable layout, i.e., the decision tag is false.

For example, the layout information of the candidate layout and the sample layout may be input to the discriminators, respectively, to perform feature extraction on the layout information by the discriminators to obtain output classifications, and then the output classifications of the discriminators are regarded as their respectively corresponding decision labels.

As another example, an exemplary implementation of determining decision tags corresponding to the candidate layout and the sample layout, respectively, according to the candidate layout, the sample layout, and the arbiter in step 21, may include:

in step 31, a candidate layout image is generated from the layout information corresponding to the candidate layout, and a sample layout image is generated from the layout information of the sample layout. The layout information may include representations of element categories and position information corresponding to the elements in the layout.

As an example, rendering of the element frame may be performed based on the layout information to obtain a candidate layout image, e.g., a size corresponding to the element frame and a position in the layout may be determined based on the position information, and a category corresponding to the element frame may be determined based on the element category. Accordingly, the sample layout image may be determined based on a similar manner.

As another example, the method of generating the candidate layout image according to the layout information corresponding to the candidate layout may include:

and inputting the layout information corresponding to the candidate layout into an encoder of the discriminator to obtain the coding feature corresponding to the layout information corresponding to the candidate layout. The encoder of the arbiter may be implemented based on an encoder in a transform model to encode the input layout information to obtain feature embedding, where the encoding feature of the i-th element in the layout information may be denoted as f (pi, θi), and f () is used to represent encoding logic.

And extracting the characteristics of the coding characteristics based on an attention mechanism to obtain the layout space characteristics of the layout information corresponding to the candidate layout.

For example, the attention mechanism may be a multi-headed attention mechanism, based on which the association between elements may be more focused to obtain a layout spatial feature, which may represent global graphic relationships between individual elements in the layout information to model global relationships between all graphic elements.

For example, the layout space features may include element types and position information of each element in the layout information, and also include global relationships between the elements, where each element may be represented in a line frame manner, and then the layout space features may be converted into line frames based on geometric conversion, that is, a candidate layout image is obtained, where each element is represented by a line frame.

As another example, the background map and the layout information may be further combined for rendering, such as superimposing the corresponding background map on the generated candidate layout image to obtain the final candidate layout image.

The method for generating the sample layout image according to the layout information of the sample layout is the same as the method for generating the candidate layout image according to the layout information of the candidate layout, which is not described herein.

By the technical scheme, the global graph relation among the elements in the candidate layout can be further extracted, the accurate representation of the relation among the elements is obtained, data support is provided for guaranteeing the accuracy and rationality of the candidate layout image, accurate data reference can be provided for carrying out rationality judgment on the image subsequently, the efficiency and accuracy of model training are improved, the layout characteristics are modeled through high-structure characteristics, and the arrangement deviation of the layout image caused by pixel dislocation during the synthetic layout rendering in the pixel space is reduced.

After determining the candidate layout image and the sample layout image, in step 32, inputting the candidate layout image into a discriminator, performing feature extraction on the candidate layout image by the discriminator, and determining a decision tag of the candidate layout according to the extracted features;

in step 33, the sample layout image is input to a discriminator to perform feature extraction on the sample layout image by the discriminator, and a determination tag of the sample layout is determined from the extracted features.

The input layout image can be subjected to feature extraction based on a CNN (Convolutional Neural Networks, convolutional neural network) in the discriminator, and input information can be subjected to translation invariant classification according to a hierarchical structure of the network in the CNN, so that the input layout image can be subjected to layout generation rationality judgment, the layout information can be accurately judged, the rationality of the layout image generated based on the layout information can be ensured, the judgment accuracy of the discriminator and the matching degree with an actual application scene are improved, and the generation precision of a generator is further improved.

Accordingly, the decision tag of the candidate layout and the decision tag of the sample layout can be determined through the above scheme, turning back to fig. 2, and in step 22, generating loss is determined according to the decision tag corresponding to the candidate layout and the labeling tag corresponding to the candidate layout;

in step 23, a discrimination loss is determined according to the determination labels corresponding to the candidate layout and the sample layout, respectively, and the labeling labels corresponding to the candidate layout and the sample layout, respectively.

In the model training stage, the candidate layout is a layout generated based on the generator, the labeling label of the candidate layout can be set to be false, the sample layout is obtained by sampling the layout in actual application, and the labeling label corresponding to the sample layout can be set to be true, so that a reference basis is provided for the discrimination accuracy of the discriminator.

For example, when determining to generate the loss, for the generator, the training target is that the generated candidate layout is similar to the sample layout, that is, the candidate layout is close to the decision tag of the sample layout, and accordingly, the generation loss LG may be represented as follows:

LG＝min log(1-D(G(z)))

where G (z) is used to represent the candidate layout and D (G (z)) is used to represent the probability that the decision tag of the candidate layout is true.

For the discriminant, the training targets are different between the generated candidate layout and the sample layout, and the sample layout needs to be determined to be true, and accordingly, the discrimination loss LD can be expressed as follows:

LD＝max(log(D(x))+log(1-D(G(z))))

wherein D (x) is used to represent the probability that the decision tag of the sample layout corresponds true.

Therefore, through the technical scheme, the generation loss of the generator and the discrimination loss of the discriminator can be respectively determined, so that the generator and the discriminator can respectively perform countermeasure optimization, the accuracy of candidate layout generated by the generator is improved, and support is improved for subsequent generation of various and reasonable layouts.

The present disclosure also provides a layout generation method, as shown in fig. 4, which may include:

in step 41, target element information in the layout to be generated is acquired, wherein the target element information may include element categories of elements and the number of elements under each element category. For example, a configuration page may be displayed for a user through a display page, in which the user may perform configuration of an element category and configuration of the number of elements, and for example, target element information obtained based on input of the user is { (main document, 1), (main view, 2), (logo, 1), (disclaimer, 1) }, where a number is used to represent the number of elements under the element category.

In step 42, a template layout corresponding to the target element information is queried from a preset layout library according to the target element information, wherein a plurality of template layouts obtained based on a layout generation model, which is generated based on the determination of the layout generation model, are stored in the layout library.

After the training of the layout generation model is completed, various layouts can be generated in advance based on the layout generation model, so that the layouts are stored to obtain a layout library, and when the layout library is used, the matched template layout can be determined from the layout library by directly aiming at element information, so that reasonable layout is provided for planar design.

In step 43, a target layout corresponding to the target element information is determined from the template layout.

For example, the determined template layout may be displayed to the user for selection by the user, and the template layout selected by the user may be determined as the target layout for subsequent floor plan layout design. As another example, the most used layout of the template layouts may be determined as the target layout to make a recommendation for the user.

Therefore, through the technical scheme, after the user determines the element types and the element numbers in the planar layout design, the corresponding template layout can be automatically matched, and the target layout is generated based on the layout generation model, so that accurate and effective target layout is provided for the user, the workload is saved for the planar layout design of the user, the planar layout design efficiency is improved, and the user experience is improved.

In one possible embodiment, the plurality of layouts in the layout library is determined by:

candidate element information for layout generation is determined, wherein the candidate element information comprises element categories and the number of elements under each element category.

The candidate element information can be randomly generated based on selectable element types and element numbers, so that a plurality of corresponding layouts are generated based on the candidate element information, the corresponding layouts under the plurality of candidate element information are obtained, and layout diversity in a layout library is improved.

And inputting the candidate element information into the layout generation model to obtain a plurality of generated layouts corresponding to the candidate element information.

The basic layout can be obtained according to the candidate element information and the probability parameters and the geometric parameters obtained after training, and the basic layout is input into the layout generation model, so that the layout output by the layout generation model can be used as the generated layout. The manner of this is described in detail above and will not be described in detail here.

The layout adjustment rule may be set based on an actual application scenario, for example, the layout adjustment rule may be element overlapping filtering, for example, if other elements overlap with the main view element in the generated layout, the element is reserved, and if other elements except the main view overlap, the element with the largest element area may be reserved. For example, the layout adjustment rule may be a height constraint rule, such as a rule that the heights between the main document and the sub document in the document are sequentially reduced, so as to conform to the browsing habit of the planar layout design. For example, the generated layout with the main view duty ratio smaller than the preset threshold value can be deleted directly.

For another example, the layout adjustment rules may also adjust the alignment of elements in the layout, such as single ended alignment, center alignment, etc., which is not limited by the present disclosure.

Further, the generated layout can be further adjusted according to the layout rule through the adjustment, so that the adjusted layout accords with the layout specification, the adjusted layout and the candidate element information can be stored in an associated mode, the follow-up query based on the target element information is facilitated, the layout associated with the candidate element information consistent with the target element information is used as the template layout corresponding to the target element information, and therefore layout diversity and rationality in a layout library are guaranteed, meanwhile, query efficiency of the template layout is improved, and user experience is further improved.

The present disclosure further provides a determining apparatus of a layout generation model, as shown in fig. 5, the apparatus 10 includes:

a first acquisition module 100 configured to acquire a base layout for layout generation;

a generating module 200 configured to input the basic layout into a generator, to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism, and generates the candidate layout based on the extracted features;

a first determining module 300 configured to determine a generation loss corresponding to the generator and a discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

a training module 400 configured to train the models formed by the generator and the arbiter according to the generation loss and the discrimination loss, and determine a generator of the trained models as a layout generation model.

Optionally, the sample layout is determined by:

Optionally, the first determining module includes:

a first determination submodule configured to determine a determination tag corresponding to the candidate layout and the sample layout respectively according to the candidate layout, the sample layout and the arbiter;

a second determining submodule configured to determine the generation loss according to the decision tag corresponding to the candidate layout and the labeling tag corresponding to the candidate layout;

and the second determination submodule is configured to determine the discrimination loss according to the determination labels respectively corresponding to the candidate layout and the sample layout and the labeling labels respectively corresponding to the candidate layout and the sample layout.

Optionally, the first determining submodule includes:

a generating sub-module configured to generate a candidate layout image according to the layout information corresponding to the candidate layout, and generate a sample layout image according to the layout information of the sample layout;

a fourth determination submodule configured to input the candidate layout image to the arbiter, to perform feature extraction on the candidate layout image by the arbiter, and to determine a determination tag of the candidate layout according to the extracted feature;

A fifth determination submodule configured to input the sample layout image to the arbiter, to perform feature extraction on the sample layout image by the arbiter, and to determine a decision tag of the sample layout according to the extracted features.

Optionally, the generating submodule includes:

the coding submodule is configured to input layout information corresponding to the candidate layout into an encoder of the discriminator to obtain coding features corresponding to the layout information corresponding to the candidate layout;

the extraction sub-module is configured to perform feature extraction on the coding features based on an attention mechanism, and obtain layout space features of layout information corresponding to the candidate layout;

and the rendering sub-module is configured to perform image rendering according to the layout space characteristics to obtain the candidate layout images.

The present disclosure also provides a layout generation method, the method including:

the query module is configured to query a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated by determining the layout generation model;

Optionally, the plurality of layouts in the layout library is determined by:

The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

The present disclosure also provides a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of determining a layout generation model or the steps of the method of layout generation provided by the present disclosure.

The present disclosure also provides a determining apparatus of a layout generation model, which may include:

A processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring a basic layout for layout generation;

The present disclosure also provides a layout generating device, including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring target element information in a layout to be generated;

inquiring a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model;

Fig. 6 is a block diagram of a layout generation model determination device or layout generation device, according to an example embodiment. For example, apparatus 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like.

Referring to fig. 6, apparatus 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls overall operation of the apparatus 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the method of determining a layout generation model or the method of layout generation described above. Further, the processing component 802 can include one or more modules that facilitate interactions between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the apparatus 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The power supply component 806 provides power to the various components of the device 800. The power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 800.

The multimedia component 808 includes a screen between the device 800 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or slide action, but also the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. The front camera and/or the rear camera may receive external multimedia data when the apparatus 800 is in an operational mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of the apparatus 800. For example, the sensor assembly 814 may detect an on/off state of the device 800, a relative positioning of the components, such as a display and keypad of the device 800, the sensor assembly 814 may also detect a change in position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, an orientation or acceleration/deceleration of the device 800, and a change in temperature of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communication between the apparatus 800 and other devices, either in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for performing the above-described method of determining a layout generation model or method of layout generation.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 804 including instructions executable by processor 820 of apparatus 800 to perform the above-described method of determining a layout generation model or method of layout generation. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

The present disclosure also provides a chip comprising a processor and an interface; the processor is configured to read instructions to perform the method of determining a layout generation model or the method of layout generation described above.

The apparatus may be a stand-alone electronic device or may be part of a stand-alone electronic device, for example, in one embodiment, the apparatus may be an integrated circuit (Integrated Circuit, IC) or a chip, where the integrated circuit may be an IC or may be a collection of ICs; the chip may include, but is not limited to, the following: GPU (Graphics Processing Unit, graphics processor), CPU (Central Processing Unit ), FPGA (Field Programmable Gate Array, programmable logic array), DSP (Digital Signal Processor ), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), SOC (System on Chip, SOC, system on Chip or System on Chip), etc. The above-described integrated circuit or chip may be used to execute executable instructions (or code) to implement the above-described method of determining a layout generation model or layout generation method. The executable instructions may be stored on the integrated circuit or chip or may be retrieved from another device or apparatus, such as the integrated circuit or chip including a processor, memory, and interface for communicating with other devices. The executable instructions may be stored in the processor, which when executed by the processor implements the above-described method of determining a layout generation model or layout generation method; alternatively, the integrated circuit or chip may receive executable instructions through the interface and transmit the executable instructions to the processor for execution to implement the above-described method for determining a layout generation model or method for generating a layout.

In another exemplary embodiment, a computer program product is also provided, comprising a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-described method of determining a layout generation model or method of layout generation when executed by the programmable apparatus.

Fig. 7 is a block diagram of a layout generation model determination device or layout generation device, according to an example embodiment. For example, the apparatus 1900 may be provided as a server. Referring to fig. 7, the apparatus 1900 includes a processing component 1922 that further includes one or more processors and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules each corresponding to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method of determining a layout generation model or method of layout generation.

The apparatus 1900 may further include a power component 1926 configured to perform power management of the apparatus 1900, a wired or wireless network interface 1950 configured to connect the apparatus 1900 to a network, and an input/output (I/O) interface 1958. The apparatus 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server ^TM ，Mac OS X ^TM ，Unix ^TM ，Linux ^TM ，FreeBSD ^TM Or the like.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

A method for determining a layout generation model, comprising:

acquiring a basic layout for layout generation;

inputting the basic layout into a generator to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

Determining the corresponding generation loss of the generator and the discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

training models formed by the generator and the discriminant according to the generation loss and the discriminant loss, and determining a generator in the trained models as a layout generation model.
The method of claim 1, wherein the sample layout is determined by:

acquiring annotation layout information, wherein each annotation layout information comprises annotation category and position information of each element in the annotation layout;

and determining element categories corresponding to the annotation categories according to the annotation layout information, and determining the element categories corresponding to each element in the annotation layout and the layout corresponding to the position information as the sample layout.
The method of claim 1, wherein determining a generation loss corresponding to the generator and a discrimination loss of the discriminator based on the candidate layout, sample layout, and discriminator comprises:

determining judging labels respectively corresponding to the candidate layout and the sample layout according to the candidate layout, the sample layout and the discriminator;

Determining the generation loss according to the judgment label corresponding to the candidate layout and the labeling label corresponding to the candidate layout;

and determining the discrimination loss according to the discrimination labels respectively corresponding to the candidate layout and the sample layout and the labeling labels respectively corresponding to the candidate layout and the sample layout.
A method according to claim 3, wherein said determining a decision tag for each of said candidate layout and said sample layout based on said candidate layout, said sample layout and said arbiter comprises:

generating a candidate layout image according to the layout information corresponding to the candidate layout, and generating a sample layout image according to the layout information of the sample layout;

inputting the candidate layout image into the discriminator, extracting features of the candidate layout image by the discriminator, and determining a judging label of the candidate layout according to the extracted features;

inputting the sample layout image into the discriminator, extracting features of the sample layout image by the discriminator, and determining a judging label of the sample layout according to the extracted features.
The method of claim 4, wherein generating a candidate layout image from layout information corresponding to the candidate layout comprises:

inputting the layout information corresponding to the candidate layout into an encoder of the discriminator to obtain coding features corresponding to the layout information corresponding to the candidate layout;

extracting the characteristics of the coding characteristics based on an attention mechanism to obtain layout space characteristics of layout information corresponding to the candidate layout;

and performing image rendering according to the layout space features to obtain the candidate layout images.
A layout generation method, the method comprising:

acquiring target element information in a layout to be generated;

inquiring a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model according to any one of claims 1-5;

and determining a target layout corresponding to the target element information according to the template layout.
The method of claim 6, wherein the plurality of layouts in the layout library are determined by:

Determining candidate element information for layout generation, wherein the candidate element information comprises element categories and the number of elements under each element category;

inputting the candidate element information into the layout generation model to obtain a plurality of generated layouts corresponding to the candidate element information;

and adjusting the generated layout according to a preset layout adjustment rule, and storing the adjusted layout and the candidate element information in a layout library in an associated manner.
A layout generation model determining apparatus, comprising:

a first acquisition module configured to acquire a base layout for layout generation;

a generation module configured to input the basic layout into a generator to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

a first determining module configured to determine a generation loss corresponding to the generator and a discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

a training module configured to train a model formed by the generator and the discriminant according to the generation loss and the discriminant loss, and determine a generator of the trained model as a layout generation model.
A layout generating apparatus, the apparatus comprising:

a second acquisition module configured to acquire target element information in a layout to be generated;

a query module configured to query a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model generated based on the determination of the layout generation model according to any one of claims 1-5 are stored in the layout library;

and the second determining module is configured to determine a target layout corresponding to the target element information according to the template layout.
A layout generation model determining apparatus, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring a basic layout for layout generation;

inputting the basic layout into a generator to obtain a candidate layout corresponding to the basic layout, wherein the generator performs feature extraction on the basic layout based on a multi-head attention mechanism and generates the candidate layout based on the extracted features;

Determining the corresponding generation loss of the generator and the discrimination loss of the discriminator according to the candidate layout, the sample layout and the discriminator;

training models formed by the generator and the discriminant according to the generation loss and the discriminant loss, and determining a generator in the trained models as a layout generation model.
A layout generating apparatus, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

acquiring target element information in a layout to be generated;

inquiring a template layout corresponding to the target element information from a preset layout library according to the target element information, wherein a plurality of layouts obtained based on a layout generation model are stored in the layout library, and the layout generation model is generated based on the determination of the layout generation model according to any one of claims 1-5;

and determining a target layout corresponding to the target element information according to the template layout.
A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the steps of the method of any of claims 1-7.
A chip, comprising a processor and an interface; the processor is configured to read instructions to perform the method of any of claims 1-7.