LU501299B1

LU501299B1 - Block group detection

Info

Publication number: LU501299B1
Application number: LU501299A
Authority: LU
Inventors: Nils Blum-Oeste; Arthur Caillau; Jelle Akkerman
Original assignee: Pitch Software Gmbh
Priority date: 2022-01-21
Filing date: 2022-01-21
Publication date: 2023-07-24

Abstract

A computer-implemented method (10, 110) and computer (100) for grouping of blocks (20a - 20r) on a canvas (15) are disclosed. The method (10) comprises detecting (S100) a user input , determining (S110) an archetype (90) for the plurality of the selected blocks (20a - 20r) of the user input , identifying (S130) blocks (20a - 20r) from the plurality of the selected blocks (20a - 20r) having a same block type (20t) as a first block (20a), analyzing (S140) the identified blocks (20a - 20r) having the same block type (20t) as the first block (20a), comparing a candidate group (S170) to the archetype (90) of the user input , and storing (S180) the candidate group as group.

Description

9257310 (v2) SOMARRISON

LU501299

Description

Title: BLOCK GROUP DETECTION

FIELD OF THE INVENTION

[0001] The field of the invention relates to a computer-implemented method and apparatus for grouping a plurality of blocks on a canvas of a presentation system.

BACKGROUND OF THE INVENTION

[0002] Effective communication with both internal and external audiences is important and electronic presentations using, for example, Microsoft’s PowerPoint software or Apple’s

Keynote software are often used for communicating information to an audience. This software creates a canvas on which a user can create slides and, on the slides, blocks. These blocks include images and texts and can usually be individually formatted.

[0003] Such presentations can take a long time to prepare and to format correctly. A presentation may require regular updating (depending on its content) and new information may need to be added. This updating and addition may require re-arrangement of the text on the slides which can lead to a need to re-arrange the blocks on the slide. This is time- consuming for a user to perform and there is therefore a need to establish an improved method for amending and reformatting the slides.

[0004] There are several patent documents known that describe a computer-implemented method for a re-arrangement of objects in a graphics editing program. For example, US

Patent 8,659,621 (Stiglitz et al, assigned to Apple) teaches a method which involves scanning of a drawing area of a graphics editing program to identify blocks and identifying an overlap of the blocks to eliminate the overlap.

[0005] US Patent US 9,412,186 B2 (Allyn Barry Christopher et al, assigned to Microsoft

Technology Licensing LLC) discloses a computer system that assists the user of computer drawing programs to accurately place elements on a canvas. The system establishes and displays visual guidelines according to elements already placed on the canvas. As a new block 1s dragged for placement on the canvas, the system analyzes areas around the new

9257310 (v2) SOMARRISON

LU501299 block for already-placed elements. The system then determines a spacing between the already-placed elements and indicates to the user a position to which the dragged block should be placed to maintain the same spacing as the blocks already placed on the canvas.

This U.S. patent describes a solution for aligning the elements with already-placed blocks of the canvas.

[0006] European Patent Application EP 0 848 351 A2 (Richard Gartland, assigned to Adobe

Systems Inc.) discloses a method and apparatus for repositioning the blocks on the canvas in response to the user requesting to change a canvas layout and/or size of the canvas. The method includes detecting the elements on the canvas and receiving a user request to change the layout or size of the canvas for display of the elements. Using the detected elements and the user request, the method realigns and resizes the elements according to the user request.

The method disclosed allows to automatically realign and/or resize elements on the canvas for improved visual presentation of the elements to the user after changing the layout and/or size of the canvas.

SUMMARY OF THE INVENTION

[0007] A computer-implemented method and system for grouping of blocks on a canvas is disclosed. It is an object of the system and computer-implemented method determine groups of elements on a canvas for placement of these elements. The method comprises the steps of detecting a user input comprising at least one of a plurality of blocks selected by a user. The method further comprises determining an archetype for the plurality of the blocks of the user input. The method also comprises identifying blocks from the plurality of the selected blocks having a same block type as a first block. The method also comprises analyzing the identified blocks having the same block type as the first block. The method further comprises comparing a candidate group to the archetype of the user input and storing the candidate group as group in a memory of the computer.

[0008] The method also comprises processing the archetype for the user input, wherein the processing of the archetype is done in a reading order. The reading order is defined by a processor. The method further comprises locating of a scaled bounding box over the identified blocks and analyzing the blocks within a search space.

9257310 (v2) SOMARRISON

LU501299

[0009] A further computer-implemented method for grouping of blocks is disclosed. The method comprises detecting of a plurality of blocks on the canvas, determining a number of groups for the plurality of the blocks on the canvas, removing of outliers from the blocks, determining candidate configurations for the plurality of blocks on the canvas, determining anumber of groups for the plurality of the blocks on the canvas, and selecting a configuration from the candidate configurations. The removing of the outliers is based on at least one of a block type or a location of the block relative to the canvas. The determining of the candidate configurations for the plurality of blocks is done by analyzing the blocks within a search space. The determining of the candidate configurations for the plurality of blocks further comprises comparing a candidate group to an archetype and storing the candidate group as a group. The determining of the candidate configurations for the plurality of the blocks further comprises storing the candidate group as group.

[0010] A computer for grouping of blocks on a canvas is also disclosed. The computer comprises a processor for determination of an archetype of a plurality of blocks, a memory for storing a group of blocks, and a display unit for displaying a canvas comprising at least one of the blocks.

DESCRIPTION OF THE FIGURES

[0011] Fig. 1 shows a first view of a computer for grouping of blocks on a canvas.

[0012] Fig. 2 shows a detailed view the canvas.

[0013] Fig. 3A shows a first example demonstrating the processing of an archetype.

[0014] Fig. 3B shows a first example demonstrating the processing of an archetype.

[0015] Fig. 4A shows a further example of the canvas comprising blocks.

[0016] Fig. 4B shows a first example of the analyzing of blocks.

[0017] Fig. 4C shows a second example of the analyzing of blocks.

[0018] Fig. SA and Fig. SB show a flow chart describing the method to assign a group of the blocks.

[0019] Fig. 6 shows a flow chart describing a method for the grouping of the blocks on the canvas.

[0020] Fig. 7 shows a first example for use of the method for placement of the blocks on the canvas.

9257310 (v2) SOMARRISON

LU501299

[0021] Fig. 8A shows a first view of a second example for use of the method for placement of the blocks on the canvas.

[0022] Fig. 8B shows a second view of the second example for use of the method for placement of the blocks on the canvas.

[0023] Fig. OA shows a first view of a third example for use of the method for swapping of groups of blocks on the canvas.

[0024] Fig. 9B shows a second view of the third example for use of the method for swapping of groups of blocks on the canvas.

DETAILED DESCRIPTION OF THE INVENTION

[0025] The invention will now be described on the basis of the figures. It will be understood that the embodiments and aspects of the invention described herein are only examples and do not limit the protective scope of the claims in any way. The invention is defined by the claims and their equivalents. It will be understood that features of one aspect or embodiment of the invention can be combined with a feature of a different aspect or aspects and/or embodiments of the invention.

[0026] Fig. 1 shows a first view of a computer 100 for alignment of blocks 20a — 20r on a canvas 15. In this example, the blocks 20a — 20r will be grouped. The computer 100 comprises a processor 150, a graphics display memory 160, and a display unit 170. The processor 150 is used for running a method 10 for the grouping of the blocks 20a — 20r on the canvas 15. The graphics display memory 160 is used for storing items of data, such as coordinates or reference to a position of the blocks 20a — 20r, relating to the grouping of the blocks 20a — 20r relative to the canvas 15. The display unit 170 is used for displaying the blocks 20a — 20r and the canvas 15.

[0027] The canvas 15 comprises a top edge 15t, a bottom edge 15b, a left edge 151, and a right edge 151. The top edge 15t indicates a top border of the canvas 15. The bottom edge 15b indicates a bottom border of the canvas 15. The left edge 151 indicates a left border of the canvas 15. The right edge 15r indicates a right border of the canvas 15. A vertical dimension 15v of the canvas 15 is defined by a vertical distance between the top edge 15t and the bottom edge 15b. A horizontal dimension 15h of the canvas 15 is defined by a horizontal distance between the left edge 151 and the right edge 15r.

9287310 (v2) SON ARRON

LU501299

[0028] The blocks 20a — 20r are displayed on the canvas 15. The blocks 20a — 20r have, for example, different geometrical shapes, such as but not limited to a rectangle, a square, a circle, or an ellipsoid. The blocks 20a — 20r are identified with a block type 20t. The block types 20t describe a content of the blocks 20a — 20r. Non-limiting examples of the content of the block types 20t include an image or a text. If, for example, a first block 20a comprises a picture, the block type 20t for this first block 20a is “image-type-block”. If, for example, the second block 20b comprises a plain text, the block type 20t for this second block 20b is “text-type-block”. If, for example, the third block 20c comprises another plain text, the block type 20t for this third block 20c is “text-type-block”.

[0029] A cursor 30 is displayed on the display unit 170. The cursor 30 is used by a user (not shown) to hover over, point at, or select the one or more blocks 20a — 20r displayed on the canvas 15. The processor 150 identifies a position of the cursor 30 relative to the canvas 15.

The processor 150 sends a signal to the display unit 170 to display an appropriate symbol at the position of the cursor 30 on the display unit 170.

[0030] The processor 150 accesses the items of data in the graphics display memory 160 and is thereby able to identify the cursor 30 position relative to the canvas 15 and the blocks 20a — 20r displayed on the canvas 15. The processor 150 can detect if the user has selected thereby one or a plurality of the blocks 20a — 20r. The processor 150 detects the one or the plurality of the blocks 20a — 20r selected by the user and stores the selected one or more blocks 20a — 20r in the graphics display memory 160 as a user input.

[0031] If, for example, the user has selected the first block 20a using the cursor 30, the processor 150 stores a reference to a position of the first block 20a on the canvas 15 as the user input in the memory 160. This reference to the position of the first block 20a is, for example, a vector pointing to the position of first block 20a expressed in coordinates relative tothe canvas 15 or a set of coordinates expressing the position of the first block 20a on the canvas 15. If, for example, the user has selected more than a single one of the plurality of the blocks 20a — 20r using the cursor 30, then the processor stores the reference to the position relating to the selected plurality of blocks 20a — 20r as the user input in the memory 160.

[0032] The processor 150 can calculate a bounding box 60 surrounding the user input. This bounding box 60 is shown as a dotted line on Figs 1 and 2. The bounding box 60 spans an area of the canvas 15 covered by the selected blocks 20a — 20r selected by the user. The

9257310 (v2) SOMARRISON

LU501299 bounding box 60 has a bounding box width 60w, a bounding box height 60h, and a bounding box area 60a. The coordinates of the bounding box 60 are stored in the graphics display memory 160 by the processor 150 as items of data expressed in coordinates relative to the canvas 15.

[0033] The coordinates of the bounding box 60 are expressed in position information regarding a horizontal or x-direction of the canvas 15 and in position information regarding a vertical or y-direction of the canvas 15. The bounding box width 60w is normalized with respect to the horizontal dimension 15h of the canvas 15. The bounding box height 60h is normalized with respect to the vertical dimension 15v of the canvas 15. The bounding box area 60a is calculated by the processor 150 to be the bounding box area 60a normalized by a canvas area. The canvas area is an area encompassed by the vertical dimension 15v of the canvas 15 and the horizontal dimension 15h of the canvas 15. This normalized bounding box area 60a is stored in the memory 160.

[0034] Fig. 2 shows a more detailed view of the canvas 15 described in Fig. 1 having the blocks 20a — 20r. Let us suppose that the user has selected the first block 20a, the second block 20b, and the third block 20c using the cursor 30. Let us also suppose that the block type 20t of the first block 20a is “image-type-block”, the block type 20t of the second block 20b is “text-type-block”, and that the block type 20t of the third block 20c is “text-type- block”. The user has not selected blocks 20d — 20r. Let us further suppose that the block type 20t of blocks 20d, 20g, 20j, 20m, and 20p is “image-type-block”, wherein the block type 20t of blocks 20e — 20f, 20h — 201, 20k — 201, 20n — 200, and 20q — 20r is “text-type-block”. The processor 150 detects (as set out in step 100 of Fig. SA) the selected blocks 20a — 20c and stores these blocks 20a — 20c as the user input.

[0035] The processor 150 determines (step S110 in Fig. SA) an archetype 90 for the blocks — 20a — 20c of the user input. The archetype 90 comprises items of data on the blocks 20a — 20c in the user input. More specifically, the archetype 90 comprises items of data on the block type 20t of the blocks 20a — 20c of the user input. The archetype 90 further comprises items of data on the normalized bounding box 60 surrounding the blocks 20a — 20c of the user input. These items of data on the normalized bounding 60 box are referred to as “archetype features”.

[0036] The archetype 90 is processed (see also step S120 below) by the processor 150 into items of data that are used for guiding a tree search algorithm (see below). The processing

9257310 (v2) SOMARRISON

LU501299 of the archetype 90 by the processor 150 is done using a so-called “reading order”. The reading order is an order in which the user would enunciate or read through the blocks 20a — 20r on the canvas 15. This reading order is used by the processor 150 to identify the blocks 20a — 20c of the user input or to identify the blocks 20a — 20r the canvas 15 (as will be shown later).

[0037] The reading order can be defined in multiple ways. In one aspect of the invention, the reading order is based on the Euclidian distance based on the distance of the blocks 20a — 20r measured to a fixed position of the canvas 15, for example, an intersection point of the top edge 15t and the left edge 151 of the canvas 15. A measure of the Euclidian distance for defining the reading order is, in this aspect, the distance from the intersection point of the canvas 15 to the to the top left hand corner of the blocks 20a — 20r.

[0038] The Euclidian distance could, in another aspect, also be calculated as a distance from the intersection point to center points of the blocks 20a — 20r. In yet another aspect, the reading order could also be defined to be “line by line” and “right-to-left”, as if reading a book from one page to another. The reading order would, in this aspect, be from block 20a in top edge 15t to block 201 in the bottom edge 15b and from block 20a in the left edge 151 to block 20r in the right edge 15r of the canvas 15.

[0039] Various other aspects could be incorporated to adjust the reading order. Machine learning tools could “learn” the natural manner in which the reading order is calculated. The exact limitation of this reading order is not limiting of the invention.

[0040] Fig. 3A shows a first example explaining the processing of the archetype 90. Let us suppose the user has selected four blocks 20a — 20d. The user input therefore comprises the four blocks 20a — 20d. The processing of the archetype 90 comprises determining the block types 20t by the processor 150. This determining of the block types 20t of these four blocks 20a — 20d is done using the reading order. The processor 150 first determines the block type 20t of the first block 20a in the reading order to be “image-type-block”, as an image of a person is shown. The processor 150 then determines in the example shown in Fig. 3A the block type 20t of the second block 20b (in the reading order on the next line) to be “text- type-block”, as the word “individuals” is used. The processor 150 then determines the block type 20t of the third block 20c (in the reading order — moving from left to right) to be also “text-type-block”, as the symbol “$” is shown. The processor 150 then determines the block type 20t of the fourth block 20d (in the reading order on the following line) to be also “text-

9257310 (v2) SOMARRISON

LU501299 type-block”, as the words “Creatives, freelances is shown”. The block types 20t are therefore, in this example, “image-type-block”, “text-type-block”, “text-type-block”, “text- type-block” for the user input shown in Fig. 3A.

[0041] The processing of the archetype 90 (see also step S120 below) also comprises the calculation of the bounding box 60 surrounding the blocks 20a — 20d as one of the archetype features. This feature of the archetype 90 are represented by a so-called “group embedding”.

The group embedding is a mathematical quantitative vector representation of the features of the archetype 90. The group embedding comprises, for example, a vector with three dimensions. The first dimension comprises the normalized bounding box width 60w, the second dimension comprises the normalized bounding box height 60h, and the third dimension comprises the normalized bounding box area 60a. An exemplary representation of the group embedding is shown in the following equation. mie bounding box width re

Group embedding = | normalized bounding box height 60h normalized bounding box area 60a

[0042] Fig. 3B shows an additional pre-processing step for the archetype 90 by the processor 150 to speed up the tree search algorithm. Fig. 3B shows the user input comprising the same blocks 20a — 20d as Fig. 3A. In this case, the pre-processing step comprises determination of offset vectors for the blocks 20a-20d. Fig. 3B further shows the reading order for calculating offset vectors between the blocks 20a — 20d which is identical to the example of

Fig. 3A. This calculating of the offset vectors of the blocks 20a — 20d with respect to the reading order is done by the processor 150. The processor 150 calculates a first offset vector between the first block 20a and the second block 20b. This first vector is a two-dimensional vector distance between the top left hand corner of the first block 20a and the top left hand corner of the second block 20b in cartesian coordinates. An example for the vector distance between the top left hand corner of the first block 20a and the top left hand corner of the second block 20b is given below and indicates that the second block 20b is shifted by only 0.01 in the x-direction, but by 0.5 in the y-direction. These units are arbitrary. These units are, for example, dimensions relative to the horizontal or x-direction of the canvas 15 and the vertical or y-direction of the canvas 15. In the present example, the second block 20b is

9257310 (v2) SOMARRISON

LU501299 shifted by 1% of the x-direction of the canvas 15 and by 50% of the y-direction of the canvas 15.

First offset vector = [0.01 0.5 ]

[0043] Similarly, the processor calculates a second offset vector between the top left hand corner of the second block 20b and the top left hand corner of the third block 20c and a third offset vector between the top left hand corner of the third block 20c and the top left hand corner of the fourth block 20d. It will be seen that the second offset vector has a value of 0.3 for the x-direction but only 0.02 for the y-direction. The other offset vectors are similarly calculated.

Second offset vector = [0.3 0.02 ]

Third offset vector = [-0.3 0.1]

[0044] These offset vectors for the user input are stored in the memory 170 by the processor 150.

[0045] Fig. 4A shows a further example of the canvas 15 comprising blocks 20a — 201. As can be seen from Fig. 4A, the block type 20t of blocks 20a, 20e, and 201 1s “image-type- block”, the block type 20t of blocks 20b — 20d, blocks 20f — 20h, and blocks 20j — 201 is “text-type-block”.

[0046] Let us suppose the user has selected blocks 20a — 20d. The processor 150 determines the archetype 90 for the blocks 20a — 20d of the user input (see step S110 in Fig. SA). This determining of the archetype 90 is done in the reading order. The processor 150 determines the archetype 90 in this example in the reading order of: 1) first block 20a, 2) second block 20b, 3) third block 20c, 4) fourth block 20d.

[0047] The first block 20a is referred to as “first reading order block”, the second block 20b is referred to as “second reading order block”, the third block is referred to as “third reading order block” and the fourth block is referred to as “fourth reading order block” to indicate the order in which the selected blocks 20a-20d are read. The processor 150 determines the block type 20t of the first block 20a to be “image-type-block”, the block type 20t of the second block 20b to be “text-type-block”, the block type 20t of the third block 20c to be

9257310 (v2) SOMARRISON

LU501299 “text-type-block” and the block type 20t of the fourth block 20d to be “text-type-block”. The archetype 90 is therefore set to comprise “image-type-block”, “text-type-block”, “text-type- block”, “text-type-block”. The bounding box 60 surrounding the blocks 20a — 20d and the offset vectors for the blocks 20a-20d are calculated by the processor 150 and stored as one of the archetype features (as already described in Fig. 2).

[0048] The processor 150 further calculates a scaled bounding box 60s (see step S140 of

Fig. 3A). The scaled bounding box 60s is calculated by increasing the bounding box width 60w and the bounding box height 60h of the bounding box 60 by, for example, +20%. Fig. 4B shows the detailed view of the canvas 15 described in Fig. 4A with the scaled bounding box 60s. The scaled bounding box 60s is located over the blocks 20e, 20i having the same block type 20t as the first block 20a of the user input, e.g., “image-type-block” (step S150 in Fig. 3A).

[0049] Fig. 4B shows a first example of the analyzing of the blocks 20e — 20h with the same block type 20t as the first block 20a in the user input. In the example shown, the processor 150 analyzes the blocks 20e — 20h. The block type 20t of a fifth block 20e is “image-type- block” (see above). The processor 150 then analyzes (see also step S140 in Fig. 3A) the block type 20t of a sixth block 20f, the block type 20t of a seventh block 20g, and the block type 20t of an eight block 20h. The processor 150 then compares the block type 20t of the second block 20b to the block type 20t of the sixth block 20f. The processor 150 further compares the block type 20t of the third block 20b to the block type 20t of the seventh block 20g. The processor 150 further compares the block type 20t of the fourth block 20d to the block type 20t of the eighth block 20h.

[0050] Fig. 4C shows a second example of the analyzing of the blocks 201 — 201 with the same block type 20t as the first block 20a in the user input. In the example shown, the processor 150 analyzes the blocks 201 — 201. The block type 20t of a ninth block 201 is “image-type-block” (see above). The processor 150 analyzes (see also step S140 in Fig. 3A) the block type 20t of a tenth block 20j, the block type 20t of an eleventh block 20k, and the block type 20t of a twelfth block 201. The processor 150 then compares the block type 20t of the second block 20b to the block type 20t of the tenth block 20j. The processor 150 further compares the block type 20t of the third block 20b to the block type 20t of the eleventh block 20k. The processor 150 further compares the block type 20t of the fourth block 20d to the block type 20t of the twelfth block 201.

9257310 (v2) SOMARRISON

LU501299

[0051] Fig. SA and Fig. SB show a flow chart describing the method 10 for grouping of the blocks 20a — 201 (as shown in Fig. 4A) on the canvas 15 using the archetype 90. The method 10 takes the archetype 90 and all the blocks 20a — 201 on the canvas 15 as input and is represented by the following function f. The blocks 20a — 20d of the archetype 90 are input into the function f as first input. The blocks 20e — 20h are input into function f as second input.

[0052] This can be represented by the following equation which returns a collection of groups (represented by the first hash sign). The groups are a set of the blocks 20e — 20h having identifications id1, 1d2, 1d3, id4 and represented by the nested hash signs #. f([20a, 20b,20c, 20d], [20e, 20f, 20g, 20h]) = #{#{id1 id2} #{id3 id4}}

[0053] The processor 150 detects in step S100 the user input. The user input comprises, for example, the selection of the first block 20a, the second block 20b, the third block 20c, and the fourth block 20d as explained in the description of Fig. 4A and the processor 150 determines in step S110 the archetype 90 for the blocks 20a — 20d of this user input. The block types 20t of the selected blocks 20a — 20d are determined to be “image-type-block”, “text-type-block”, “text-type-block”, “text-type-block” (see description of Fig. 4A) and these block types 20t form part of the features of the archetype 90. Another one of the features of the archetype 90 is the dimension of the bounding box 60 (see description of Fig. 2) and the offset vectors. The processing of the archetype 90 in step S120 comprises, for example, defining the dimensions of the bounding box 60 surrounding the first block 20a, the second block 20b, the third block 20c, and the fourth block 20d.

[0054] The processor 150 identifies, in step S130, those blocks 20e — 201 on the canvas 15 which have the same block type 20t as the first block 20a. In the present example, the block type 20t of the first block 20a is the “image-type-block”. The processor 150 iterates through those of the blocks 20e — 201 which are not selected by the user. In the example shown in

Fig. 4A, the processor 150 iterates through the blocks 20e — 201 on the canvas 15 to identify those ones of the blocks 20e — 201 having the same block type 20t as the first block 20a in the user input (e.g., “image-type-block”).

[0055] The processor 150 analyzes in step S140 the identified blocks 20 having the same block type 20t as the first block 20a of the user input. The processor 150 locates a scaled

9257310 (v2) SOMARRISON

LU501299 bounding box 60s over ones of the blocks 20e — 201 having the same block type 20t as the first block 20. In the example shown in Fig. 4A, the block type 20t of the first block 20a of the user input is “image-type-block”. In the example shown in Fig. 4A, the blocks 20e and 201 also have the block type 20t “image-type-block”. The processor 150 therefore locates a first scaled bounding box 60s over the block 20e and a second scaled bounding box 60s over the block 201 having the same block type 20t as the first block 20a of the user input in step

S150.

[0056] This locating of the first scaled bounding box 60s and the second scale bounding box 60s over the corresponding blocks 20e and 20i defines a first search space 35a and a second search space 35b for an area of the canvas 15 proximate to each of the identified blocks 20e, 201 having the same block type 20t as the first block 20a. A number of search spaces 35 is, in a preferred aspect, identical to a number of the blocks 20e — 201 having the same block type 20t as the first block 20a of the user input. The processor 150 identifies the blocks 20e — 20h located within this first search space 35a and the blocks 201 — 201 located within this second search space 35b.

[0057] The processor 150 then analyzes in step S160 the blocks 20e — 20h located within this first search space 35a and the blocks 201 — 201 located within this second search space 35b. This analyzing is done in the reading order defined by the archetype 90.

[0058] In the example shown in Fig. 4A the processor 150 analyzes the first search space 35a defined around the fifth block 20e. The processor 150 determines the block type 20t of the sixth block 20f, the seventh block 20g, and the eight block 20h. If the processor 150 determines that the sixth block 20f of the first search space 35a has the same block type 20t as the second block 20b of the archetype 90, the processor 150 continues to analyze the seventh block 20g in the first search space 35a. If the processor 150 determines that the sixth block 20f of the first search 35a space does not have the same block type 20t as the second block 20b of the archetype 90, the processor 150 does not continue to analyze the first search space 35a. If the processor 150 determines that the seventh block 20g of the first search space 35a has the same block type as the third block 20c of the archetype 90, the processor 150 continues to analyze the eighth block 20h in the first search space 35a. If the processor 150 determines that the seventh block 20g of the first search space 35a space does not have the same block type 20t as the third block 20c of the archetype 90, the processor 150 does not continue to analyze the first search space 35a. If the processor 150 determines that the eighth

9257310 (v2) SOMARRISON

LU501299 block 20h of the first search space 35a has the same block type as the fourth block 20d of the archetype 90, the processor 150 stores the blocks 20e, 20f, 20g, and 20h as a so-called “candidate group” 40c.

[0059] The processor 150 further calculates the group embedding for the candidate group 40c and stores the calculated group embedding for the candidate group 40c as feature of the candidate group 40c in the memory 160. If the processor 150 determines that the eighth block 20h of the first search 35a space does not have the same block type 20t as the fourth block 20d of the archetype 90, the processor 150 does not calculated the group embedding for the first search space 35a.

[0060] More simply put, in one aspect the processor 150 analyzes each of the blocks 20e — 20h located within the first search space 35a and each of the blocks 201 — 201 located within the second search space 35b in the reading order. This analyzing comprises comparing the block type 20t of each of the blocks 20e — 20h, 201 — 201 in the search space 35a, 35b to the block type 20t of the respective blocks 20a — 20d in the archetype 90.

[0061] The analyzing of the blocks 20e — 20h located within the first search space 35a and the blocks 20i — 201 located within the second search space 35b by the processor 150 is referred to as using a “tree search algorithm”. This tree search ensures that the nodes in a tree data structure are only iterated once during so-called “iterations”. For example, a Depth

First Search (DFS) with guiding heuristics is used to find groups within the data structure with fewer iterations. To provide real time group prediction under constraints of very low latency of, for example, less than 50ms and low CPU processing power, a guiding heuristic h is applied to the tree search algorithm.

[0062] This guiding heuristic h is adapted to pick a best one of the blocks 20d — 20r from the search space 35. In one aspect, the best one of the blocks 20d — 20r is the one of the blocks 20d — 20r having a similar offset vector to the offset vector in the archetype 90 (see also description of Fig. 3B). A number of iterations for the tree search algorithm is, for example, limited by an iteration budget k. The iteration budget k forces the tree search algorithm to stop and return the already found groups if the tree search algorithm has iterated over the tree for longer than k steps. Other tree search algorithms may also be used for the analyzing of the blocks 20d — 20r. For example, a breadth first search (BFS), a best first search, an A*, or a Dijkstra algorithm may also be used.

9257310 (v2) SOMARRISON

LU501299

[0063] The processor 150 further calculates the offset vectors for all blocks 20f — 20h in the first search space 35a having the same block type 20t as the second block 20b. These computed offset vectors for the blocks 20f — 20h in the first search space 35a having the same block type 20t as the second block 20b are then sorted by the Euclidian Distance from the first offset vector (see description of Fig. 3B). The processor 150 also calculates offset vectors for all blocks 20f — 20h in the first search space 35a having the same block type 20t as the third block 20c. These computed offset vectors for the blocks 20f — 20h in the first search space 35a having the same block type 20t as the third block 20c are then sorted by the Euclidian Distance from the second offset vector (see description of Fig. 3B). The processor 150 further calculates offset vectors for all blocks 20f — 20h in the first search space 35a having the same block type 20t as the fourth block 20d. These computed offset vectors for the blocks 20f — 20h in the first search space 35a having the same block type 20t as the fourth block 20d are then sorted by the Euclidian Distance from the third offset vector (see description of Fig. 3B).

[0064] The processor 150 further calculates offset vectors for all blocks 20; — 201 in the second search space 35b having the same block type 20t as the second block 20b. These computed offset vectors for the blocks 20j — 201 in the second search space 35b having the same block type 20t as the second block 20b are then sorted by the Euclidian Distance from the first offset vector (see description of Fig. 3B). The processor 150 also calculates offset vectors for all blocks 20j — 201 in the second search space 35b having the same block type 20t as the third block 20c. These computed offset vectors for the blocks 20j — 201 in the second search space 35b having the same block type 20t as the third block 20c are then sorted by the Euclidian Distance from the second offset vector (see description of Fig. 3B). The processor 150 further calculates offset vectors for all blocks 20j — 201 in the second search space 35b having the same block type 20t as the fourth block 20d. These computed offset vectors for the blocks 20; — 201 in the second search space 35b having the same block type 20t as the fourth block 20d are then sorted by the Euclidian Distance from the third offset vector (see description of Fig. 5B).

[0065] Fig. SB shows a second part of a flow chart describing the method 10. The processor 150 compares in step S170 the candidate group 40c to the archetype 90. This comparing by the processor 150 is done by calculating the Euclidian distance between the group embedding of the archetype 90 and the group embedding of the candidate group 40c.

92573LU (vz) SONNENGERG

LU501299

[0066] In the example shown in Fig. 4A, the group embedding of the archetype 90 comprising the blocks 20a — 20d is, for example, as follows. normalized bounding box width 60w

Group embedding [archetype] = | normalized bounding box height 60h normalized bounding box area 60a 0.23 = | 0.34 0.12

[0067] In the example shown in Fig. 4A, the group embedding of the candidate group 40c of the first search space 35a comprising the blocks 20e — 20h is, for example, as follows.

Group embedding [candidate group 40c of the first search space 35a] width of bounding box of first search space 35 0.25 = | height of bounding box of first search space 35| = | 045 area of bounding box of first search space 35 0.23

[0068] The Euclidian distance between the group embedding of the archetype 90 and the candidate group 40c of the first search space 35a can then be calculated as is shown in the following example.

Euclidian distance = /(0.25 — 0.23)? + (0.45 — 0.34)? + (0.23 — 0.12)? = 0.156844

[0069] If the Euclidean distance between the archetype 90 and the group embedding of the candidate group 40c is less than a defined threshold, such as 0.2, the candidate group 40c is stored in the memory 160 as a first group 40a in step S180. If the Fuclidean distance is larger than the defined threshold value, the candidate group 40c is not stored in the memory and discarded in step S190.

[0070] The processor 150 checks in step S200 if all of the blocks 20d — 20r having the same block type 20t as the first block 20a have been analyzed. If all blocks 20d — 20r having the same block type 20t as the first block 20a have been analyzed, the method 10 ends. If not all

9257310 (v2) SOMARRISON

LU501299 of the blocks 20d — 20r having the same block type 20t as the first block 20a have been analyzed, the processor 150 continues to analyze this one of blocks 20d — 20r /these blocks 20d — 20r by reiterating step S140 to step S200.

[0071] Fig. 6 shows a flow chart describing a method 110 for the grouping of the blocks 20a — 20r on the canvas 15. The method 110 initially comprises the steps S100 to S200 of the method 10 shown in Fig. SA and Fig. SB. The method 110 further comprises identifying a configuration of the blocks 20a — 20r on the canvas 15 as will be explained below. Let us suppose the user has not selected any of the blocks 20a — 20r but wishes to, for example, group the blocks 20a — 20r. In step SS of the method 110 the processor 150 detects the blocks 20a — 20r on the canvas 15. The detecting of the plurality of the blocks 20a — 20r on the canvas 15 further comprises removing outliers from the blocks 20a — 20r on the canvas 15 in step S10.

[0072] The detecting of the outliers is based on, for example, the block type 20t of the blocks 20a — 20r. The detecting of the outliers is also, for example, based on a location of the blocks 20a — 20r relative to the canvas 15. If, for example, the processor 150 determines the blocks 20a — 20r to be close to the left edge 151, the right edge 15r, the top edge 15t, or the bottom edge 15b of the canvas 15, the processor 150 ignores the block 20a — 20r in the grouping of the blocks 20a — 20r. Also, ones of the blocks 20a — 20r having a different block type 20r from the other ones of the blocks 20a — 20r may also be removed as outliers.

[0073] An outlier embeddings is calculated for the blocks 20a — 20r. This outlier embedding comprises, for example, the block type 20t or an orientation of the block 20a — 20r relative to the canvas 15. The blocks 20a — 20r in the outlier embedding are then clustered using, for example, Density-Based Spatial Clustering of Applications with Noise (DBSCAN).

DBSCAN is a density-based clustering algorithm. DBSCAN analyzes a proximity of points in a data set, marking points as outliers that lie alone in low-density regions. The blocks 20a — 20r having a low proximity to other blocks in the outlier detection embedding are ignored by the processor 150 in the grouping of the blocks 20a — 20r.

[0074] The processor 150 determines in step S20 a plurality of candidate configurations for the plurality of the blocks 20a — 20r on the canvas 15. The candidate configurations are stored as a HashMap data structure comprising items of data on the number of groups and the block type-frequencies. The number of “image-type-blocks” and “non-image-type-

9257310 (v2) SOMARRISON

LU501299 blocks” for the candidate configurations c are stored by the processor 150 as block type- frequency. An example of the HashMap data structure is shown in below equation. {number — groups 4: group — constraints {: block — type — frequencies {: image 2 [: text : paragraph] 3}}

[0075] In the example shown in Fig. 2, the processor 150 identifies the candidate configurations comprising the blocks 20a — 20r. The determining of the plurality of candidate configurations is also done using a the steps described above (see step 160 to 180 in Fig. SA and Fig. 5B).

[0076] The processor 150 determines in step S30 possible groups for the grouping of the blocks 20a — 20r on the canvas 15. The step S30 of determining the number of groups is done by determining a greatest common denominator (GCD) for the plurality of blocks 20a — 20r on the canvas 15. More specifically, the GCD is calculated by comparing the number of “image-type-blocks” to the number of “non-image-type blocks” such as “text-type- blocks”. The GCD is calculated using, for example, algorithm given by the following equation.

GCD = (image — type — block, non — image — type — block)

[0077] The canvas 15 shown in Fig. 2 comprises, for example, eighteen blocks 20a — 20r.

Let us again suppose (as already described above) that the block type 20t of blocks 20a, 20d, 20g, 20j, 20m, 20p is, for example, “image-type-block”. The block type 20t of blocks 20b — 20c, 20e — 20f, 20h — 201, 20k — 201, 20m — 200, 20q — 20r is, for example, “text-type- blocks”. The block type 20t of these blocks 20b — 20c, 20e — 20f, 20h — 201, 20k — 201, 20m — 200, 20q — 20r is, for example, “non-image-type-blocks”. The GCD of the example shown in Fig. 2 is calculated by the processor 150 as follows. There are six “image-type-blocks” and twelve “non-image-type-blocks” on the canvas 15. The GCD 1s calculated using the following equation.

GCD(6,12) = 6

9257310 (v2) SOMARRISON

LU501299

[0078] The determining of the number of groups is further done by calculating the number of possible divisors strictly greater than 1 for the GCD. In the aforementioned example, the divisor 1s 6. From this calculation of the divisor, the processor 150 determines that the number of groups 1s either 2, or 3, or 6.

An exemplary list of possible candidate configurations c in the aforementioned example is shown below.

Number of possible groups | Block type-frequency Block type-frequency

TETE

PRE

[0079] The processor 150 then selects in step S40 one of the candidate configurations from the plurality of the candidate configurations as the configuration. The selecting is based on, for example, the number of groups. The processor 150 selects, for example, the candidate configuration with largest number of groups as configuration. In the aforementioned example, the processor 150 selects as the candidate configuration comprising six groups as the configuration from the first line of the table. The processor 150 then determines in step

S110 of the method 10 the archetype 90 for the blocks 20a — 20r on the canvas 15. This determining is based on the block type-frequency. In the aforementioned example, the block type-frequency is one “image-type-block” and two “text-type-blocks”. The archetype 90 is determined by the processor 150, in this example, to be in reading order “image-type-block”, “text-type-block”, “text-type-block”. The processor 150 continues with step S120 to S200 of the method 10 as described above.

Example 1: Smart Duplication / Structural Editing

[0080] Fig. 7 shows a first example for use of the method 10, 110 for creation of a new group of blocks 20j — 201 in a free space 15f of the canvas 15. This creation of a new group of blocks is also referred to as “smart duplication” or “structural editing”. In this first example, the user has selected the blocks 20a — 201 using the cursor 30 (see step S100 in Fig. SA). The processor 150 can identify the selected blocks 20a — 201 and detect the archetype 90 for the selected blocks 20a — 201 (steps S110 to S120 in Fig. SA). The processor 150 further

9257310 (v2) SOMARRISON

LU501299 determines that the free canvas space 15f is available on the canvas 15. If the user wishes, for example by clicking a command symbol displayed on the display unit 170, to add a new block 20*, the processor 150 determines that the user wishes to create the new group of blocks 20j — 201 in the free canvas space 15f. The processor 150 creates the new grouping of the blocks 20j — 201 based on the detected archetype 90 in the free canvas space 15f.

Example 2: Collection Tidy

[0081] Fig. 8A shows a first view of a second example for use of the method 10, 110 for placement of the blocks 20a — 201 on the canvas 15. In this second example, the user has selected blocks 20a — 201 using the cursor 30 (see step S100 in Fig. SA) for alignment. The processor 150 determines the archetype 90 for the blocks 20a — 201 in the user input (see steps S110 to S120 in Fig. SA). The processor 150 then determines candidate configurations for the blocks 20a — 201. The processor 150 selects one configuration from the candidate configurations. The blocks 20a — 201 of the configuration are then aligned by the processor 150 if the user clicks a command symbol displayed on the display unit 170. Fig. 8B shows a second view of the second example for use of the method 10, 110 for placement of the blocks 20a — 201 on the canvas 15. Fig. 8B shows the placement of the blocks 20a — 201 on the canvas 15 after the alignment of the blocks 20a — 201 by the processor 150.

Example 3: Group Swap

[0082] Fig. 9A shows a first view of a third example for use of the method 10, 110 for placement of groups 40 of blocks 20a — 201 on the canvas 15. In this third example, the placement involves swapping the groups of blocks 40 and the user selects a group 40 of blocks comprising the blocks 20; — 201 using the cursor 30 (see step S100 in Fig. SA). The processor 150 determines the archetype 90 for the selected group 40 of blocks 20j — 201 (see steps S110 to S120 in Fig. SA). The processor 150 then determines the groups 40 for the blocks 20a — 20i on the canvas 15. If the user moves the selected group 40 of blocks 20j — 201 on the canvas 15 over another group 40 of blocks 20a — 201, such as blocks 20g — 201, the processor 150 swaps the blocks 20g — 201 with the selected group 40 of blocks 20j — 201 onthe canvas 15. Fig. 9B shows a second view of a third example for use of the method 10, 110 for placement of groups 40 of blocks 20a — 201 on the canvas 15. Fig. 9B shows the

92573LU (vz) SONNENBERG

HARRISON

LU501299 placement of the blocks 20a — 201 on the canvas 15 after the after the alignment of the blocks 20a — 201 by the processor 150.

9257310 (v2) SOMARRISON

LU501299

Reference numerals 10 method 15 canvas 151 left edge 15r right edge 15t top edge 15b bottom edge 15h horizontal dimension 15v vertical dimension 20a — 20r, 20% block 20t block type 30 cursor 35 search space 60 bounding box 60s scaled bounding box 90 archetype 100 computer 110 method 150 processor 160 memory 170 display unit

Claims

9257310 (v2) SOMARRISON LU501299 Claims

1. À computer-implemented method (10) for grouping of blocks (20a — 20r) on a canvas (15), the method (10) comprising: detecting (S100) a user input, wherein the user input comprises at least one of a plurality of blocks (20a — 20r) selected by a user; determining (S110) an archetype (90) for the plurality of the selected blocks (20a — 20r) of the user input; identifying (S130) blocks (20a — 20r) from the plurality of the selected blocks (20a — 20r) having a same block type (20t) as a first block (20a); analyzing (S140) the identified blocks (20a — 20r) having the same block type (20t) as the first block (20a); comparing a candidate group (S170) to the archetype (90) of the user input; and storing (S180) the candidate group as group.

2. The computer-implemented method (10) according to claim 1, further comprising processing (S120) the archetype (90) for the user input.

3. The computer-implemented method (10) according to claim 2, wherein the processing (S120) of the archetype (90) is done in a reading order.

4. The computer-implemented method (10) according to claim 3, wherein the reading order is determined by the processor (150) based on a Euclidian distance of the blocks (20a — 20r) relative to a fixed position of the canvas (15).

5. The computer-implemented method (10) according to claims 1 or 2, further comprising locating (S150) of a scaled bounding box (60s) over the identified blocks (20a — 201).

9257310 (v2) SOMARRISON LU501299

6. The computer-implemented method (10) according to any of the above claims, further comprising analyzing (S160) the blocks (20a — 20r) within a search space.

7. A computer-implemented method (110) for grouping of blocks (20a — 20r) on a canvas (15), the method (110) comprising: detecting (SS) a plurality of blocks (20a — 20r) on the canvas (15); removing (S10) outliers from the blocks (20a — 20r); determining (S20) candidate configurations for the plurality of blocks (20a — 20r) on the canvas (15); determining (S30) a number of groups for the plurality of the blocks (20a — 20r) on the canvas (15); and selecting (S40) a configuration from the candidate configurations.

8. The computer-implemented method (10) according to claim 7, wherein the removing (S10) of the outliers is based on at least one of a block type (20t) of the blocks (20a — 20r) or a location of the blocks (20a — 20r) relative to the canvas (15).

9. The computer-implemented method (10) according to claim 7 or 8, wherein the determining (S20) of the candidate configurations for the plurality of the blocks (20a — 20r) is done by analyzing the blocks (20a — 20r) within a search space.

10. The computer-implemented method (10) according to according to any one of claims 7 to 9, wherein the determining (S20) of the candidate configurations for the plurality of the blocks (20a — 20r) further comprises comparing a candidate group to an archetype (90).

11. The computer-implemented method (10) according to according to any one of claims 7 to 10, wherein

9287310 (v2) SON ARRON LU501299 the determining (S20) of the candidate configurations for the plurality of the blocks (20a — 20r) further comprises storing the candidate group as group.

12. A computer (100) for grouping of blocks (20) on a canvas (15), the computer (100) comprising: a processor (150) for determining an archetype (90); a memory (160) for storing a group of blocks (20a — 20r), wherein the group of blocks is determined using the archetype (90); and a display unit (170) for displaying a canvas (15) comprising at least one of the blocks (20a — 20r).