WO2022257594A1

WO2022257594A1 - High-precision three-dimensional data real-time progressive rendering method and system

Info

Publication number: WO2022257594A1
Application number: PCT/CN2022/086556
Authority: WO
Inventors: 潘威; 叶景杨; 陈豪; 游锦钊; 卢盛林
Original assignee: 广东奥普特科技股份有限公司
Priority date: 2021-06-10
Filing date: 2022-04-13
Publication date: 2022-12-15
Also published as: CN113269860A; CN113269860B

Abstract

A high-precision three-dimensional data real-time progressive rendering method and system. The method comprises: setting acquisition parameters of three-dimensional data; performing equal-interval sampling on M rows of three-dimensional data acquired each time according to a sampling ratio; initializing a data point array vertex, a valid tag array valid, and a color array color; traversing the acquired three-dimensional data according to rows and columns to generate a data point array vertex, and assigning the valid tag array valid of the three-dimensional data which is an invalid value as false; obtaining the color array color from a linear look-up table (LUT) by means of linear mapping of the three-dimensional data; generating a triangular patch index array according to the valid tag array valid and calculating a normal array; and transmitting the data point array vertex, the color array color, the normal array, and the triangular patch index array into a shader for rendering. By progressively calculating triangular patch indexes and transmitting same into the shader for rendering, the present invention can solve the problems of frame dropping and stuttering during a rendering process, thereby improving rendering efficiency.

Description

A real-time progressive rendering method and system for high-precision three-dimensional data

This application claims the priority of the Chinese patent application submitted to the China Patent Office on June 10, 2021, with the application number 202110651218.3 and the title of the invention "A method and system for real-time progressive rendering of high-precision three-dimensional data", the entire content of which Incorporated in this application by reference.

technical field

The invention relates to the technical field of machine vision, in particular to a method and system for real-time progressive rendering of high-precision three-dimensional data.

Background technique

In recent years, with the rapid development of automation technology, in some dangerous working environments that are not suitable for manual work or where artificial vision is difficult to meet the requirements, machine vision has been introduced and replaced traditional manual inspection methods, greatly improving production efficiency. efficiency and automation. Machine vision is to use computer to simulate human visual function, but it is not just a simple extension of the human eye, but more importantly, it has part of the function of the human brain, that is, it has the ability to extract information from the image of the target (objective thing) and process it. And understand it, and finally use it for practical detection, measurement and control. An important feature of machine vision imaging is to obtain target information from images. Earlier machine vision imaging mainly relied on two-dimensional (2D) machine vision technology: obtain information such as texture, shape, position, size and direction of objects in the target according to the grayscale or grayscale features of pixels in color images, but with As the current "intelligent manufacturing" technology has gradually increased the requirements for machine vision performance, the limitations of two-dimensional machine vision technology have become more and more obvious, so the development and application of three-dimensional (3D) visual imaging technology is urgently needed.

At present, 3D visual imaging technologies mainly include: Time Of Flight (TOF), grating structured light method, binocular stereo vision method and laser triangulation method, etc., which can realize the perception and collection of 3D information of the target. Among them, laser triangulation, although it has been around for decades, is still widely used. Its working principle is to use laser light source, target and camera to define a space triangle, and calculate the three-dimensional data of the target by determining the intersection angle between the three, such as surface height and profile information, with very high resolution and measurement accuracy. However, the only disadvantage is that when using the existing rendering method to process and render the 3D data collected by the laser triangulation method, it not only requires high hardware requirements, but also tends to drop frames and freeze during the rendering process.

Therefore, the prior art still needs to be improved and developed.

The above information is presented as background information only to aid in understanding the present disclosure, and it is not a determination or admission that any of the above might be available as prior art with respect to the present disclosure.

Contents of the invention

The invention provides a high-precision three-dimensional data real-time progressive rendering method and system to solve the deficiencies of the prior art.

To achieve the above object, the present invention provides the following technical solutions:

In the first aspect, an embodiment of the present invention provides a method for real-time progressive rendering of high-precision 3D data, the method comprising:

Set the acquisition parameters of the three-dimensional data, the acquisition parameters include the number of rows row of the acquisition image, the number of acquisition columns col, the resolution xr in the X direction, the resolution yr in the Y direction, the resolution zr in the Z direction, and the sampling rate ratio;

According to the sampling rate, the M rows of three-dimensional data collected each time are sampled at equal intervals to obtain a data array with a size of M*width/ratio, wherein the Width is the width value of the image;

Initialize three arrays whose length is M*width/ratio: data point array vertex, valid mark array valid and color array color;

Traversing the collected 3D data according to the row and column, generating the data point array vertex, and assigning false to the valid tag array valid of the 3D data with invalid values;

Obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data;

Generate a triangular patch index array according to the valid tag array valid;

Calculate the normal array according to the generated triangular patch index array;

The data point array vertex, the color array color, the normal array and the triangle patch index array are passed into the shader for rendering.

Further, in the method for real-time progressive rendering of high-precision three-dimensional data, the step of traversing the collected three-dimensional data according to the row and column, generating the data point array vertex, and assigning false to the valid flag array valid of the three-dimensional data with invalid values include:

Traverse the collected 3D data according to the row and column, and generate the data point array vertex according to the following rules:

For the data data[i][j] of row i and column j, the data points are pixels (i*xr, j*yr, zr*data[i][j]);

Will assign false to the valid marker array valid of the three-dimensional data data[i][j] with invalid values.

Further, in the high-precision three-dimensional data real-time progressive rendering method, the step of obtaining the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data includes:

The mapping lookup table LUT of the color array color is generated by the following formula:

Color(val) = colormap(val, min, max, ColorMapType);

The color array color is obtained in the linear lookup table LUT by linearly mapping the three-dimensional data.

Further, in the method for real-time progressive rendering of high-precision three-dimensional data, the step of generating an index array of triangular facets according to the valid tag array valid includes:

Generate a triangular patch index array according to the valid tag array valid and the following rules:

a) If it is the first time to collect data after initialization, then for the i-th vertex (i==row*width+col), define down=(row+1)*width+col, right=row*width+col +1,downright=(row+1)*width+col+1;

b) If valid[i], valid[down], valid[downright], valid[right] are all true, push the index sequence i, down, downright, down, downright, right of the triangular patch; if only valid[downright ] is false, push the triangular patch index sequence i, down, right; if only valid[down] is false, push the sequence i, downright, right; if only valid[right] is false, push i, down, downright; if Only when valid[i] is false, push down, downright, right; otherwise, do not push;

c) If it is not the first time to collect data after initialization, before performing step a) and step b), follow the rules of step b) for the previous line of scan data to construct and connect the triangular patch sequence.

In the second aspect, the embodiment of the present invention provides a real-time progressive rendering system for high-precision 3D data, and the system includes:

The parameter setting module is used to set the acquisition parameters of the three-dimensional data, and the acquisition parameters include the number of rows row of the acquisition image, the number of acquisition columns col, the resolution xr of the X direction, the resolution yr of the Y direction, and the resolution zr of the Z direction , sampling rate ratio;

The data sampling module is used to sample at equal intervals to the M rows of three-dimensional data collected each time according to the sampling rate, to obtain a data array whose size is M*width/ratio, wherein the Width is the width value of the image;

The array initialization module is used to initialize three arrays whose length is M*width/ratio: data point array vertex, valid mark array valid and color array color;

The first generation module of the array is used for traversing the collected three-dimensional data according to the rows and columns, generating the data point array vertex, and assigning false to the valid marker array valid of the three-dimensional data of invalid values;

The array acquisition module is used to obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data;

The second generation module of the array is used to generate the triangular surface index array according to the effective tag array valid;

An array calculation module, configured to calculate a normal array according to the generated triangular patch index array;

The array rendering module is used to pass the data point array vertex, the color array color, the normal array and the triangle patch index array into the shader for rendering.

Further, in the real-time progressive rendering system for high-precision three-dimensional data, the first generation module of the array is specifically used for:

Further, in the real-time progressive rendering system for high-precision three-dimensional data, the array acquisition module is specifically used for:

Color(val) = colormap(val, min, max, ColorMapType);

The color array color is obtained in the linear lookup table LUT by linearly mapping the 3D data.

Further, in the real-time progressive rendering system for high-precision three-dimensional data, the second array generation module is specifically used for:

Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

The embodiment of the present invention provides a high-precision three-dimensional data real-time progressive rendering method and system. By gradually calculating the index of the triangle patch and passing it into the shader for rendering, it can solve the problem of frame drop and stuck in the rendering process. It solves the pause problem, improves the rendering efficiency, optimizes the rendering performance, and improves the user experience.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings on the premise of not paying creative efforts.

FIG. 1 is a schematic flowchart of a real-time progressive rendering method for high-precision 3D data provided by Embodiment 1 of the present invention;

Fig. 2 is a schematic structural diagram of the acquisition system in Embodiment 1 of the present invention;

Fig. 3 is a schematic diagram of the connection of the triangular patch sequence structure in the first embodiment of the present invention;

FIG. 4 is a schematic diagram of functional modules of a real-time progressive rendering system for high-precision 3D data provided by Embodiment 2 of the present invention.

Detailed ways

In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the following description The embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In the description of the present invention, it is to be understood that when a component is said to be "connected" to another component, it may be directly connected to the other component or there may be an intervening component at the same time. When a component is said to be "set on" another component, it can be set directly on another component or there may be a centered component at the same time.

In addition, the terms "long", "short", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention, rather than indicating or implying the referred device Or that the original must have this particular orientation, operate in a particular orientation configuration, and this should not be construed as a limitation of the invention.

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and through specific implementation methods.

Embodiment one

In view of the defects existing in the above-mentioned existing rendering technology, the applicant, based on his rich practical experience and professional knowledge in this industry for many years, combined with the application of academic theory, actively researched and innovated, hoping to create a solution that can solve the defects in the existing technology technology, making rendering technology more practical. Through continuous research, design, and after repeated trial samples and improvements, the present invention with practical value is finally created.

Please refer to accompanying drawing 1, which is a schematic flow chart of a real-time progressive rendering method for high-precision 3D data provided by Embodiment 1 of the present invention. This method is suitable for scenes of real-time rendering of 3D data. Rendering system, which can be implemented by software and/or hardware, and integrated inside the mobile shopping operation platform. The method specifically includes the following steps:

S101. Set the acquisition parameters of the three-dimensional data. The acquisition parameters include the number of rows of the acquired image row, the number of acquisition columns col, the resolution xr in the X direction, the resolution yr in the Y direction, the resolution zr in the Z direction, and the sampling rate ratio .

It should be noted that the acquisition system for collecting three-dimensional data in this embodiment includes a camera, a laser, and an embedded processor. There is a Sham angle relationship between the measurement plane and the camera lens and photosensitive chip, which can ensure that the laser lines within the measurement range are all It can image clearly and improve the system precision. As shown in Figure 2, the working process is as follows: 1) The laser projects a line of laser light onto the surface of the object to form laser stripes; 2) The laser stripes are projected to the image sensor through the lens; 4) According to the principle of triangulation, convert the image coordinate points of the laser stripe center into physical coordinate points in the camera coordinate system in the embedded processor; 5) smooth and Sampling at equal intervals; 6) smoothing filtering; 7) transmitting to the host through the network port; 8) repeating the above sampling until the end; 9) generating a frame of point cloud data.

S102. Sampling the M rows of three-dimensional data collected each time at equal intervals according to the sampling rate to obtain a data array with a size of M*width/ratio, wherein the Width is a width value of the image.

S103. Initialize three arrays whose length is M*width/ratio: data point array vertex, valid mark array valid and color array color.

S104. Traversing the collected three-dimensional data according to rows and columns, generating a data point array vertex, and assigning a value of false to a valid flag array valid of three-dimensional data with invalid values.

Preferably, said step S104 further includes:

S105. Obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data.

Preferably, said step S105 further includes:

Color(val) = colormap(val, min, max, ColorMapType);

S106. Generate a triangular patch index array according to the valid tag array valid.

Preferably, said step S106 further includes:

c) If it is not the first time to collect data after initialization, before performing step a) and step b), follow the rules of step b) for the previous line of scan data to construct and connect the triangular patch sequence. As shown in the dotted line in Figure 3.

S107. Calculate the normal array according to the generated triangular patch index array.

S108. Pass the data point array vertex, the color array color, the normal array and the triangle patch index array into the shader for rendering.

It should be noted that the model rendering method provided by the present invention can be applied to a shader. Exemplarily, it can be applied to computer animation (CoMP21009942uter Graphics, CG), high-level shader language (High Level Shading Language, HLSL), Open Graphics Library (Open Graphics Library, OpenGL) or a node editor based on a game engine.

The embodiment of the present invention provides a real-time progressive rendering method for high-precision 3D data. By gradually calculating the index of the triangle facet and passing it into the shader for rendering, it can solve the problem of frame drop and freeze in the rendering process. , which improves rendering efficiency, optimizes rendering performance, and improves user experience.

Embodiment two

Please refer to Figure 4, which is a schematic diagram of functional modules of a high-precision three-dimensional data real-time progressive rendering system provided by Embodiment 2 of the present invention. The system specifically includes the following modules:

The parameter setting module 201 is used to set the acquisition parameters of the three-dimensional data, and the acquisition parameters include the number of rows row of the acquisition image, the number of acquisition columns col, the resolution xr in the X direction, the resolution yr in the Y direction, and the resolution in the Z direction zr, sampling rate ratio;

The data sampling module 202 is used to sample the M rows of three-dimensional data collected at equal intervals according to the sampling rate to obtain a data array with a size of M*width/ratio, wherein the Width is the width value of the image;

The array initialization module 203 is used to initialize three arrays whose length is M*width/ratio: data point array vertex, effective marker array valid and color array color;

The first generation module 204 of the array is used for traversing the collected three-dimensional data according to the rows and columns, generating the data point array vertex, and assigning false to the effective tag array valid of the three-dimensional data of invalid value;

The array obtaining module 205 is used to obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data;

The second array generating module 206 is configured to generate a triangular patch index array according to the effective tag array valid;

The array calculation module 207 is used to calculate the normal array according to the generated triangular patch index array generated;

The array rendering module 208 is configured to pass the data point array vertex, the color array color, the normal array and the triangle patch index array into the shader for rendering.

Preferably, the first generation module 204 of the array is specifically used for:

Preferably, the array obtaining module 205 is specifically used for:

Color(val) = colormap(val, min, max, ColorMapType);

Preferably, the second array generation module 206 is specifically used for:

A real-time progressive rendering system for high-precision 3D data provided by the embodiment of the present invention can solve the problem of frame drop and freeze in the rendering process by gradually calculating the index of the triangle surface and passing it into the shader for rendering , which improves rendering efficiency, optimizes rendering performance, and improves user experience.

The above-mentioned system can execute the method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

The description thus far of the foregoing embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and used in a selected embodiment, even if not specifically shown or described. In many respects, the same elements or features may also be varied. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous details are set forth, such as examples of specific parts, devices and methods, in order to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.

The terminology used herein is for the purpose of describing particular example embodiments only and no limitation is intended. As used herein, the singular forms "a" and "the" may be meant to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprising" and "having" are inclusive and thus specify the presence of stated features, integers, steps, operations, elements and/or components, but do not exclude the presence or additional presence of one or more other features , as a whole, steps, operations, elements, components and/or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring performance in the specific order discussed and illustrated, unless an order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be employed.

When an element or layer is referred to as being "on," "bonded to," "connected to," or "coupled to" another element or layer, it can be directly on, or in contact with, another element or layer. An element or layer may be bonded, connected or coupled to another element or layer, and intervening elements or layers may also be present. In contrast, when an element or layer is referred to as being "directly on," "directly engaged with," "directly connected to," or "directly coupled to" another element or layer, there may not be intervening elements or layers in between. Other words used to describe the relationship between elements should be interpreted in a like fashion (eg, "between" versus "directly between," "adjacent" versus "directly adjacent," etc.). As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items. Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections are not constrained by these terms. limit. These terms may be only used to distinguish one element, component, region or section from another element, component, region or section. The use of terms such as the terms "first," "second," and other numerical values herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

Spatial relative terms, such as "inside", "outside", "underneath", "beneath", "below", "above", "upper", etc., may be used herein for convenience of description , to describe the relationship between one element or feature and one or more other elements or features as shown in the figures. Spatially relative terms may be meant to encompass different orientations of the device in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the example term "below" can encompass both an orientation of upward and downward. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and interpreted in terms of the spatially relative descriptions herein.

Claims

A real-time progressive rendering method for high-precision three-dimensional data, characterized in that the method includes:

Set the acquisition parameters of the three-dimensional data, the acquisition parameters include the number of rows row of the acquisition image, the number of acquisition columns col, the resolution xr in the X direction, the resolution yr in the Y direction, the resolution zr in the Z direction, and the sampling rate ratio;

According to the sampling rate, the M rows of three-dimensional data collected each time are sampled at equal intervals to obtain a data array with a size of M*width/ratio, wherein the Width is the width value of the image;

Initialize three arrays whose length is M*width/ratio: data point array vertex, valid mark array valid and color array color;

Traversing the collected 3D data according to the row and column, generating the data point array vertex, and assigning false to the valid tag array valid of the 3D data with invalid values;

Obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data;

Generate a triangular patch index array according to the valid tag array valid;

Calculate the normal array according to the generated triangular patch index array;

The data point array vertex, the color array color, the normal array and the triangle patch index array are passed into the shader for rendering.
The real-time progressive rendering method for high-precision three-dimensional data according to claim 1, wherein the three-dimensional data collected is traversed according to the row and column to generate the data point array vertex, and the effective mark array valid of the three-dimensional data that is an invalid value The steps to assign false include:

Traverse the collected 3D data according to the row and column, and generate the data point array vertex according to the following rules:

For the data data[i][j] of row i and column j, the data points are pixels (i*xr, j*yr, zr*data[i][j]);

Will assign false to the valid marker array valid of the three-dimensional data data[i][j] with invalid values.
The real-time progressive rendering method for high-precision three-dimensional data according to claim 2, wherein the step of obtaining the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data comprises:

The mapping lookup table LUT of the color array color is generated by the following formula:

Color(val) = colormap(val, min, max, ColorMapType);

The color array color is obtained in the linear lookup table LUT by linearly mapping the three-dimensional data.
The real-time progressive rendering method for high-precision three-dimensional data according to claim 1, wherein the step of generating the triangle patch index array according to the valid tag array valid comprises:

Generate a triangular patch index array according to the valid tag array valid and the following rules:

a) If it is the first time to collect data after initialization, then for the i-th vertex (i==row*width+col), define down=(row+1)*width+col, right=row*width+col +1,downright=(row+1)*width+col+1;

b) If valid[i], valid[down], valid[downright], valid[right] are all true, push the index sequence i, down, downright, down, downright, right of the triangular patch; if only valid[downright ] is false, push the triangular patch index sequence i, down, right; if only valid[down] is false, push the sequence i, downright, right; if only valid[right] is false, push i, down, downright; if Only when valid[i] is false, push down, downright, right; otherwise, do not push;

c) If it is not the first time to collect data after initialization, before performing step a) and step b), follow the rules of step b) for the previous line of scan data to construct and connect the triangular patch sequence.
A real-time progressive rendering system for high-precision three-dimensional data, characterized in that the system includes:

The parameter setting module is used to set the acquisition parameters of the three-dimensional data, and the acquisition parameters include the number of rows row of the acquisition image, the number of acquisition columns col, the resolution xr of the X direction, the resolution yr of the Y direction, and the resolution zr of the Z direction , sampling rate ratio;

The data sampling module is used to sample at equal intervals to the M rows of three-dimensional data collected each time according to the sampling rate, to obtain a data array whose size is M*width/ratio, wherein the Width is the width value of the image;

The array initialization module is used to initialize three arrays whose length is M*width/ratio: data point array vertex, valid mark array valid and color array color;

The first generation module of the array is used for traversing the three-dimensional data collected according to the rows and columns, generating the data point array vertex, and assigning false to the effective marker array valid of the three-dimensional data of invalid values;

The array acquisition module is used to obtain the color array color in the linear lookup table LUT by linearly mapping the three-dimensional data;

The second generation module of the array is used to generate the triangular surface index array according to the effective tag array valid;

An array calculation module, configured to calculate a normal array according to the generated triangular patch index array;

The array rendering module is used to pass the data point array vertex, the color array color, the normal array and the triangle patch index array into the shader for rendering.
The real-time progressive rendering system for high-precision three-dimensional data according to claim 5, wherein the first generation module of the array is specifically used for:

Traverse the collected 3D data according to the row and column, and generate the data point array vertex according to the following rules:

For the data data[i][j] of row i and column j, the data points are pixels (i*xr, j*yr, zr*data[i][j]);

Will assign false to the valid marker array valid of the three-dimensional data data[i][j] with invalid values.
The real-time progressive rendering system for high-precision three-dimensional data according to claim 6, wherein the array obtaining module is specifically used for:

The mapping lookup table LUT of the color array color is generated by the following formula:

Color(val) = colormap(val, min, max, ColorMapType);

The color array color is obtained in the linear lookup table LUT by linearly mapping the three-dimensional data.
The real-time progressive rendering system for high-precision three-dimensional data according to claim 5, wherein the second generation module of the array is specifically used for:

Generate a triangular patch index array according to the valid tag array valid and the following rules:

a) If it is the first time to collect data after initialization, then for the i-th vertex (i==row*width+col), define down=(row+1)*width+col, right=row*width+col +1,downright=(row+1)*width+col+1;

b) If valid[i], valid[down], valid[downright], valid[right] are all true, push the index sequence i, down, downright, down, downright, right of the triangular patch; if only valid[downright ] is false, push the triangular patch index sequence i, down, right; if only valid[down] is false, push the sequence i, downright, right; if only valid[right] is false, push i, down, downright; if Only when valid[i] is false, push down, downright, right; otherwise, do not push;

c) If it is not the first time to collect data after initialization, before performing step a) and step b), follow the rules of step b) for the previous line of scan data to construct and connect the triangular patch sequence.