WO2022088544A1

WO2022088544A1 - Following robot multi-target identification system and method based on coded thermal infrared marker

Info

Publication number: WO2022088544A1
Application number: PCT/CN2021/074877
Authority: WO
Inventors: 张慧; 严志国; 马凤英; 赵永国; 肖永飞; 汪尚
Original assignee: 齐鲁工业大学
Priority date: 2020-10-29
Filing date: 2021-02-02
Publication date: 2022-05-05
Also published as: AU2021368390B2; CN112287831B; CN112287831A; AU2021368390A1

Abstract

A following robot multi-target identification system and method based on a coded thermal infrared marker. The system comprises: a thermal radiation mark, which is arranged on a target to be identified, wherein a plurality of heating coils are arranged in the thermal radiation mark, and the heating coils are heated according to different coding rules; a thermal infrared image collection module, which is used for collecting a thermal infrared image of said target and transferring the thermal infrared image to a data processing module; and the data processing module, which is used for identifying a thermal radiation marker image from the thermal infrared image, identifying coding information of the thermal radiation marker according to the thermal radiation marker image, and identifying said target by means of the coding information. A thermal radiation mark worn by a target to be identified is identified, and coding information in the thermal radiation mark is extracted, such that the identification of different targets by a following robot is realized according to the coding information.

Description

Multi-target recognition system and method for following robot based on coded thermal infrared sign

technical field

The invention relates to the technical field of robot target recognition, in particular to a multi-target recognition system and method of a following robot based on a coded thermal infrared mark.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

Target person recognition is the key to realize the task of following the walking robot. However, the existing target person recognition technology is still in the laboratory stage and cannot meet the application requirements of the walking robot in the real environment. In addition, when the number of following robots running in the same scene increases, the targets they follow will also increase accordingly. However, the existing target recognition methods of following robots are often suitable for a single target object, and for robots with multiple target people at the same time. The problem of autonomous identification has not yet achieved a satisfactory solution.

In order to improve the stability of target person recognition and realize the multi-target recognition function, a target recognition device and method for a walking robot based on thermal radiation signs and thermal infrared image recognition are proposed. The target recognition method based on thermal infrared images avoids the traditional color The image recognition method is easily affected by changes in ambient light, which can meet the application requirements in indoor and outdoor environments, and has good environmental adaptability.

The inventor found that thermal imaging technology also has its drawbacks, such as the edge of infrared thermal imaging is relatively blurry, and the features are difficult to extract; it is easily affected by external factors such as the surface characteristics of the object and the wavelength of radiation, the thickness and direction of the navigator's clothes, and the clothes Materials all affect the brightness of an image. Therefore, human detection based on infrared thermal imaging is still a very challenging problem.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present disclosure proposes a multi-target recognition system and method for a follower robot based on a coded thermal infrared mark. Recognition of different targets by the robot.

To achieve the above object, the present disclosure adopts the following technical solutions:

In the first aspect, a multi-target recognition system for following robots based on encoded thermal infrared signs is proposed, including:

a heat radiation mark, used for setting on the target to be identified, a plurality of heating wires are arranged in the heat radiation mark, and the heating wires are heated according to different coding rules;

The thermal infrared image acquisition module is used to collect the thermal infrared image of the target to be identified and transmit it to the data processing module;

The data processing module identifies the thermal radiation sign image from the thermal infrared image, identifies the encoding information of the thermal radiation sign according to the thermal radiation sign image, and identifies the target to be identified through the encoded information.

In the second aspect, a multi-target recognition method for following robots based on encoded thermal infrared signs is proposed, including:

Collect thermal infrared images of the target to be identified;

Identify thermal radiation signature images from thermal infrared images;

Identify the encoded information of the thermal radiation mark according to the thermal radiation mark image;

The target to be identified is identified through the encoded information.

In a third aspect, an electronic device is proposed, comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, the computer instructions, when executed by the processor, complete a following robot based on an encoded thermal infrared sign The steps described in the multi-target recognition method.

In a fourth aspect, a computer-readable storage medium is provided for storing computer instructions that, when executed by a processor, complete the steps described in the method for multi-target recognition of a follower robot based on an encoded thermal infrared mark.

Compared with the prior art, the beneficial effects of the present disclosure are:

1. The present disclosure encodes the heating wire in the thermal radiation sign, so as to have multi-target recognition capability, and realizes the identification of different targets by the following robot according to the encoded information by identifying the encoded information of the thermal radiation sign heated according to the encoding rules.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will become apparent from the description which follows, or may be learned by practice of the invention.

Description of drawings

The accompanying drawings that form a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute improper limitations on the present application.

1 is a schematic structural diagram of Embodiment 1 of the present disclosure;

Fig. 2 is the identification flow chart of Embodiment 1 of the present disclosure;

Fig. 3 is the coding rule of heat radiation mark in the disclosed embodiment 1;

4 is a thermal infrared image collected in Embodiment 1 of the present disclosure;

5 is an image after median filtering of the thermal infrared image according to Embodiment 1 of the present disclosure;

FIG. 6 is an edge detection image extracted by Embodiment 1 of the present disclosure;

FIG. 7 is the vertical edge pixel search rule of Embodiment 1 of the present disclosure;

8 is an image of removing vertical edges from an edge detection image according to Embodiment 1 of the present disclosure;

FIG. 9 is an image after the image of FIG. 8 is expanded three times in Embodiment 1 of the present disclosure;

FIG. 10 is the embodiment of the present disclosure, which corrodes the image of FIG. 9 four times;

FIG. 11 is an image of the area where the heat radiation mark is located according to Embodiment 1 of the present disclosure;

FIG. 12 is a thermal radiation sign image extracted in Embodiment 1 of the present disclosure;

13 is an image of bright and dark stripes identified in Embodiment 1 of the present disclosure;

FIG. 14 is a code recognition image confirmed in Embodiment 1 of the present disclosure.

Among them: 1. Follow the robot, 2. Thermal infrared camera, 3. Thermal radiation sign.

Detailed ways:

The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

In this disclosure, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. The orientation or positional relationship is based on the orientation or positional relationship shown in the drawings, and is only a relational word determined for the convenience of describing the structural relationship of each component or element of the present disclosure, and does not specifically refer to any component or element in the present disclosure, and should not be construed as a reference to the present disclosure. public restrictions.

In the present disclosure, terms such as "fixed connection", "connected", "connected", etc. should be understood in a broad sense, indicating that it may be a fixed connection, an integral connection or a detachable connection; it may be directly connected, or through an intermediate connection. media are indirectly connected. For the relevant scientific research or technical personnel in the field, the specific meanings of the above terms in the present disclosure can be determined according to specific situations, and should not be construed as limitations on the present disclosure.

Example 1

In this embodiment, in order to realize the recognition of multiple targets by the following robot, a multi-target recognition system for a following robot based on a coded thermal infrared mark is disclosed, including:

Further, the heat radiation mark uses binary rules to encode the heating wire.

Further, the encoded information carried by the thermal radiation mark corresponds to the target to be identified one-to-one.

Further, the heating wires in the heat radiation sign are arranged in parallel and at equal intervals.

Further, the specific process for the data processing module to identify the thermal radiation sign image from the thermal infrared image is as follows:

Filter the thermal infrared image;

Extract the edge of the heating wire on the filtered thermal infrared image to obtain the edge detection image;

The vertical edge and discrete edge of the edge detection image are eliminated, and the thermal radiation mark image is extracted from the edge detection image according to the largest area of the thermal radiation mark.

Further, the specific process for the data processing module to identify the encoded information of the thermal radiation mark according to the thermal radiation mark image is as follows:

The heating wire image and the unheated heating wire image in the heat radiation mark image are identified according to the pixel gray value, and the coding information of the heat radiation mark is determined according to the height information of the heated heating wire image and the unheated heating wire image in the vertical direction.

Further, the data processing module determines whether the image of the heated heating wire and the image of the unheated heating wire in the heat radiation sign image satisfy the shape constraint, and when the shape constraint is satisfied, the height in the vertical direction of the image of the heated heating wire and the image of the unheated heating wire is determined according to the shape constraint. The information determines the encoded information of the thermal radiation signature.

The multi-target recognition system of the following robot based on the encoded thermal infrared mark is described in detail with reference to Fig. 1-Fig. 14 .

The multi-target recognition system for following robots based on coded thermal infrared marks, as shown in Figure 1, includes a thermal radiation mark 1 worn by the target to be identified, and a plurality of parallel carbon fiber heating wires are arranged in the thermal radiation mark. Heating is carried out according to different coding rules, and then the thermal infrared camera can obtain gray-scale images with obvious differences in characteristics, and further extract the mutually independent coding information carried by different thermal radiation marks through image processing, so as to realize the identification of different targets.

The heating wire in the heat radiation mark is encoded by binary rules, as shown in Figure 3, the coding rule is explained by taking the heat radiation mark containing 5 heating wires as an example, the top heating wire and the bottom heating wire are used as the heat radiation mark The boundary determination mark is always in the heating state, and the 3 heating wires inside are heated according to the binary coding rule, which has a total of 8 combinations from 0 to 7, that is, if the number is 0, all heating wires will not be heated. When the number is 1, only the bottom heating wire is heated, when the number is 2, only the middle heating wire is heated, ..., when the number is 7, all the three heating wires are heated. This method can realize the multi-feature distinction of the same thermal radiation sign, so it can meet the requirements of robot identification under the condition of multi-target human existence.

Figure 3 shows the thermal infrared images of the targets to be identified, numbered 0, 2, 5, and 7, corresponding to the thermal radiation sign composed of 5 heating wires obtained by the thermal infrared camera when the above coding rules are used. It can be seen from Figure 3 that the heating The rear heating wire is clearly distinguished from the surrounding environment, which proves the feasibility of the identification scheme.

Although the human eye can easily identify the position of the barcode area in the image, if the robot realizes the recognition function, it is necessary to use digital image processing technology to analyze the characteristics of the image step by step. Infrared image for identification. The following only takes the thermal infrared image recognition process of the target to be recognized when the code is 7 as an example to describe the specific principle of the thermal infrared image recognition by the data processing module.

Step 1: Image Filtering

In general, in the process of forming, transmitting, receiving and processing the actual image, there are inevitably external noise interference and internal noise interference, which will affect the quality of the image, make the image blurred, and submerge the features, which brings difficulties to the analysis. . It is a very important process in image preprocessing to remove noise and restore the image from noise-contaminated images.

Here, median filtering is mainly used to remove the salt and pepper noise brought by the optical acquisition system in the barcode image, because when there is strong salt and pepper noise interference in the image, the gray value corresponding to these interference points and the gray value of the adjacent pixels There is a big difference in the degree value, so by sorting and removing the median value, the gray value of these interference points is changed to be similar to the gray value of some adjacent pixels, so as to achieve the effect of removing noise.

Median filtering is a nonlinear smoothing method to reduce edge blur. Its basic idea is as follows: set up a one-dimensional sequence f ₁ , f ₂ , f ₃ ,..., f _n , take the window length as m, for one The median filter is performed on the dimensional sequence, that is, m numbers f _iv ,...,f _i-1 ,f ₁ ,f _i+1 ...,f _i+v are successively extracted from the sequence. The m numbers are sorted according to the size of their values, and the middle number is taken as the filter output, that is:

y _i =median{f _iv ,...,f _i-1 ,f ₁ ,f _i+1 ...,f _i+v }

where i∈Z, v=(m-1)/2.

The effect of median filtering depends on two elements: the spatial extent of the neighborhood and the number of pixels involved in the median calculation. Median filtering can suppress random noise without blurring edges. A 3*3 template is used to perform median filtering on the original image with noise, and the obtained processing result is shown in Figure 5.

Step 2: Edge Detection

For the heated thermal radiation sign image, the gradients on the edge points of the heating wire are consistent, the color reflectance on both sides of the heating wire edge is very different, and the edges are relatively dense. The above features can be processed by extracting the pixels with obvious features and large amount of information in the filtered thermal infrared image by means of edge detection, thereby reducing the amount of calculation and improving the efficiency of the algorithm. Common edge detection algorithms include Prewitts operator, Roberts operator, Soble operator, Canny operator and other classic edge detection operators. Since the heating wire is basically in a horizontal state when moving with the human body, the Prewitts operator is more conducive to thermal radiation. Edge extraction of logos.

The Prewitt operator is a differential operator for image edge detection. Its principle is to use the difference generated by the pixel gray value in a specific area to achieve edge detection. The Prewitt operator uses a 3*3 template to calculate the pixel values in the area, so the edge detection results are more obvious in the horizontal and vertical directions. Prewitt operator is suitable for identifying images with more noise and grayscale gradient. FIG. 6 is an edge detection image after edge extraction is performed by using the Prewitt operator.

Step 3: Eliminate Vertical Edges

The original edge information of the above-mentioned edge detection image contains a lot of edge data irrelevant to the thermal radiation sign, such as the edge information of the target person's body and the edge information of other objects, etc. These edges will introduce unnecessary interference information to the subsequent processing. Use certain filtering operations to eliminate redundant edge data.

It can be seen from Figure 6 that since the edge of the heating wire is approximately horizontally distributed, the following rules are used to delete the obvious vertical edge, and set 1 as the edge pixel and 0 as the background pixel, then the pixel point P(i, j) satisfies the following formula. It is the vertical edge point:

An image description of the formula is shown in Figure 7.

The result after using the above method is shown in Fig. 8. Compared with Fig. 7, it can be seen that the algorithm can effectively eliminate the vertical edge information.

Step 4: Eliminate Discrete Edges

Since the upper and lower edges of the thermal imaging of a single heating wire are relatively close, and the edges of other objects are relatively scattered, the edges of the heating wire will be merged to form an area with a large connected area after the expansion operation is used, and the pixels of the remaining object edges after expansion are basically different. Areas overlap to form connected areas. For this reason, the discrete edges are eliminated by first performing n expansions on Figure 8, and then performing n+1 erosions. Figure 9 shows the results after 3 expansions, and Figure 10 shows the results after 4 erosions.

Step 5: Extract the thermal radiation sign image

As shown in Figure 10, there are multiple connected areas with different sizes, and the area where the thermal radiation mark is located has the largest area. Therefore, the connectivity of image pixels can be analyzed, and the region with the largest connected area in the image can be extracted, which is the region where the thermal radiation sign is located. The result is shown in Figure 11.

Since the connected area shown in Fig. 11 is the result of expanding the edge image 3 times and then eroding it 4 times, expanding once more on the basis of Fig. 11 is the edge area of the thermal radiation mark. 12 shows the thermal radiation sign image.

Step 6: Identification of bright and dark stripes

Calculate the average value of the gray value of the pixels in the heat radiation sign area in the heat radiation sign image, and then regard the pixels whose gray value is higher than the average value as the heated heating wire image, and the remaining pixels are regarded as the unheated heating wire and the background. image, the processing result is shown in Figure 13.

Among them, the bright stripes refer to the image of the heated heating wire, and the dark stripes refer to the image of the unheated heating wire.

Step 7: Image Confirmation of Thermal Radiation Sign

The number of pixels L (L ₁ , L ₂ , ..., L _i ) contained in each bright stripe in the vertical direction and the pixels corresponding to the dark stripes are counted in the column image where the central pixel of the thermal radiation sign area is located in the order from top to bottom. The number D (D ₁ , D ₂ , . . . , D _i ), and the mean value A of the number of pixels included in the bright stripes and the pixel synthesis T of all stripes are calculated.

Since the 5 heating wires are arranged in parallel and at equal intervals, the average value of the pixel heights occupied by the above bright stripes in the vertical direction is proportional to the overall pixel height of the heat radiation sign, so it is considered that the above stripe information satisfies the heat radiation when the following formula is satisfied Logo shape condition constraints.

6*A≤T≤14*A

Step 8: Code Identification

For the thermal radiation sign image that satisfies the shape constraints as shown in Figure 14, the encoding information Num of the current thermal radiation sign can be obtained according to the height information of the bright and dark stripes in the vertical direction and using the following formula.

if

if

if

The complete identification steps are shown in Figure 2.

The data processing module identifies the target to be identified through the encoded information.

This embodiment is aimed at the blurred edge of infrared thermal imaging, which is difficult to extract features, and is easily affected by external factors such as the surface characteristics of the object and the wavelength of radiation, which are faced in the field of target recognition based on thermal infrared images. All materials will affect the inherent problems such as the brightness of the image. The invention provides a multi-target recognition system for following robots based on encoded thermal infrared signs. Differentiate from the background.

The heating wire in the thermal radiation sign is encoded in binary form, so that it has the ability of multi-target recognition. Steps such as edge, extraction of thermal radiation sign images, bright and dark stripe identification, morphological constraints, thermal radiation mark image confirmation, and coding recognition can accurately identify thermal radiation marks and extract their coding information, so as to realize the recognition of the target by the robot.

Example 2

In this embodiment, a multi-target recognition method for following robots based on encoded thermal infrared signs is disclosed, including:

Collect thermal infrared images of the target to be identified;

Identify thermal radiation signature images from thermal infrared images;

Identify the coded information of the heat radiation mark according to the heat radiation mark image;

The target to be identified is identified through the encoded information.

Further, the specific process of identifying the thermal radiation signature from the thermal infrared image is as follows:

Filter the thermal infrared image;

The heating wire edge detection is performed on the filtered thermal infrared image, and the image after edge detection is obtained;

The vertical edge and discrete edge of the image after edge detection are eliminated, and the thermal radiation mark image is extracted from the image after edge detection according to the largest area of the thermal radiation mark.

Further, the specific process of identifying the encoded information of the thermal radiation mark according to the thermal radiation mark image is as follows:

The heating wire curve and the unheated heating wire curve in the heat radiation mark image are identified according to the pixel gray value, and the encoding information of the heat radiation mark is determined according to the height information of the heated heating wire curve and the unheated heating wire curve in the vertical direction.

Example 3

In this embodiment, an electronic device is disclosed, which includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the based on the The steps described in the multi-target recognition method of the following robot with encoded thermal infrared signs.

Example 4

In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions, and when the computer instructions are executed by a processor, the multi-target identification method for a following robot based on an encoded thermal infrared mark disclosed in Embodiment 2 is completed the steps described.

The above are only preferred embodiments of the present application, and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Modifications or equivalent replacements are made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be included within the protection scope of the claims of the present invention.

Claims

The following robot multi-target recognition system based on coded thermal infrared mark is characterized in that, it includes:

The heat radiation mark is arranged on the target to be identified, and a plurality of heating wires are arranged in the heat radiation mark, and the heating wires are heated according to different coding rules;

The thermal infrared image acquisition module is used to collect the thermal infrared image of the target to be identified and transmit it to the data processing module;

The data processing module identifies the thermal radiation sign image from the thermal infrared image, recognizes the encoding information of the thermal radiation sign according to the thermal radiation sign image, and identifies the target to be identified through the encoded information.
The multi-target recognition system for following robots based on coded thermal infrared marks according to claim 1, wherein the thermal radiation marks use binary rules to encode the heating wire.
The multi-target recognition system for following robots based on coded thermal infrared marks as claimed in claim 1, wherein the heating wires in the thermal radiation marks are arranged in parallel and at equal intervals.
The following robot multi-target recognition system based on the coded thermal infrared mark as claimed in claim 1, the specific process that the data processing module identifies the thermal radiation mark image from the thermal infrared image is:

Filter the thermal infrared image;

Extract the edge of the heating wire on the filtered thermal infrared image to obtain the edge detection image;

The vertical edge and discrete edge of the edge detection image are eliminated, and the thermal radiation mark image is extracted from the edge detection image according to the largest area of the thermal radiation mark.
The following robot multi-target recognition system based on coded thermal infrared marks as claimed in claim 1, the specific process that the data processing module recognizes the thermal radiation mark coding information according to the thermal radiation mark image is:

The heating wire image and the unheated heating wire image in the heat radiation mark image are identified according to the pixel gray value, and the encoding information of the heat radiation mark is determined according to the height information of the heated heating wire image and the unheated heating wire image in the vertical direction.
The multi-target recognition system for following robots based on coded thermal infrared marks as claimed in claim 5, wherein the data processing module judges whether the heated heating wire image and the unheated heating wire image in the thermal radiation mark image satisfy the morphological constraint, and when the morphological constraint is satisfied, The encoding information of the heat radiation mark is determined according to the height information of the heated heating wire image and the unheated heating wire image in the vertical direction.
The multi-target recognition method for following robots based on coded thermal infrared signs is characterized in that, comprising:

Collect thermal infrared images of the target to be identified;

Identify thermal radiation signature images from thermal infrared images;

Identify the coded information of the heat radiation mark according to the heat radiation mark image;

The target to be identified is identified through the encoded information.
The multi-target identification method of a following robot based on a coded thermal infrared mark as claimed in claim 7, wherein the specific process of recognizing the encoded information of the thermal radiation mark according to the thermal radiation mark image is:

The heating wire curve and the unheated heating wire curve in the heat radiation mark image are identified according to the pixel gray value, and the encoding information of the heat radiation mark is determined according to the height information of the heated heating wire curve and the unheated heating wire curve in the vertical direction.
An electronic device, comprising a memory, a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are executed by the processor, the encoding-based heat-based coding of any one of claims 7-8 is completed. The steps are described in the multi-target recognition method of the following robot of the infrared mark.
A computer-readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the method for identifying multiple targets of a following robot based on a coded thermal infrared mark according to any one of claims 7-8 is completed. A step of.