CN115795278B - Intelligent cloth paving machine control method and device and electronic equipment - Google Patents
Intelligent cloth paving machine control method and device and electronic equipment Download PDFInfo
- Publication number
- CN115795278B CN115795278B CN202211537523.0A CN202211537523A CN115795278B CN 115795278 B CN115795278 B CN 115795278B CN 202211537523 A CN202211537523 A CN 202211537523A CN 115795278 B CN115795278 B CN 115795278B
- Authority
- CN
- China
- Prior art keywords
- attention
- self
- characteristic data
- data
- feature data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000004744 fabric Substances 0.000 title claims abstract description 83
- 238000000034 method Methods 0.000 title claims abstract description 39
- 239000011159 matrix material Substances 0.000 claims abstract description 99
- 230000033001 locomotion Effects 0.000 claims abstract description 79
- 230000007246 mechanism Effects 0.000 claims description 23
- 230000004927 fusion Effects 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 claims description 6
- 238000003892 spreading Methods 0.000 claims description 6
- 230000007480 spreading Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000000875 corresponding effect Effects 0.000 description 42
- 238000010586 diagram Methods 0.000 description 5
- 230000001815 facial effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000009958 sewing Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention provides a control method, a device and electronic equipment of an intelligent cloth paving machine, which realize a series of operations such as completing cloth paving, processing uneven cloth surfaces, suspending cloth paving and the like by artificial voice, greatly improve the production efficiency, reduce the labor intensity and help ensure the product quality; the intelligent cloth paving machine control method comprises the following steps: determining motion characteristic data corresponding to a cloth paving machine based on a motion signal corresponding to the cloth paving machine; determining voice characteristic data corresponding to the cloth paving machine based on voice signals corresponding to operators; determining face attribute image feature data and human key point image feature data corresponding to the spreader based on a face attribute image signal and a human key point image signal corresponding to a human body; and outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data.
Description
Technical Field
The invention relates to the field of intelligent voice control, in particular to a control method and device of an intelligent cloth paving machine and electronic equipment.
Background
The automatic cloth paving machine is also called as a digital intelligent cloth paving machine, and is professional cloth paving equipment with the functions of automatic conveying, cutting, sewing, positioning and the like. The whole machine adopts advanced technologies of transmission, control, measurement and electric elements, adopts a PLC programmable controller and a man-machine interface for configuration, and is assisted with an electronic vacuum disk and a grating ruler for measurement. The automatic cloth paving machine is a product for realizing digitization and intellectualization, and is an integrated high and new technology product.
At present, no voice-controlled cloth paving machine product is used in the market, and the manual operation of a physical button is mainly used. At present, the operation mode of the cloth paving machine is that two workers operate together to perform cloth paving, when the cloth paving machine performs cloth paving, the two workers walk along with the cloth paving machine, hold tools in hands, and process the condition of uneven cloth surfaces at any time. When the cloth cover paved by the cloth paving machine has problems or other conditions need to pause cloth paving, workers need to put down tools in hands and run to a state synchronous with the cloth paving machine, and then the operation is carried out, the working distance of the cloth paving machine can reach 20 meters and the working speed of the cloth paving machine can reach 1.6 meters per second, so that the time is very consumed, and sometimes, the equipment cannot be stopped in time, so that safety accidents can be caused.
Disclosure of Invention
In order to solve the technical problems, the invention provides a control method and device of an intelligent cloth paving machine and electronic equipment, and the specific technical scheme is as follows:
an intelligent cloth paving machine control method comprises the following steps:
determining motion characteristic data corresponding to a cloth paving machine based on a motion signal corresponding to the cloth paving machine;
determining voice characteristic data corresponding to the cloth paving machine based on voice signals corresponding to operators;
determining face attribute image feature data and human key point image feature data corresponding to the spreader based on a face attribute image signal and a human key point image signal corresponding to a human body;
and outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data.
Further, the outputting a control matrix based on the motion feature data, the voice feature data, the face attribute image feature data, and the human body key point image feature data includes:
based on the motion characteristic data and the self-attention mechanism, determining motion self-attention characteristic data F corresponding to the cloth paving machine m ；
Based on the voice characteristic data and the self-attention mechanism, determining voice self-attention characteristic data F corresponding to the cloth paving machine v ；
Based on the face attribute image feature data and the human key point image feature data, determining face attribute self-attention feature data and human key point self-attention feature data corresponding to the spreader;
self-attention based on the motionForce characteristic data F m Speech self-attention feature data F v And outputting a control matrix by the face attribute self-attention characteristic data and the human key point self-attention characteristic data.
Further, the self-attention characteristic data F based on the motion m Speech self-attention feature data F v Face attribute self-attention characteristic data and human key point self-attention characteristic data, output control matrix includes:
determining image fusion characteristic data based on the face attribute self-attention characteristic data and the human key point self-attention characteristic data;
based on the speech self-attention feature data F v And image fusion feature data, determining cross-attention feature data F cross ；
Based on the cross-attention profile data F cross And motion self-attention characteristic data F m Determining multi-headed attention profile data
Based on the multi-headed attention profile dataAnd outputting a control matrix.
Further, the motion feature data, the voice feature data, the face attribute image feature data and the human body key point image feature data are respectively determined based on a motion signal corresponding to the spreader, a voice signal corresponding to an operator, a human face attribute image signal corresponding to a human body and a human body key point image signal by utilizing an encoder and a convolution module.
Further, the multi-head attention-based feature dataAn output control matrix comprising:
multi-head attention feature data based using feature mappersAnd outputting the control matrix data.
Further, the self-attention mechanism gives a Query matrix q, and obtains self-attention characteristics by calculating the attention of the Key matrix k and adding the attention to the Value matrix v, wherein q represents Query, k represents Key, v represents Value, and q and k are subjected to dot multiplication firstly and divided by a scaled k For the vector dimensions of q and k, normalizing the result into probability distribution by softmax, and finally multiplying the probability distribution by a value matrix v to obtain weight summation, wherein the specific calculation formula is as follows:
further, the cross-attention characteristic data F cross By calculating speech self-attention feature data F v The correlation with the features of different scales of the fusion feature data is obtained, namely, a query matrix q is taken as one feature, a key matrix k and a value matrix v are taken as one feature, and the common points of the two features are subjected to cross calculation to obtain a cross attention feature F cross The inputs to the cross-attention mechanism are, in order from top to bottom, a query matrix q, a key matrix k, and a value matrix v.
Further, multi-head attention profile dataFrom speech self-attention feature data F by using multiple parallel queries v And extracting a plurality of groups of different subspaces from the fusion characteristic data to acquire information, and capturing key information of the sequence from multiple aspects, wherein the calculation mode is as follows:
linearly transforming the output S of the feature extraction layer to generate a query vector matrix Q, a key vector matrix K, and a value vector matrix V, wherein W Q ，W K， W V For converting the matrix, the query vector matrix Q, the key vector matrix K and the value vector matrix V are projected to h different subspaces, and the method is specifically as follows:
wherein An ith conversion matrix of Q, K, V, respectively;
by scaling dot product pairs Q i and Ki An inner product operation is carried out, and the inner product operation is normalized by using a softmax function and then is combined with V i Multiplication results in an attention value head of a single head attention mechanism i The attention value is calculated in parallel in h subspaces, and is specifically as follows:
where d is the scaling factor, changing the inner product of Q, K to a standard normal distribution; finally, the attention values of all subspaces are fused:
A(Q、K、V＝Concat(head 1 ,…,head i )W O
wherein ,WO For the transformation matrix, a (Q, K, V is the attention value of the multi-headed attention.
Further, the feature expressionThe final control matrix is calculated by adopting a softmax classifier, and the specific formula is as follows:
wherein ,to output the control matrix.
An intelligent cloth paving machine device, comprising:
the motion characteristic determining module is used for acquiring motion signals corresponding to the cloth paving machine and determining motion characteristic data corresponding to the cloth paving machine;
the voice characteristic determining module is used for acquiring a voice signal corresponding to an operator and determining voice characteristic data corresponding to the cloth paving machine;
the image feature determining module is used for acquiring face attribute image signals and human key point image signals corresponding to a human body and determining face attribute image feature data and human key point image feature data corresponding to the spreading machine;
and the output module is used for outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data.
An electronic device, comprising:
a processor: a memory for storing the processor-executable instructions, wherein the processor is configured to perform the control method of any one of the preceding claims 1 to 9. The invention has the beneficial effects that:
the intelligent cloth paving machine control system based on the voice control algorithm realizes a series of operations of manually finishing cloth paving, processing uneven cloth surfaces, suspending cloth paving and the like by controlling the output control signals of the matrix, greatly improves the production efficiency, reduces the manual labor intensity and is beneficial to ensuring the product quality; the synchronous state with the cloth paving machine is not required to be ensured manually, and the safety of staff is ensured.
Drawings
Fig. 1 is a schematic flow chart of a control method of an intelligent paving machine according to an embodiment of the disclosure.
Fig. 2 is a schematic flow chart of outputting a control matrix based on the motion feature data, the voice feature data and the image feature data according to an embodiment of the disclosure.
FIG. 3 shows the self-attention characteristic data F based on the movement provided by an embodiment of the present disclosure m Speech self-attention feature data F v And outputting a flow diagram of the control matrix by the face attribute self-attention characteristic data and the human key point self-attention characteristic data.
Fig. 4 is a schematic flow chart of determining the motion feature data, the voice feature data, the facial attribute image feature data and the human key point image feature data according to an embodiment of the disclosure.
Fig. 5 is a schematic flow chart of outputting a control matrix based on multi-head attention characteristic data according to an embodiment of the disclosure.
Fig. 6 is a schematic flow chart of a control method of a spreaders according to another embodiment of the disclosure.
Fig. 7 is a schematic structural diagram of a control system of an intelligent cloth paving machine according to an embodiment of the present disclosure.
Detailed Description
The following description of the technical solutions in the embodiments of the present invention will be clear and complete, and it is obvious that the described embodiments are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be configured and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
In the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," "secured," and the like are to be construed broadly, and may be connected, detachably connected, or adult, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements.
The control method of the intelligent cloth paving machine disclosed by the invention is briefly described below with reference to fig. 1 to 4.
As shown in fig. 1, the control method of the intelligent cloth paving machine provided by the embodiment of the disclosure includes the following steps:
determining motion characteristic data corresponding to a cloth paving machine based on a motion signal corresponding to the cloth paving machine;
illustratively, the cloth paving machine comprises a professional cloth paving device with functions of automatic conveying, cutting, sewing, positioning and the like, and the embodiment of the disclosure is not uniformly limited thereto.
The motion signal refers to a motion signal obtained by monitoring the motion state of the cloth paving machine by using a sensing device, such as speed, height position and the like.
The motion signal of the cloth paving machine is subjected to feature extraction to obtain motion feature data corresponding to the cloth paving machine.
Determining voice characteristic data corresponding to the cloth paving machine based on voice signals corresponding to operators;
the voice signal refers to a voice signal which is converted by accommodating an operating voice by utilizing the voice recognition device, and the voice signal comprises control words sent by an operator, such as sixty control words including start, zero setting, forward, cloth feeding opening, cloth pressing opening and the like, so that the working process of each step of cloth paving machine can be accurately controlled, and voice signal mood characteristics are extracted to obtain voice characteristic data.
Determining face attribute image feature data and human key point image feature data corresponding to the spreader based on a face attribute image signal and a human key point image signal corresponding to a human body;
the face attribute image signal and the human body key point image signal refer to image signals obtained by shooting face attributes and human body key points of operators around the spreader by using an image shooting device. The cloth paving machine can prevent other personnel except operators around the cloth paving machine from causing wrong quality and safety accidents caused by too close distance of the operators.
The face attribute image signal and the human key point image signal are subjected to feature extraction respectively to obtain face attribute image feature data and human key point image feature data.
And outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data.
Fig. 2 is a schematic flow chart of outputting a control matrix based on the motion feature data, the voice feature data and the image feature data according to an embodiment of the disclosure. The embodiment shown in fig. 2 is extended from the embodiment shown in fig. 1, and differences between the embodiment shown in fig. 2 and the embodiment shown in fig. 1 are described in the following, and the details of the differences are not repeated.
As shown in fig. 2, in the disclosed embodiment of the present invention, a control matrix is output based on the motion feature data, the voice feature data, and the image feature data, including the following steps.
Based on the motion characteristic data and the self-attention mechanism, determining motion self-attention characteristic data F corresponding to the cloth paving machine m ；
Based on the speech feature data and self-correlationAttention mechanism, determining the voice self-attention characteristic data F corresponding to the cloth paving machine v ；
Based on the face attribute image feature data and the human key point image feature data, determining face attribute self-attention feature data and human key point self-attention feature data corresponding to the spreader;
based on the motion self-attention characteristic data F m Speech self-attention feature data F v And outputting a control matrix by the face attribute self-attention characteristic data and the human key point self-attention characteristic data.
Illustratively, the self-attention mechanism gives a Query matrix q, and the self-attention characteristic attention is obtained by calculating the attention of the Key matrix k and adding the attention to the Value matrix v, wherein q represents Query, k represents Key, v represents Value, and q and k are subjected to point multiplication firstly and divided by a scaled k For the vector dimensions of q and k, normalizing the result into probability distribution by softmax, and finally multiplying the probability distribution by a value matrix v to obtain weight summation, wherein the specific calculation formula is as follows:
due to movement self-attention characteristic data F m Speech self-attention feature data F v The face attribute self-attention characteristic data and the human key point self-attention characteristic data are obtained by further strengthening the motion characteristic data, the voice characteristic data and the image characteristic data respectively by utilizing corresponding self-attention mechanisms, so that the embodiment of the disclosure can combine abundant motion characteristic data, voice characteristic data and image characteristic data to further improve the control freedom.
FIG. 3 shows the self-attention characteristic data F based on the movement provided by an embodiment of the present disclosure m Speech self-attention feature numberAccording to F v And outputting a flow diagram of the control matrix by the face attribute self-attention characteristic data and the human key point self-attention characteristic data. The embodiment shown in fig. 3 is extended from the embodiment shown in fig. 2, and differences between the embodiment shown in fig. 3 and the embodiment shown in fig. 2 are described in the following, and are not repeated.
As shown in fig. 3, in an embodiment of the present disclosure, the self-attention characteristic data F is based on the motion m Speech self-attention feature data F v The face attribute self-attention characteristic data and the human key point self-attention characteristic data are output to a control matrix, and the method comprises the following steps:
determining image fusion characteristic data based on the face attribute self-attention characteristic data and the human key point self-attention characteristic data;
illustratively, the face attribute self-attention characteristic data and the human key point self-attention characteristic data are fused to obtain image fusion characteristic data.
Based on the speech self-attention feature data F v And image fusion feature data, determining cross-attention feature data F cross ；
Illustratively, the speech is self-attention to feature data F v And image fusion characteristic data, adopting a cross attention mechanism to obtain cross attention characteristic data F cross 。
Based on the cross-attention profile data F cross And motion self-attention characteristic data F m Determining multi-headed attention profile data
Cross-attention profile data F cross By calculating speech self-attention feature data F v Interrelation with different scale features of the fused feature data, namely taking a query matrix q as one feature, taking a key matrix k and a value matrix v as one feature, and executing intersection calculation on common points of the two features to obtain an intersection attention feature F cross Cross injectionThe input of the semantic mechanism is a query matrix q, a key matrix k and a value matrix v in sequence from top to bottom.
Illustratively, for cross-attention profile data F cross And motion self-attention characteristic data F m Splicing to obtain multi-head attention characteristic data
Based on the multi-headed attention profile dataAnd outputting a control matrix.
The spreading machine disclosed by the embodiment of the invention also needs to consider the motion state of the spreading machine after receiving the voice command of an operator and whether potential safety hazards are caused to surrounding personnel or not, but the characteristic data fusion and splicing modes disclosed by the embodiment of the invention can fully consider the actual working process, and work under the condition of meeting other conditions after receiving the voice command.
Fig. 4 is a schematic flow chart of determining the motion feature data, the voice feature data, the facial attribute image feature data and the human key point image feature data according to an embodiment of the disclosure. The embodiment shown in fig. 4 is extended from the embodiment shown in fig. 1, and differences between the embodiment shown in fig. 4 and the embodiment shown in fig. 1 are described in detail, so that details of the differences will not be repeated.
As shown in fig. 4, in the embodiment of the present disclosure, the motion feature data, the voice feature data, the face attribute image feature data, and the human key point image feature data are respectively determined based on a motion signal corresponding to the spreader, a voice signal corresponding to the operator, a human face attribute image signal corresponding to the human body, and a human key point image signal using an encoder and a convolution module.
The following further illustrates the specific implementation flow of the intelligent paver control method in conjunction with fig. 6:
fig. 5 is a schematic flow chart of a control method of a spreaders according to another embodiment of the disclosure, where, as shown in fig. 6, the control method includes a spreaders motion signal, a voice signal, a facial attribute image signal, and a human key point image signal, and the spreaders motion signal is input to an encoder and a convolution module to obtain motion feature data; the voice signal is input to an encoder and a convolution module to obtain voice characteristic data; the human face attribute image signal and the human body key point image signal are respectively input into an encoder and a convolution module to respectively obtain human face attribute characteristic data and human body characteristic point characteristic data.
On the basis, the motion characteristic data, the voice characteristic data, the human face attribute characteristic data and the human body characteristic point characteristic data are respectively processed by utilizing a self-attention mechanism to obtain the motion self-attention characteristic data F m Speech self-attention feature data F v The human face attribute self-attention characteristic data and the human key point self-attention characteristic data are fused by a human face attribute self-attention characteristic data and human key point self-attention characteristic data machine to obtain fused characteristic data; the speech self-attention feature data F is then used by the cross-attention mechanism v Splicing the cross attention characteristic data with the fusion characteristic data to obtain cross attention characteristic data F cross The method comprises the steps of carrying out a first treatment on the surface of the Cross attention profile data F cross With exercise self-attention characteristic data F m Fusing to obtain multi-head attention characteristic dataFinally, as shown in FIG. 5, the multi-head attention profile data +.>Input to a feature mapper, which outputs a control matrix.
Illustratively, the multi-headed attention profile data mentioned in the embodiment shown in FIG. 6To utilize moreSub-parallel query from speech self-attention feature data F v And extracting a plurality of groups of different subspaces from the fusion characteristic data to acquire information, and capturing key information of the sequence from multiple aspects, wherein the calculation mode is as follows:
linearly transforming the output S of the feature extraction layer to generate a query vector matrix Q, a key vector matrix K, and a value vector matrix V, wherein W Q ，W K， W V For converting the matrix, the query vector matrix Q, the key vector matrix K and the value vector matrix V are projected to h different subspaces, and the method is specifically as follows:
wherein An ith conversion matrix of Q, K, V, respectively;
by scaling dot product pairs Q i and Ki An inner product operation is carried out, and the inner product operation is normalized by using a softmax function and then is combined with V i Multiplication results in an attention value head of a single head attention mechanism i The attention value is calculated in parallel in h subspaces, and is specifically as follows:
where d is the scaling factor, changing the inner product of Q, K to a standard normal distribution; finally, the attention values of all subspaces are fused:
A(Q、K、V＝Concat(head 1 ,…,head i )W O
wherein ,WO For the transformation matrix, a (Q, K, V is the attention value of the multi-headed attention.
By way of example only, and in an illustrative,multi-headed attention profile data as mentioned in the embodiment shown in FIG. 5The input to the feature mapper, the feature mapper outputs a control matrix, specifically:
the expression of the characteristicsThe final control matrix is calculated by adopting a softmax classifier, and the specific formula is as follows:
wherein ,to output the control matrix.
The intelligent cloth paving machine control method based on voice control can capture and extract the operation instruction sent by an operator in time, reduce the labor intensity and help ensure the product quality; the synchronous state with the cloth paving machine is not required to be ensured manually, and the safety of staff is ensured.
The method embodiments of the present disclosure are described above in detail with reference to fig. 1 to 6, and the apparatus embodiments of the present disclosure will be described below in detail with reference to fig. 7. Furthermore, it should be understood that the description of the method embodiments corresponds to the description of the apparatus embodiments, and thus, reference may be made to the preceding method embodiments for a detailed description.
Fig. 7 is a schematic structural diagram of a control system of an intelligent paving machine according to an embodiment of the present disclosure, and as shown in fig. 7, the schematic structural diagram of the control system of the intelligent paving machine according to the embodiment of the present disclosure includes a motion feature determining module, a voice feature determining module, an image feature determining module and an output module. Specifically, the motion characteristic determining module is used for acquiring a motion signal corresponding to the cloth paving machine and determining motion characteristic data corresponding to the cloth paving machine; the voice characteristic determining module is used for acquiring a voice signal corresponding to an operator and determining voice characteristic data corresponding to the cloth paving machine; the image feature determining module is used for acquiring face attribute image signals and human key point image signals corresponding to the human body and determining face attribute image feature data and human key point image feature data corresponding to the spreading machine; the output module is used for outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data.
In some embodiments, the motion feature determination module is further configured to determine motion self-attention feature data F corresponding to the spreader based on the motion feature data and self-attention mechanism m The method comprises the steps of carrying out a first treatment on the surface of the The voice characteristic determining module is also used for determining voice self-attention characteristic data F corresponding to the cloth paving machine based on the voice characteristic data and the self-attention mechanism v The method comprises the steps of carrying out a first treatment on the surface of the The image feature determining module is further used for determining face attribute self-attention feature data and human key point self-attention feature data corresponding to the spreader based on the face attribute image feature data and the human key point image feature data; the output module is also operable to self-attention characteristic data F based on said movement m Speech self-attention feature data F v And outputting a control matrix by the face attribute self-attention characteristic data and the human key point self-attention characteristic data.
In some embodiments, the output module is further operable to utilize a feature mapper based on multi-headed attention feature dataAnd outputting the control matrix data.
In some embodiments, the invention discloses an electronic device comprising: a processor: a memory for storing the processor-executable instructions.
In some embodiments, during the training phase of the intelligent paver control method: inputting a working state video, a control instruction and a video fragment of an operator and a human face and a human body contained in the fragment of the operator into the cloth spreading machine. Through the encoder, the convolution network and the proposal provided by the invention, the characteristic of strong expressive force of the video is obtained, and finally the mean square error loss is adopted. And (3) after the forward propagation is carried out to obtain the output of each branch, calculating the gradient according to the loss function, carrying out feedback training, and iterating until the model converges.
In some embodiments, at the test stage of the intelligent paver control method: and inputting a working state video and a control command of the cloth laying machine, a video segment of an operator and a human face and a human body contained in the segment, and outputting the video segment and the human face and the human body as a control matrix corresponding to the segment command.
The invention provides an intelligent spreader control system based on a voice control algorithm, which can effectively extract and fuse the attention characteristics of each input branch signal through a designed module, establish the correlation between a plurality of input branch images and voice, automatically extract high-level semantic characteristics highly correlated with a target task and realize accurate voice operation.
The foregoing description of the preferred embodiments of the invention has been presented for purposes of illustration and description, but is not to be construed as limiting the scope of the invention, it being understood that various modifications and changes may be made therein without departing from the spirit of the invention.
Claims (9)
1. The intelligent cloth paving machine control method is characterized by comprising the following steps:
determining motion characteristic data corresponding to a cloth paving machine based on a motion signal corresponding to the cloth paving machine;
determining voice characteristic data corresponding to the cloth paving machine based on voice signals corresponding to operators;
determining face attribute image feature data and human key point image feature data corresponding to the spreader based on a face attribute image signal and a human key point image signal corresponding to a human body;
outputting a control matrix based on the motion feature data, the voice feature data, the face attribute image feature data and the human body key point image feature data;
the outputting control matrix based on the motion feature data, the voice feature data, the face attribute image feature data and the human body key point image feature data comprises the following steps:
based on the motion characteristic data and the self-attention mechanism, determining motion self-attention characteristic data F corresponding to the cloth paving machine m ；
Based on the voice characteristic data and the self-attention mechanism, determining voice self-attention characteristic data F corresponding to the cloth paving machine v ；
Based on the face attribute image feature data and the human key point image feature data, determining face attribute self-attention feature data and human key point self-attention feature data corresponding to the spreader;
based on the motion self-attention characteristic data F m Speech self-attention feature data F v The face attribute self-attention characteristic data and the human key point self-attention characteristic data are output to control matrixes;
the self-attention characteristic data F based on the motion m Speech self-attention feature data F v Face attribute self-attention characteristic data and human key point self-attention characteristic data, output control matrix includes:
determining image fusion characteristic data based on the face attribute self-attention characteristic data and the human key point self-attention characteristic data;
based on the speech self-attention feature data F v And image fusion feature data, determining cross-attention feature data F cross ；
Based on the cross-attention profile data F cross And motion self-attention characteristic data F m Determining multi-headed attention profile data
Based on the multi-headed attention profile dataAnd outputting a control matrix.
2. The method for controlling an intelligent cloth paving machine according to claim 1, wherein,
and respectively determining the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data based on a motion signal corresponding to the spreader, a voice signal corresponding to an operator, a human face attribute image signal corresponding to a human body and a human body key point image signal by utilizing an encoder and a convolution module.
3. The intelligent paving machine control method of claim 1, wherein the multi-head attention profile dataAn output control matrix comprising:
multi-head attention feature data based using feature mappersAnd outputting the control matrix data.
4. The method according to claim 1, wherein the self-attention mechanism gives a Query matrix q, and the self-attention characteristic attention is obtained by calculating the attention of the Key matrix k and adding the obtained attention to the Value matrix v, wherein q represents Query, k represents Key, v represents Value, and q and k are multiplied by a scale, respectivelyd k For the vector dimensions of q and k, normalizing the result into probability distribution by softmax, and finally multiplying the probability distribution by a value matrix v to obtain weight summation, wherein the specific calculation formula is as follows:
5. an intelligent spreader control method as claimed in claim 1, wherein the cross-attention profile data F cross By calculating speech self-attention feature data F v The correlation with the features of different scales of the fusion feature data is obtained, namely, a query matrix q is taken as one feature, a key matrix k and a value matrix v are taken as one feature, and the common points of the two features are subjected to cross calculation to obtain a cross attention feature F cross The inputs to the cross-attention mechanism are, in order from top to bottom, a query matrix q, a key matrix k, and a value matrix v.
6. The intelligent paving machine control method of claim 1, wherein the multi-head attention profile dataFrom speech self-attention feature data F by using multiple parallel queries v And extracting a plurality of groups of different subspaces from the fusion characteristic data to acquire information, and capturing key information of the sequence from multiple aspects, wherein the calculation mode is as follows:
linearly transforming the output S of the feature extraction layer to generate a query vector matrix Q, a key vector matrix K, and a value vector matrix V, wherein W Q ，W K， W V For converting the matrix, the query vector matrix Q, the key vector matrix K and the value vector matrix V are projected to h different subspaces, and the method is specifically as follows:
wherein An ith conversion matrix of Q, K, V, respectively;
by scaling dot product pairs Q i and Ki An inner product operation is carried out, and the inner product operation is normalized by using a softmax function and then is combined with V i Multiplication results in an attention value head of a single head attention mechanism i The attention value is calculated in parallel in h subspaces, and is specifically as follows:
where d is the scaling factor, changing the inner product of Q, K to a standard normal distribution; finally, the attention values of all subspaces are fused:
A(Q、K、V）＝Concat(head 1 ,…,head i )W O
wherein ,WO For the transformation matrix, a (Q, K, V) is the attention value of the multi-headed attention.
7. A control method of an intelligent spreader according to claim 3, wherein the characteristic expression isThe final control matrix is calculated by adopting a softmax classifier, and the specific formula is as follows:
wherein ,to output the control matrix.
8. An intelligent cloth paving machine device, which is characterized by comprising:
the motion characteristic determining module is used for acquiring motion signals corresponding to the cloth paving machine and determining motion characteristic data corresponding to the cloth paving machine;
the voice characteristic determining module is used for acquiring a voice signal corresponding to an operator and determining voice characteristic data corresponding to the cloth paving machine;
the image feature determining module is used for acquiring face attribute image signals and human key point image signals corresponding to a human body and determining face attribute image feature data and human key point image feature data corresponding to the spreading machine;
the output module is used for outputting a control matrix based on the motion characteristic data, the voice characteristic data, the face attribute image characteristic data and the human body key point image characteristic data;
the outputting control matrix based on the motion feature data, the voice feature data, the face attribute image feature data and the human body key point image feature data comprises the following steps:
based on the motion characteristic data and the self-attention mechanism, determining motion self-attention characteristic data F corresponding to the cloth paving machine m ；
Based on the voice characteristic data and the self-attention mechanism, determining voice self-attention characteristic data F corresponding to the cloth paving machine v ；
Based on the face attribute image feature data and the human key point image feature data, determining face attribute self-attention feature data and human key point self-attention feature data corresponding to the spreader;
based on the motion self-attention characteristic data F m Speech self-attention feature data F v The face attribute self-attention characteristic data and the human key point self-attention characteristic data are output to control matrixes;
the self-attention characteristic data F based on the motion m Speech self-attention feature data F v Face attribute self-attention characteristic data and human key point self-attention characteristic data, output control matrix includes:
determining image fusion characteristic data based on the face attribute self-attention characteristic data and the human key point self-attention characteristic data;
based on the speech self-attention feature data F v And image fusion feature data, determining cross-attention feature data F cross ；
Based on the cross-attention profile data F cross And motion self-attention characteristic data F m Determining multi-headed attention profile data
Based on the multi-headed attention profile dataAnd outputting a control matrix.
9. An electronic device, comprising:
a processor: a memory for storing the processor-executable instructions, wherein the processor is configured to perform the control method of any of the preceding claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211537523.0A CN115795278B (en) | 2022-12-02 | 2022-12-02 | Intelligent cloth paving machine control method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211537523.0A CN115795278B (en) | 2022-12-02 | 2022-12-02 | Intelligent cloth paving machine control method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115795278A CN115795278A (en) | 2023-03-14 |
| CN115795278B true CN115795278B (en) | 2023-08-04 |
Family
ID=85444937
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211537523.0A Active CN115795278B (en) | 2022-12-02 | 2022-12-02 | Intelligent cloth paving machine control method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115795278B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017215297A1 (en) * | 2016-06-15 | 2017-12-21 | 深圳光启合众科技有限公司 | Cloud interactive system, multicognitive intelligent robot of same, and cognitive interaction method therefor |
| CN110434859A (en) * | 2019-05-30 | 2019-11-12 | 上海大学 | A kind of the intellect service robot system and its operating method of commercial presence working environment |
| CN113753656A (en) * | 2021-10-13 | 2021-12-07 | 拓卡奔马机电科技有限公司 | Cloth paving machine with identity recognition function and identity recognition method thereof |
| WO2022033556A1 (en) * | 2020-08-14 | 2022-02-17 | 华为技术有限公司 | Electronic device and speech recognition method therefor, and medium |
| CN114644276A (en) * | 2022-04-11 | 2022-06-21 | 伊萨电梯有限公司 | Intelligent elevator control method under mixed scene condition |
| CN114812551A (en) * | 2022-03-09 | 2022-07-29 | 同济大学 | Indoor environment robot navigation natural language instruction generation method |
-
2022
- 2022-12-02 CN CN202211537523.0A patent/CN115795278B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017215297A1 (en) * | 2016-06-15 | 2017-12-21 | 深圳光启合众科技有限公司 | Cloud interactive system, multicognitive intelligent robot of same, and cognitive interaction method therefor |
| CN110434859A (en) * | 2019-05-30 | 2019-11-12 | 上海大学 | A kind of the intellect service robot system and its operating method of commercial presence working environment |
| WO2022033556A1 (en) * | 2020-08-14 | 2022-02-17 | 华为技术有限公司 | Electronic device and speech recognition method therefor, and medium |
| CN113753656A (en) * | 2021-10-13 | 2021-12-07 | 拓卡奔马机电科技有限公司 | Cloth paving machine with identity recognition function and identity recognition method thereof |
| CN114812551A (en) * | 2022-03-09 | 2022-07-29 | 同济大学 | Indoor environment robot navigation natural language instruction generation method |
| CN114644276A (en) * | 2022-04-11 | 2022-06-21 | 伊萨电梯有限公司 | Intelligent elevator control method under mixed scene condition |
Non-Patent Citations (1)
| Title |
|---|
| 人工智能与服装设计的融合模式及其要求;易莉莉;;毛纺科技(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115795278A (en) | 2023-03-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111722714B (en) | AR technology-based digital substation metering operation and detection auxiliary method | |
| CN111508066B (en) | Unordered stacking workpiece grabbing system based on 3D vision and interaction method | |
| CN107139179B (en) | Intelligent service robot and working method | |
| CN109203479A (en) | System and method for advanced increasing material manufacturing | |
| CN106125925B (en) | Intelligence based on gesture and voice control arrests method | |
| CN109203478A (en) | System and method for advanced increasing material manufacturing | |
| CN107239728A (en) | Unmanned plane interactive device and method based on deep learning Attitude estimation | |
| CN109202074A (en) | System and method for advanced increasing material manufacturing | |
| CN109446668A (en) | A kind of component Schedule monitoring system and method based on threedimensional model | |
| CN110443843A (en) | A kind of unsupervised monocular depth estimation method based on generation confrontation network | |
| CN107186708A (en) | Trick servo robot grasping system and method based on deep learning image Segmentation Technology | |
| CN110009614A (en) | Method and apparatus for output information | |
| CN109993734A (en) | Method and apparatus for output information | |
| CN109145759A (en) | Vehicle attribute recognition methods, device, server and storage medium | |
| CN112329117A (en) | Building construction supervision method based on BIM | |
| CN110909898A (en) | Zero-knowledge bank machine room fault diagnosis and maintenance guidance system and method based on AR glasses | |
| CN103530900B (en) | Modeling method, face tracking method and the equipment of three-dimensional face model | |
| CN110161964B (en) | Automatic point-searching machining process for numerical control machine tool | |
| CN109978287A (en) | Polysilicon intelligence production method and system | |
| CN108596148A (en) | A kind of construction worker's labour state analysis system based on computer vision and method | |
| CN109473168A (en) | A kind of medical image robot and its control, medical image recognition methods | |
| CN114742987B (en) | Automatic positioning control method and system for cutting of non-metallic materials | |
| CN115795278B (en) | Intelligent cloth paving machine control method and device and electronic equipment | |
| CN110212551A (en) | Microgrid reactive power autocontrol method based on convolutional neural networks | |
| CN109993108B (en) | Gesture error correction method, system and device under a kind of augmented reality environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |