CN110070870B - Signal processing method and system of vehicle-mounted system - Google Patents
Signal processing method and system of vehicle-mounted system Download PDFInfo
- Publication number
- CN110070870B CN110070870B CN201910372205.5A CN201910372205A CN110070870B CN 110070870 B CN110070870 B CN 110070870B CN 201910372205 A CN201910372205 A CN 201910372205A CN 110070870 B CN110070870 B CN 110070870B
- Authority
- CN
- China
- Prior art keywords
- control signal
- vehicle
- voice
- signal
- image control
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/223—Execution procedure of a spoken command
Abstract
The present disclosure provides a signal processing method of a vehicle-mounted system, including: identifying a control signal received by the vehicle-mounted system as an image control signal or a voice control signal; when the control signal is recognized to be a voice control signal, detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process; if the fact that the received unprocessed image control signals are stored in the vehicle-mounted system in the voice recording process is detected, whether the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process can be recalled or not is further detected; when it is detected that at least two control signals can be recalled from the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process, one control signal with the maximum confidence coefficient is determined from the at least two control signals which can be recalled to serve as a decision signal. The present disclosure also provides a signal processing system of the vehicle-mounted system.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a signal processing method and system of a vehicle-mounted system.
Background
With the increasing development of science and technology, the vehicle-mounted system is more and more intelligent, and the existing vehicle-mounted system can be intelligently interacted with a user and can respond according to a control signal input by the user.
In the prior art, when an on-board system interacts with a user, the on-board system can receive a voice signal input by the user and can also receive an image signal input by the user, for example, the on-board system receives the voice signal input by the user through a voice input device (microphone), and meanwhile, the on-board system can also collect the image signal, such as a head action, a gesture action, an expression action and the like, made by the user through an image collection device (camera).
Disclosure of Invention
The embodiment of the disclosure provides a signal processing method and a signal processing system of a vehicle-mounted system.
In a first aspect, an embodiment of the present disclosure provides a signal processing method for an on-vehicle system, including:
identifying a control signal received by the vehicle-mounted system as an image control signal or a voice control signal;
when the control signal is recognized to be a voice control signal, detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process;
if the fact that the received unprocessed image control signals are stored in the vehicle-mounted system in the voice recording process is detected, further detecting whether the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process can be recalled;
and when detecting that at least two control signals can be recalled from the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process, determining one control signal with the maximum confidence coefficient from the at least two control signals which can be recalled as a decision signal.
In some embodiments, when the control signal is identified as an image control signal, detecting whether the vehicle-mounted system is in a voice recording state;
if the vehicle-mounted system is detected not to be in a voice recording state, determining to take the image control signal as a decision;
and if the vehicle-mounted system is detected to be in a voice recording state, storing the image control signal, and continuously executing the step of identifying the control signal received by the vehicle-mounted system as the image control signal or the voice control signal when the vehicle-mounted system receives the control signal next time.
In some embodiments, in the step of determining the signal with the highest confidence level from the at least two recallable signals as the decision signal, the confidence level of the image control signal is obtained based on the following formula:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is the confidence of the image control signal, a and b are preset weight parameters, conf _0 is the initial confidence of the image signal, tendAnd t is the ending time of the voice control signal recording process, and t is the occurrence time of the image control signal.
In some embodiments, when it is detected that only one control signal can be recalled from the voice control signal and the image control signals received by the vehicle-mounted system during voice recording, the control signal that can be recalled is determined as the decision signal.
In some embodiments, when it is detected that neither the voice control signal nor each image control signal received by the in-vehicle system during the voice recording process can be recalled, a message of control operation failure is fed back to the user.
In some embodiments, if it is detected that the vehicle-mounted system does not receive the image control signal during the voice recording process, the voice control signal is determined as the decision signal.
In some embodiments, after the step of determining the decision signal, the method further comprises:
and making a decision according to the pre-acquired vehicle machine state information and the decision signal so as to control the vehicle machine to change the state.
In some embodiments, the signal processing method of the in-vehicle system further includes:
and determining the state information of the car machine according to the touch operation of the user.
On the other hand, the embodiment of the present disclosure further provides a signal processing system of a vehicle-mounted system, including:
the recognition module is used for recognizing the control signal received by the vehicle-mounted system as an image control signal or a voice control signal;
the first detection module is used for detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process when the control signal is identified as the voice control signal by the identification module;
the second detection module is used for further detecting whether the voice control signal and each image control signal received by the vehicle-mounted system in the voice recording process can be recalled when the first detection module detects that the vehicle-mounted system receives the image control signal in the voice recording process;
the screening module is used for screening out a control signal with the maximum confidence coefficient from at least two recalled control signals when the second detection module detects that the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process exist at least two control signals which can be recalled;
and the determining module is used for determining the control signal with the maximum confidence coefficient screened by the screening module as a decision signal.
In some embodiments, the signal processing system further comprises: a third detection module and a storage module;
the third detection module is used for detecting whether the vehicle-mounted system is in a voice recording state or not when the recognition module recognizes that the control signal is an image control signal;
the storage module is used for storing the image control signal when the third detection module detects that the vehicle-mounted system is in a voice recording state;
the determining module is further configured to determine the image control signal as a decision signal when the third detecting module detects that the vehicle-mounted system is not in a voice recording state.
In some embodiments, the screening module comprises: an image confidence calculation unit for calculating a confidence of the image control signal based on:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is the confidence of the image control signal, a and b are preset weight parameters, conf _0 is the initial confidence of the image signal, tendIs the end time of the voice control signal recording process, and t is image controlThe time of occurrence of the signal.
In some embodiments, the determining module is further configured to determine that only one control signal can be recalled from among the voice control signals detected by the second detecting module and the image control signals received by the vehicle-mounted system during the voice recording process is a decision signal.
In some embodiments, the signal processing system further comprises:
and the feedback module is used for feeding back information of control operation failure to a user when the second detection module detects that the voice control signal and each image control signal received in the voice recording process cannot be recalled.
In some embodiments, the determining module is further configured to determine the voice control signal as a decision signal when the first detecting module detects that the image control signal is not received by the vehicle-mounted system during the voice recording process.
In some embodiments, the signal processing system of the in-vehicle system further comprises:
and the control module is used for making a decision according to the vehicle state information acquired in advance and the decision signal so as to control the vehicle machine to change the state.
In some embodiments, the signal processing system of the in-vehicle system further comprises:
and the state acquisition module is used for determining the state information of the vehicle according to the touch operation of the user.
In another aspect, an embodiment of the present disclosure further provides an electronic device, including:
one or more processors;
a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the methods provided by the above-described embodiments of the present disclosure.
In a final aspect, the embodiments of the present disclosure further provide a computer-readable medium, on which a computer program is stored, where the program, when executed by a processor, implements the method provided by the above-mentioned embodiments of the present disclosure.
According to the signal processing method and system of the vehicle-mounted system, when the vehicle-mounted system receives image control signals in the voice recording process, the image control signals are stored without being processed, after the voice signals are received, whether the voice control signals and the image control signals can be recalled is judged, and then one control signal with the maximum confidence coefficient is selected from the control signals which can be recalled to serve as a decision signal.
Drawings
The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing in detail exemplary embodiments thereof with reference to the attached drawings, in which:
fig. 1 is a flowchart of a signal processing method of an in-vehicle system according to an embodiment of the present disclosure;
FIG. 2 is a flowchart illustrating another method for processing signals of an onboard system according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a signal processing system of an on-vehicle system according to an embodiment of the present disclosure.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the following describes in detail a signal processing method and system of an on-board system provided by the present invention with reference to the accompanying drawings.
Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but which may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
Embodiments described herein may be described with reference to plan and/or cross-sectional views in light of idealized schematic illustrations of the disclosure. Accordingly, the example illustrations can be modified in accordance with manufacturing techniques and/or tolerances. Accordingly, the embodiments are not limited to the embodiments shown in the drawings, but include modifications of configurations formed based on a manufacturing process. Thus, the regions illustrated in the figures have schematic properties, and the shapes of the regions shown in the figures illustrate specific shapes of regions of elements, but are not intended to be limiting.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
In the prior art, when intelligently interacting with a user, an in-vehicle system may receive a voice control signal input by the user, and may also receive an image control signal, where the voice control signal may be generated by collecting voice of the user with a voice input device, such as a microphone, and the image control signal may be generated by collecting image information related to the user with an image input device, such as a camera, for example, collecting head movements of the user and gestures of the user.
In practical applications, a certain time is required for the recording of the voice signal, when the vehicle-mounted system is in a recording state, if the vehicle-mounted system receives an image signal at the moment, the vehicle-mounted system immediately processes the image signal, and in some cases, the image signal received by the vehicle-mounted system in the recording state is not a signal that a user wants to input into the vehicle-mounted system, for example, when a plurality of passengers are on the vehicle, when one passenger is recording the voice signal to the vehicle-mounted system, and a gesture that another passenger unintentionally makes is also collected by the vehicle-mounted system, which inevitably causes the vehicle-mounted system to immediately process the collected image signal, and the state information of the vehicle-mounted system is correspondingly changed along with the image signal, and when the recording of the voice signal is finished, the vehicle-mounted system processes the voice signal, which results in that the vehicle-mounted system cannot correctly process the voice signal, and further, the interaction between the user and the vehicle-mounted system is wrong, and the user experience is reduced.
The disclosure is a technical solution for how to select a most appropriate control signal from a voice control signal and an image control signal as a decision signal when a vehicle-mounted system processes the voice control signal and the image control signal input by a user in the prior art.
As shown in fig. 1, a signal processing method of an in-vehicle system according to an embodiment of the present disclosure includes:
and S101, identifying that the control signal received by the vehicle-mounted system is an image control signal or a voice control signal.
In the present disclosure, when the in-vehicle system receives the control signal, the control signal is recognized, two different process determinations are performed according to the type of the control signal, and when the control signal is recognized as the voice control signal, step S102 is performed to detect whether the in-vehicle system stores the received but unprocessed image control signal during the recording of the voice.
And step S102, detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process.
When the control signal is recognized to be the voice control signal, the voice input operation is performed by the user, and at the moment, whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice input operation stage needs to be detected so as to perform further screening. Specifically, when it is detected that the vehicle-mounted system stores the received but unprocessed image control signal during the voice recording process, step S103 is executed accordingly to further detect whether the voice control signal and each image control signal received by the vehicle-mounted system during the voice recording process can be recalled.
Step S103, detecting whether the voice control signal and each image control signal received by the vehicle-mounted system can be recalled in the voice recording process.
Whether the control signal can be recalled or not refers to whether the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system or not, namely, when the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system, the control signal can be recalled, and when the user intention represented by the control signal is not matched with the current state information of the vehicle-mounted system, the control signal cannot be recalled; the state information of the in-vehicle system refers to a state where the in-vehicle machine is located, for example, a state of waiting for user confirmation, a state of waiting for a user input area, and the like.
When it is detected that at least two control signals can be recalled from the voice control signal and the image control signals received by the vehicle-mounted system during the voice recording process, step S104 is performed to determine one control signal with the highest confidence level from the at least two control signals that can be recalled as the decision signal.
And step S104, determining the control signal with the highest confidence coefficient from the at least two control signals which can be recalled as a decision signal.
According to the signal processing method of the vehicle-mounted system, when the vehicle-mounted system receives image control signals in the voice recording process, the image control signals are stored without being processed, after the voice signals are received, whether the voice control signals and the image control signals can be recalled is judged, and then one control signal with the maximum confidence coefficient is selected from the control signals which can be recalled to serve as a decision signal.
The embodiment of the present disclosure further provides a signal processing method of a vehicle-mounted system, as shown in fig. 2, the method includes:
step S201, recognizing that the control signal received by the vehicle-mounted system is an image control signal or a voice control signal.
In the present disclosure, when the in-vehicle system receives the control signal, two different processes are executed according to the kind of the control signal.
Specifically, when the control signal is recognized as a voice control signal, steps S202-S207 are executed to detect whether a received but unprocessed image control signal is stored in the vehicle-mounted system during the voice recording process, and then select one control signal from the voice control signal and the image control signal as a decision signal according to a detection result, or directly use the voice control signal as the decision signal; and when the control signal is recognized to be the image control signal, executing the steps S208-S210 to detect whether the vehicle-mounted system is in the voice recording state, and further storing the image control signal according to the detection result or directly taking the image control signal as a decision signal.
Step S202, detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process.
When the control signal is recognized to be a voice control signal, it is indicated that the user performs a voice input operation, and at this time, it needs to be detected whether the vehicle-mounted system stores the received but unprocessed image control signal in the voice input operation stage, so as to perform further screening, and it needs to be indicated that, in the voice recording process, the system may receive one image control signal or a plurality of image control signals.
Specifically, when it is detected that the received but unprocessed image control signal is stored in the vehicle-mounted system during the voice recording process, steps S203-S206 are correspondingly performed to further detect whether the voice control signal and each image control signal received by the vehicle-mounted system during the voice recording process can be recalled; and when detecting that the received unprocessed image control signal is not stored in the vehicle-mounted system during the voice recording process, executing step S207.
Step S203, detecting whether the voice control signal and each image control signal received by the vehicle-mounted system can be recalled in the voice recording process.
Whether the control signal can be recalled or not refers to whether the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system or not, namely, when the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system, the control signal can be recalled, and when the user intention represented by the control signal is not matched with the current state information of the vehicle-mounted system, the control signal cannot be recalled; the state information of the in-vehicle system refers to a state where the in-vehicle machine is located, for example, a state of waiting for user confirmation, a state of waiting for a user input area, and the like.
In practical application, when judging whether the voice control signal can be recalled, firstly, acquiring the current state information of the vehicle-mounted system; secondly, semantic analysis is performed on the voice control signal based on a preset algorithm to obtain the user intention represented by the voice control signal, for example, the voice semantic analysis algorithm is adopted for analysis, and the technology is the prior art in the field and is not described herein any more; and finally, acquiring the matching degree of the user intention represented by the analyzed voice control signal and the current state information of the vehicle-mounted system, if the matching degree exceeds a preset threshold value, determining that the voice control signal is matched with the current state information of the vehicle-mounted system, namely determining that the voice control signal can be recalled, otherwise, determining that the voice control signal cannot be recalled.
When judging whether the image signal can be recalled, firstly, acquiring the current state information of the vehicle-mounted system; secondly, identifying the image control signal based on a preset image identification algorithm to obtain an identification result, matching the identification result with the image control signal stored in a preset image control signal database to determine the user intention represented by the identified image control signal, finally, calculating the matching degree of the user intention represented by the identified image control signal and the current state information of the vehicle-mounted system, if the matching degree exceeds a preset threshold value, determining that the image control signal is matched with the current state information of the vehicle-mounted system, namely determining that the image control signal can be recalled, otherwise, determining that the image control signal cannot be recalled, wherein the method for acquiring the matching degree of the user intention represented by the image control signal and the state information of the vehicle-mounted system is a conventional technical means in the field, and can adopt the method for acquiring the matching degree of the user intention represented by the voice control signal and the state information of the vehicle-mounted system, and will not be described in detail herein.
It should be noted that, in the present disclosure, an image control signal database is preset, a plurality of preset image control signals are stored in the database, each image control signal is provided with a user intention, and the image control signals and the user intentions have a corresponding relationship, which may be set according to the habitual actions of the user in the vehicle in daily life or for safety reasons.
In step S203, when detecting whether the voice control signal and the image control signal can be recalled, it may be detected that only one control signal can be recalled, that a plurality of control signals can be recalled (one voice control signal and at least one image control signal, or two or more image control signals), and that no control signal can be recalled. Specifically, when it is detected that a plurality of (at least two) control signals can be recalled from among the voice control signal and each image control signal received by the in-vehicle system during recording of the voice, step S204 is performed; when detecting that only one control signal can be recalled from the voice control signal and the image control signals received by the vehicle-mounted system in the voice recording process, executing step S205; when it is detected that neither the voice control signal nor each image control signal received by the in-vehicle system during recording of the voice can be recalled, step S206 is performed.
Step S204, one control signal with the maximum confidence coefficient is determined from at least two control signals which can be recalled and is used as a decision signal.
In this disclosure, the calculation of the confidence level of the voice control signal is obtained based on the semantic meaning of the voice control signal detected in step S203, that is, the confidence level of the voice control signal refers to the truth obtained when the semantic meaning is analyzed based on a preset algorithm, which belongs to the prior art and is not described herein again.
In the present disclosure, the confidence of the image control signal is derived based on the following equation:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is a confidence level of the image control signal, conf _0 is an initial confidence level of the image signal, the initial confidence level of the image signal is obtained based on whether the image signal can be recalled in step S203, specifically, when the image control signal is identified based on a preset algorithm, the image control signal is identified as a certain image, the certain identification reality degree is the initial confidence level of the image control signal; t is tendThe method comprises the steps of setting the end time of a voice control signal recording process, setting t as the occurrence time of an image control signal, setting a and b as preset weight parameters, setting a as the weight occupied by the initial confidence coefficient of the image control signal, and setting b as the occurrence time of the image control signal in the recording processIn the formula, the greater the initial confidence of the image signal, and the closer the occurrence time of the image signal to the recording end time, the greater the final confidence of the image signal.
And step S205, determining the only control signal which can be recalled as a decision signal.
And step S206, feeding back information of control operation failure to the user.
In the present disclosure, when none of the control signals can be recalled, a message indicating that the control operation failed is fed back to the user, for example, a word "control signal input failed" may be displayed on the system interface, and "control signal input failed" may be broadcasted by voice.
In the embodiment of the present disclosure, step S203' and step S203 ″ are further included before step S203, and are configured to perform deduplication processing on the image control signals when the number of the image control signals is multiple, so as to improve the processing efficiency of the image control signals.
Step S203' judges the number of image control signals.
Step S203 ″, the deduplication processing is performed on the plurality of image control signals.
In the present disclosure, performing deduplication processing on image control signals is already a conventional technical means in the art, and as one implementation, filtering processing may be performed based on calculating similarity of image control signals.
When the number of image control signals is 2 or more, step S203 ″ is performed to perform a deduplication process on the image control signals.
In step S202, when it is detected that the in-vehicle system does not store the received but unprocessed image control signal during the recording of the voice, step S207 is executed to make a decision according to the voice control signal.
And step S207, determining the voice control signal as a decision signal.
When detecting that the received unprocessed image control signal is not stored in the vehicle-mounted system in the voice recording process, the system takes the voice control signal as a decision signal.
In the above step S201, when the control signal is recognized as the image control signal, steps S208 to S210 are performed.
And step S208, detecting whether the vehicle-mounted system is in a voice recording state.
If the vehicle-mounted system is detected to be in the voice recording state, executing step S209, and when the vehicle-mounted system receives the control signal next time, continuing to execute step S201 to continue to identify whether the control signal received by the vehicle-mounted system is an image control signal or a voice control signal; if the in-vehicle system is not detected to be in the voice recording state, step S210 is executed.
Step S209 stores the image control signal.
And step S210, determining the image control signal as a decision signal.
And when the vehicle-mounted system is detected not to be in the voice recording state, the system takes the image control signal as a decision signal.
According to the signal processing method of the vehicle-mounted system, when the vehicle-mounted system receives image control signals in the voice recording process, the image control signals are stored without being processed, after the voice signals are received, whether the voice control signals and the image control signals can be recalled is judged, and then one control signal with the maximum confidence coefficient is selected from the control signals which can be recalled to serve as a decision signal.
In this disclosure, after the step of determining the decision signal, the method further includes:
and S211, making a decision according to the pre-acquired vehicle machine state information and the decision signal so as to control the vehicle machine to change the state.
In step S211, the state information of the vehicle device is determined according to the touch operation of the user, specifically, after the user performs the touch operation on the vehicle-mounted system, for example, touches or clicks a display screen of the vehicle device, presses a button on a steering wheel, the vehicle device system calls a corresponding interface provided by the background server in response to the corresponding touch operation, and the interface returns to the current touch operation to enable the state information of the vehicle device to be located, and stores the state information; when a user inputs a control signal (a voice control signal or an image control signal), the vehicle-mounted machine system determines a decision signal based on the method for determining the decision signal in the step, and then the vehicle-mounted machine system controls the vehicle-mounted machine to enter the next state according to the stored state information of the vehicle-mounted machine and the determined decision signal.
In the embodiment of the disclosure, the car machine system can perform intelligent interaction with the car machine system by combining the touch operation of the user, the voice input operation of the user and the image input operation of the user, so that the user can realize a more free interaction mode with the car machine system in a vehicle-mounted environment, and the user experience is greatly improved.
On the other hand, the embodiment of the present disclosure provides a signal processing system of an on-board system, which is used to implement the signal processing method of the on-board system provided in the foregoing embodiment, and as shown in fig. 3, the system includes: an identification module 11, a first detection module 12, a second detection module 13, a screening module 14 and a determination module 15.
The recognition module 11 is configured to recognize that the control signal received by the vehicle-mounted system is an image control signal or a voice control signal.
The first detecting module 12 is configured to detect whether the vehicle-mounted system receives the image control signal during the voice recording process when the identifying module 11 identifies that the control signal is the voice control signal.
The second detection module 13 is configured to, when the first detection module 12 detects that the vehicle-mounted system receives the image control signal during the voice recording process, further detect whether the voice control signal and each image control signal received by the vehicle-mounted system during the voice recording process can be recalled.
The screening module 14 is configured to screen one control signal with the highest confidence from the at least two recalled control signals when the second detecting module 13 detects that at least two control signals are recalled from the voice control signal and the image control signals received by the vehicle-mounted system during the voice recording process.
And a determining module 15, configured to determine the control signal with the highest confidence selected by the selecting module 14 as the decision signal.
The signal processing system of the vehicle-mounted system provided by the embodiment of the disclosure stores the image control signal without processing when the vehicle-mounted system receives the image control signal in the voice recording process, judges whether the voice control signal and the image control signal can be recalled after the voice signal is received, and then selects a control signal with the maximum confidence coefficient from the recalled control signals as a decision signal.
Whether the control signal can be recalled or not refers to whether the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system or not, namely, when the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system, the control signal can be recalled, and when the user intention represented by the control signal is not matched with the current state information of the vehicle-mounted system, the control signal cannot be recalled; the state information of the in-vehicle system refers to a state where the in-vehicle machine is located, for example, a state of waiting for user confirmation, a state of waiting for a user input area, and the like.
In practical application, when the second detection module 13 determines whether the voice control signal can be recalled, first, the current state information of the vehicle-mounted system is acquired; secondly, semantic analysis is performed on the voice control signal based on a preset algorithm to obtain the user intention represented by the voice control signal, for example, the voice semantic analysis algorithm is adopted for analysis, and the technology is the prior art in the field and is not described herein any more; and finally, calculating the matching degree of the user intention represented by the analyzed voice control signal and the current state information of the vehicle-mounted system, if the matching degree exceeds a preset threshold value, determining that the voice control signal is matched with the current state information of the vehicle-mounted system, namely determining that the voice control signal can be recalled, otherwise, determining that the voice control signal cannot be recalled. In the present disclosure, it is a conventional technical means in the art to acquire the matching degree of the user intention represented by the voice control signal and the state information of the in-vehicle system, and as an embodiment, the matching degree may be acquired by storing a relationship list of the matching degree of each user intention and the state information of various in-vehicle systems in advance, and then, after acquiring the user intention represented by the voice control signal, by inquiring the relationship list.
When the second detection module 13 judges whether the image signal can be recalled, firstly, the current state information of the vehicle-mounted system is acquired; secondly, identifying the image control signal based on a preset image identification algorithm to obtain an identification result, matching the identification result with the image control signal stored in a preset image control signal database to determine the user intention represented by the identified image control signal, finally, calculating the matching degree of the user intention represented by the identified image control signal and the current state information of the vehicle-mounted system, if the matching degree exceeds a preset threshold value, determining that the image control signal is matched with the current state information of the vehicle-mounted system, namely determining that the image control signal can be recalled, otherwise, determining that the image control signal cannot be recalled. In the present disclosure, obtaining the matching degree between the user intention represented by the image control signal and the state information of the vehicle-mounted system is a conventional technical means in the art, and the above method for obtaining the matching degree between the user intention represented by the voice control signal and the state information of the vehicle-mounted system may be adopted, and is not described herein again.
It should be noted that, in the present disclosure, an image control signal database is preset, a plurality of preset image control signals are stored in the database, each image control signal is provided with a user intention, and the image control signals and the user intentions have a corresponding relationship, which may be set according to the habitual actions of the user in the vehicle in daily life or for safety reasons.
In the present disclosure, the screening module 14 includes a voice confidence calculation unit and an image confidence calculation unit; the voice confidence calculation unit obtains the confidence of the voice control signal based on the semantic meaning of the voice control signal detected by the second detection module 13 when determining the confidence of the voice control signal, that is, the confidence of the voice control signal refers to the truth obtained when the semantic meaning is analyzed based on a preset algorithm, which belongs to the prior art and is not described herein again.
The image confidence calculation unit derives the confidence of the image control signal based on the following formula:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is a confidence level of the image control signal, conf _0 is an initial confidence level of the image signal, and the initial confidence level of the image signal is obtained based on whether the image signal is recalled in step S203, specifically, when the second detection module 13 identifies the image control signal based on the preset algorithm, the image control signal has a certain identification truth when being identified as a certain image, and the identification truth is the initial confidence level of the image control signal; t is tendThe method comprises the steps that the end time of a voice control signal recording process is set, t is the occurrence time of an image control signal, a and b are preset weight parameters, a represents the weight occupied by the initial confidence coefficient of the image control signal, and b represents the weight occupied by the occurrence time of the image control signal in the recording process.
In the present disclosure, when detecting whether the voice control signal and the image control signal can be recalled, the second detecting module 13 may detect that only one control signal can be recalled, may detect that a plurality of control signals can be recalled (one voice control signal and at least one image control signal, or more than two image control signals), and may detect that no control signal can be recalled.
Accordingly, the determining module 15 is further configured to determine that only one control signal can be recalled as the decision signal when the second detecting module 13 detects that the voice control signal and only one control signal among the image control signals received by the vehicle-mounted system during the recording of the voice are recalled.
Further, the signal processing system provided by the embodiment of the present disclosure further includes: and a feedback module 16, configured to feed back, to the user, information of a control operation failure when the second detection module 13 detects that the voice control signal and each image control signal received during the voice recording process cannot be recalled. The feedback module 16 may display the word "control signal input failure" on the system interface, or may report "control signal input failure" through voice.
In some embodiments, the signal processing system provided in the embodiment of the present disclosure further includes a determining module and a duplicate removal module, where the determining module is configured to determine whether the number of received image control signals is greater than or equal to 2 when the first detecting module 12 detects that the vehicle-mounted system receives the image control signals in the process of recording the voice; the duplication eliminating module is used for carrying out duplication eliminating processing on the image control signals when the judging module judges whether the number of the received image control signals is more than or equal to 2 or not so as to improve the processing efficiency of the image control signals.
Further, the signal processing system provided by the embodiment of the present disclosure further includes: a third detection module 17 and a storage module 18.
The third detecting module 17 is configured to detect whether the vehicle-mounted system is in a voice recording state when the identifying module 11 identifies that the control signal is the image control signal.
The storage module 18 is configured to store the image control signal when the third detection module 17 detects that the in-vehicle system is in a voice recording state.
The determining module 15 is further configured to determine an image control signal as a decision signal when the third detecting module 17 detects that the vehicle-mounted system is not in a voice recording state; and is further configured to determine the voice control signal as a decision signal when the first detection module 12 detects that the vehicle-mounted system does not receive the image control signal in the voice recording process.
Further, the system provided by the embodiment of the present disclosure further includes: a control module 19 and a status acquisition module 20; the control module 19 is configured to make a decision according to vehicle state information and a decision signal acquired in advance, so as to control the vehicle machine to change the state; the state obtaining module 20 is configured to determine state information of the vehicle according to a touch operation of a user.
In the present disclosure, the state information of the car machine is determined according to the touch operation of the user, specifically, after the user performs the touch operation on the car-mounted system, for example, touching or clicking a display screen of the car machine, pressing a button on a steering wheel, the car-mounted system calls a corresponding interface provided by the background server in response to the corresponding touch operation, and the interface returns the state information of the car machine located by the current touch operation and stores the state information; when a user inputs a control signal (a voice control signal or an image control signal), the vehicle-mounted machine system determines a decision signal based on the method for determining the decision signal in the step, and then the vehicle-mounted machine system controls the vehicle-mounted machine to enter the next state according to the stored state information of the vehicle-mounted machine and the determined decision signal.
An embodiment of the present disclosure further provides an electronic device, including: one or more processors and a storage device, the storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the methods provided by the above-described embodiments of the present disclosure.
Embodiments of the present disclosure also provide a computer readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method provided by the above-mentioned embodiments of the present disclosure.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purposes of limitation. In some instances, features, characteristics and/or elements described in connection with a particular embodiment may be used alone or in combination with features, characteristics and/or elements described in connection with other embodiments, unless expressly stated otherwise, as would be apparent to one skilled in the art. Accordingly, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as set forth in the appended claims.
Claims (16)
1. A signal processing method of an in-vehicle system includes:
identifying a control signal received by the vehicle-mounted system as an image control signal or a voice control signal;
when the control signal is recognized to be a voice control signal, detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process;
if the fact that the received unprocessed image control signals are stored in the vehicle-mounted system in the voice recording process is detected, further detecting whether the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process can be recalled;
when detecting that at least two control signals can be recalled from the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process, determining one control signal with the maximum confidence coefficient from the at least two control signals which can be recalled as a decision signal;
when the control signal is identified to be an image control signal, detecting whether the vehicle-mounted system is in a voice recording state;
if the vehicle-mounted system is detected not to be in a voice recording state, determining to take the image control signal as a decision;
if the vehicle-mounted system is detected to be in a voice recording state, storing the image control signal, and continuously executing the step of identifying the control signal received by the vehicle-mounted system as the image control signal or the voice control signal when the vehicle-mounted system receives the control signal next time;
the control signal can be recalled when the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system, and the control signal cannot be recalled when the user intention represented by the control signal is not matched with the current state information of the vehicle-mounted system.
2. The signal processing method according to claim 1, wherein in the step of determining, as the decision signal, one of the at least two signals that can be recalled that has the highest confidence level, the confidence level of the image control signal is derived based on the following equation:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is the confidence of the image control signal, a and b are preset weight parameters, conf _0 is the initial confidence of the image signal, tendAnd t is the ending time of the voice control signal recording process, and t is the occurrence time of the image control signal.
3. The signal processing method according to claim 1, wherein when it is detected that only one of the voice control signal and the image control signals received by the in-vehicle system during recording of the voice can be recalled, the control signal that can be recalled is determined as the decision signal.
4. The signal processing method according to claim 1, wherein when it is detected that neither the voice control signal nor each image control signal received by the in-vehicle system during recording of the voice is recalled, information of a control operation failure is fed back to the user.
5. The method of claim 1, wherein if it is detected that the image control signal is not received by the vehicle-mounted system during the voice recording process, determining the voice control signal as the decision signal.
6. The signal processing method according to any one of claims 1 to 5, wherein after the step of determining the decision signal, further comprising:
and making a decision according to the pre-acquired vehicle machine state information and the decision signal so as to control the vehicle machine to change the state.
7. The signal processing method of claim 6, wherein the method further comprises:
and determining the state information of the car machine according to the touch operation of the user.
8. A signal processing system of an in-vehicle system, comprising:
the recognition module is used for recognizing the control signal received by the vehicle-mounted system as an image control signal or a voice control signal;
the first detection module is used for detecting whether the received unprocessed image control signal is stored in the vehicle-mounted system in the voice recording process when the control signal is identified as the voice control signal by the identification module;
the second detection module is used for further detecting whether the voice control signal and each image control signal received by the vehicle-mounted system in the voice recording process can be recalled when the first detection module detects that the vehicle-mounted system receives the image control signal in the voice recording process;
the screening module is used for screening out a control signal with the maximum confidence coefficient from at least two recalled control signals when the second detection module detects that the voice control signals and the image control signals received by the vehicle-mounted system in the voice recording process exist at least two control signals which can be recalled;
the determining module is used for determining the control signal with the maximum confidence coefficient screened by the screening module as a decision signal;
the control signal can be recalled when the user intention represented by the control signal is matched with the current state information of the vehicle-mounted system, and the control signal cannot be recalled when the user intention represented by the control signal is not matched with the current state information of the vehicle-mounted system;
the signal processing system further includes: a third detection module and a storage module;
the third detection module is used for detecting whether the vehicle-mounted system is in a voice recording state or not when the recognition module recognizes that the control signal is an image control signal;
the storage module is used for storing the image control signal when the third detection module detects that the vehicle-mounted system is in a voice recording state;
the determining module is further configured to determine the image control signal as a decision signal when the third detecting module detects that the vehicle-mounted system is not in a voice recording state.
9. The signal processing system of claim 8, wherein the screening module comprises: an image confidence calculation unit for calculating a confidence of the image control signal based on:
conf_R＝a×conf_0+b/(tent-t)
wherein conf _ R is the confidence of the image control signal, a and b are preset weight parameters, conf _0 is the initial confidence of the image signal, tendAnd t is the ending time of the voice control signal recording process, and t is the occurrence time of the image control signal.
10. The signal processing system according to claim 8, wherein the determining module is further configured to determine that only one of the voice control signal and the image control signal received by the vehicle-mounted system during the recording of the voice is recallable as the decision signal when the second detecting module detects that the voice control signal and the image control signal are recallable.
11. The signal processing system of claim 8, wherein the signal processing system further comprises:
and the feedback module is used for feeding back information of control operation failure to a user when the second detection module detects that the voice control signal and each image control signal received in the voice recording process cannot be recalled.
12. The system of claim 8, wherein the determining module is further configured to determine the voice control signal as a decision signal when the first detecting module detects that the image control signal is not received by the vehicle-mounted system during voice recording.
13. The system of any one of claims 8-12, wherein the system further comprises:
and the control module is used for making a decision according to the vehicle state information acquired in advance and the decision signal so as to control the vehicle machine to change the state.
14. The system of claim 13, wherein the system further comprises:
and the state acquisition module is used for determining the state information of the vehicle according to the touch operation of the user.
15. An electronic device, comprising:
one or more processors;
storage means having one or more programs stored thereon which, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
16. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910372205.5A CN110070870B (en) | 2019-05-06 | 2019-05-06 | Signal processing method and system of vehicle-mounted system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910372205.5A CN110070870B (en) | 2019-05-06 | 2019-05-06 | Signal processing method and system of vehicle-mounted system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110070870A CN110070870A (en) | 2019-07-30 |
| CN110070870B true CN110070870B (en) | 2022-02-08 |
Family
ID=67370022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910372205.5A Active CN110070870B (en) | 2019-05-06 | 2019-05-06 | Signal processing method and system of vehicle-mounted system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110070870B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103137128A (en) * | 2011-11-18 | 2013-06-05 | 索尼公司 | Gesture and voice recognition for control of a device |
| CN104123936A (en) * | 2013-04-25 | 2014-10-29 | 伊莱比特汽车公司 | Method for automatic training of a dialogue system, dialogue system, and control device for vehicle |
| CN106125925A (en) * | 2016-06-20 | 2016-11-16 | 华南理工大学 | Method is arrested based on gesture and voice-operated intelligence |
| CN106457563A (en) * | 2014-04-17 | 2017-02-22 | 软银机器人欧洲公司 | Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method |
| CN108099816A (en) * | 2017-12-25 | 2018-06-01 | 芜湖皖江知识产权运营中心有限公司 | A kind of vehicle audio input control method for intelligent travel |
| CN108363557A (en) * | 2018-02-02 | 2018-08-03 | 刘国华 | Man-machine interaction method, device, computer equipment and storage medium |
| CN108665893A (en) * | 2018-03-30 | 2018-10-16 | 斑马网络技术有限公司 | Vehicle-mounted audio response system and method |
| CN108701458A (en) * | 2016-03-01 | 2018-10-23 | 微软技术许可有限责任公司 | speech recognition |
| CN108848012A (en) * | 2018-06-22 | 2018-11-20 | 广州钱柜软件科技有限公司 | A kind of home entertainment device intelligence control system |
| CN109036405A (en) * | 2018-07-27 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | Voice interactive method, device, equipment and storage medium |
-
2019
- 2019-05-06 CN CN201910372205.5A patent/CN110070870B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103137128A (en) * | 2011-11-18 | 2013-06-05 | 索尼公司 | Gesture and voice recognition for control of a device |
| CN104123936A (en) * | 2013-04-25 | 2014-10-29 | 伊莱比特汽车公司 | Method for automatic training of a dialogue system, dialogue system, and control device for vehicle |
| CN106457563A (en) * | 2014-04-17 | 2017-02-22 | 软银机器人欧洲公司 | Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method |
| CN108701458A (en) * | 2016-03-01 | 2018-10-23 | 微软技术许可有限责任公司 | speech recognition |
| CN106125925A (en) * | 2016-06-20 | 2016-11-16 | 华南理工大学 | Method is arrested based on gesture and voice-operated intelligence |
| CN108099816A (en) * | 2017-12-25 | 2018-06-01 | 芜湖皖江知识产权运营中心有限公司 | A kind of vehicle audio input control method for intelligent travel |
| CN108363557A (en) * | 2018-02-02 | 2018-08-03 | 刘国华 | Man-machine interaction method, device, computer equipment and storage medium |
| CN108665893A (en) * | 2018-03-30 | 2018-10-16 | 斑马网络技术有限公司 | Vehicle-mounted audio response system and method |
| CN108848012A (en) * | 2018-06-22 | 2018-11-20 | 广州钱柜软件科技有限公司 | A kind of home entertainment device intelligence control system |
| CN109036405A (en) * | 2018-07-27 | 2018-12-18 | 百度在线网络技术（北京）有限公司 | Voice interactive method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110070870A (en) | 2019-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9330308B2 (en) | Apparatus method and computer-readable medium that detects different regions of user's hand for recognizing gesture for carrying out operation of vehicle | |
| KR101893315B1 (en) | Fingerprint information dynamic updating method and fingerprint recognition apparatus | |
| EP3258417A1 (en) | Method and device for improving fingerprint template and terminal device | |
| JP2017091520A (en) | Methods and apparatuses for adaptively updating enrollment database for user authentication | |
| EP3716194A1 (en) | Method and device for processing vehicle loss assessment data and processing device | |
| CN107422901B (en) | Method and system for preventing mistaken touch of touch screen | |
| CN108595646B (en) | Method and device for commenting online document | |
| CN109686368B (en) | Voice wake-up response processing method and device, electronic equipment and storage medium | |
| CN109828576B (en) | Gesture control method, device, equipment and medium for unmanned distribution robot | |
| CN111737670A (en) | Multi-mode data collaborative man-machine interaction method and system and vehicle-mounted multimedia device | |
| US20190369825A1 (en) | Electronic device and method for providing information related to image to application through input unit | |
| CN115720253A (en) | Video processing method, device, vehicle and storage medium | |
| CN111954868A (en) | Multi-voice assistant control method, device, system and computer readable storage medium | |
| CN105653193A (en) | Searching method and terminal | |
| CN105760084A (en) | Voice input control method and device | |
| CN110070870B (en) | Signal processing method and system of vehicle-mounted system | |
| JP6649329B2 (en) | Vehicle control device and method for recognizing traffic sign | |
| CN110704592B (en) | Statement analysis processing method and device, computer equipment and storage medium | |
| CN115904103A (en) | Control method of terminal touch screen, terminal equipment and storage medium | |
| CN112525556A (en) | Diagnostic method, apparatus, device and storage medium for automatically identifying vehicle brand | |
| CN110580430A (en) | identity input method, device and system | |
| CN110942619A (en) | Vehicle determination method, device and system and electronic equipment | |
| CN113071438B (en) | Control instruction generation method and device, storage medium and electronic equipment | |
| CN117590978B (en) | Vehicle-mounted multimedia integrated management system and method based on artificial intelligence | |
| KR102658832B1 (en) | Method for analyzing vehicle forensic and computing device for executint the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211013 Address after: 100176 101, floor 1, building 1, yard 7, Ruihe West 2nd Road, Beijing Economic and Technological Development Zone, Daxing District, Beijing Applicant after: Apollo Zhilian (Beijing) Technology Co.,Ltd. Address before: 100085 Baidu Building, 10 Shangdi Tenth Street, Haidian District, Beijing Applicant before: BAIDU ONLINE NETWORK TECHNOLOGY (BEIJING) Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |