CN111753552A - Training mode and recognition mode dynamic switching method based on NLP - Google Patents
Training mode and recognition mode dynamic switching method based on NLP Download PDFInfo
- Publication number
- CN111753552A CN111753552A CN202010624878.8A CN202010624878A CN111753552A CN 111753552 A CN111753552 A CN 111753552A CN 202010624878 A CN202010624878 A CN 202010624878A CN 111753552 A CN111753552 A CN 111753552A
- Authority
- CN
- China
- Prior art keywords
- nlp
- request
- service
- training
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
Abstract
The invention relates to a method for dynamically switching a training mode and an identification mode based on NLP, which comprises the following steps: s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type; s2: judging the request type, if the request type is a semantic recognition request, entering a recognition mode, selecting an idle or recognized NLP service, outputting a semantic recognition result and returning the recognition result, and if no idle or recognized NLP service exists, continuously trying to select the NLP service until the request is overtime; s3: and judging the request type, if the request is a request of the corpus publishing type, entering a training model, adding the corpus into a publishing queue, and directly returning a result. The invention can display whether the NLP service currently provides an identification mode or a training mode, and switches the flow according to the state of the current NLP service, thereby solving the problem of inaccurate semantic identification caused by the fact that an identification request is mistakenly input into the NLP service of the training mode.
Description
Technical Field
The invention relates to the field of data processing, in particular to a training mode and recognition mode dynamic switching method based on NLP.
Background
NLP is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that enable efficient communication between humans and computers using natural language. Natural language processing is a subject integrating linguistics, computer science and mathematics. NLP consists of two main areas of technology: natural language understanding and natural language generation.
The natural language understanding direction is mainly aimed at helping a machine to better understand human language, and comprises semantic understanding of basic lexical, syntax and the like and high-level understanding of requirements, sections and emotional levels.
Natural language generation direction, the main goal is to help the machine generate languages that people can understand, such as text generation, automatic abstractions, etc.
The NLP technology is based on technologies and resources such as big data, knowledge maps, machine learning, linguistics and the like, can form a specific application system of machine translation, deep question answering and dialogue systems, and further serves various actual businesses and products
At present, most of intention and semantic recognition adopt NLP service clusters, while NLP service occupies a large amount of cpu resources in the training mode process, so that no idle resources are available for processing semantic recognition requests, but if the entry of the semantic recognition requests is not closed, the semantic recognition results cannot be processed if recognition requests enter, so that the recognition rate is greatly reduced;
when the NLP service is processing the recognition request, if the linguistic data are issued to the NLP service, the request of processing the semantic recognition is directly interrupted, the NLP service is triggered to carry out model training, and the training is not carried out after the recognition request is processed, so that the recognition rate is reduced
When the problem occurs at present, because the NLP service has no elegant switching mode, the overall recognition rate is interfered, the recognition rate is unstable and suddenly high and suddenly low, and meanwhile, the false image of model recognition inaccuracy obtained by NLP service training is also caused.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method for dynamically switching between a training mode and an identification mode based on NLP, which can display whether the current NLP service provides the identification mode or the training mode, and switch traffic according to the current state of the NLP service, thereby solving the problem of inaccurate semantic identification due to the fact that an identification request is mistakenly entered into the NLP service of the training mode.
The technical scheme of the invention is as follows:
a training mode and recognition mode dynamic switching method based on NLP comprises the following steps:
s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type;
s2: judging the request type, if the request type is a semantic recognition request, entering a recognition mode, selecting an idle or recognized NLP service, outputting a semantic recognition result and returning the recognition result, and if no idle or recognized NLP service exists, continuously trying to select the NLP service until the request is overtime;
s3: and judging the request type, if the request is a request of the corpus publishing type, entering a training model, adding the corpus into a publishing queue, and directly returning a result.
Preferably, the NLP service includes 4 states, specifically: idle, meaning that any type of request can be accepted; in the synchronization, the service is in the data synchronization, and the service is in a locked state at the moment and does not accept any new request; in the training, the service is represented to be in corpus training, and the service is in a locked state at the moment and does not accept any new request; in recognition, the service is indicating that it is providing the functionality of semantic recognition, and can accept requests of the type of semantic recognition.
Preferably, the step S3 asynchronously performs the following processes: the method comprises the steps that corpora are added into a release queue, a corpus task processor periodically checks a task queue, if data exist and idle NLP service exists, corpus data are pushed into the NLP service to conduct model training, meanwhile, the state of the NLP service is actively reported, after model training is finished, data of a training result are automatically pushed into a training result buffer area, the data are sequentially synchronized into the NLP service until all NLP service completes data synchronization of a latest model, and the training result buffer area is emptied.
Preferably, the NLP synchronization process specifically includes: if the NLP service in the idle state exists, starting to synchronize a single NLP service and report the state, reporting the state after the single NLP service is synchronized, then judging whether all the NLP services are synchronously completed, if so, emptying a training result buffer area, otherwise, starting the next synchronization process; if no NLP service is available, the next synchronization process is started.
Preferably, the NLP service registers the current state in the NLP gateway, and actively synchronizes the self state to the NLP gateway every 60 seconds.
The invention has the beneficial effects that: the method has the advantages that whether the NLP service provides an identification mode or a training mode currently is displayed in real time, the NLP gateway can switch flow according to the state of the current NLP service, and the problem that semantic identification is inaccurate due to the fact that an identification request is mistakenly input into the NLP service of the training mode is solved; the problem that in the switching process, the request for semantic recognition is forcibly interrupted, and an error result is returned, so that the semantic recognition is inaccurate is solved; the NLP gateway can be used for adjusting the flow control of the recognition and training requests, ensuring the flow of the voice recognition requests and improving the resource utilization rate of the machine.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be further described with reference to the accompanying drawings.
The invention provides a method for dynamically switching a training mode and an identification mode based on NLP (non-line-of-sight), which comprises the steps of firstly registering the current state into an NLP gateway by an NLP service, and actively synchronizing the state of the NLP into the NLP gateway every 60 s.
The NLP service has four states:
idle, meaning that any type of request can be accepted;
in the synchronization, the service is in the data synchronization, and the service is in a locked state at the moment and does not accept any new request;
in the training, the service is represented to be in corpus training, and the service is in a locked state at the moment and does not accept any new request;
in recognition, the service is indicating that it is providing the functionality of semantic recognition, and can accept requests of the type of semantic recognition.
The NLP gateway has the functions of requesting flow control, requesting type routing, service state synchronization and service heartbeat detection, the NLP gateway can be used for adjusting the request flow control of recognition and training, when the NLP gateway is busy, 90% of NLP service nodes are switched into a recognition mode, the voice recognition request flow is guaranteed, when the NLP gateway is idle, the training mode is started, the NLP training function is provided, and the resource utilization rate of a machine is improved.
As shown in fig. 1, the specific implementation flow of the present invention is as follows:
when NLP requests pass through the NLP gateway, the gateway routes through the request types, at the moment, a request of a semantic recognition type is input, NLP service in idle or recognition is automatically selected, if no NLP service in idle or recognition exists at the moment, NLP service selection is continuously tried until the request is overtime, if NPL service is selected, voice recognition is started, meanwhile, the state of the NLP service can be actively reported to the NLP gateway, a recognition mode at the moment is displayed, and a result is returned after recognition is finished.
When a request (namely a training request) of a corpus publishing type is input, firstly, corpora are added into a publishing queue, a result is directly returned, then a corpus task queue processor continuously checks a task queue every few seconds, if data exist and an idle NLP service exists, the corpus data are pushed into the NLP service to carry out model training, meanwhile, the state of the NLP service is actively reported, namely, a training mode is entered, the situation that the semantic recognition request enters the NLP service at the moment to cause that the semantic recognition cannot be processed and an error result is returned is avoided, after model training is finished, the corpus data are automatically pushed into a training result buffer area, if the idle NLP service is found, the corpus data are actively and sequentially synchronized to the NLP service until all NLP services finish data synchronization of a latest model, and then the training result buffer area is emptied.
The training request and the identification request are not input into the NPL service at the same time, and when any type of request is input, the NPL gateway determines whether to execute the request according to the real-time state of the NPL service.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the present invention in its spirit and scope. Are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (5)
1. A training mode and recognition mode dynamic switching method based on NLP is characterized by comprising the following steps:
s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type;
s2: judging the request type, if the request type is a semantic recognition request, entering a recognition mode, selecting an idle or recognized NLP service, outputting a semantic recognition result and returning the recognition result, and if no idle or recognized NLP service exists, continuously trying to select the NLP service until the request is overtime;
s3: and judging the request type, if the request is a request of the corpus publishing type, entering a training model, adding the corpus into a publishing queue, and directly returning a result.
2. The method according to claim 1, wherein the NLP service includes 4 states, specifically: idle, meaning that any type of request can be accepted; in the synchronization, the service is in the data synchronization, and the service is in a locked state at the moment and does not accept any new request; in the training, the service is represented to be in corpus training, and the service is in a locked state at the moment and does not accept any new request; in recognition, the service is indicating that it is providing the functionality of semantic recognition, and can accept requests of the type of semantic recognition.
3. The method of dynamically switching between NLP-based training mode and recognition mode according to claim 1, wherein said step S3 asynchronously performs the following processes: the method comprises the steps that corpora are added into a release queue, a corpus task processor periodically checks a task queue, if data exist and idle NLP service exists, corpus data are pushed into the NLP service to conduct model training, meanwhile, the state of the NLP service is actively reported, after model training is finished, data of a training result are automatically pushed into a training result buffer area, the data are sequentially synchronized into the NLP service until all NLP service completes data synchronization of a latest model, and the training result buffer area is emptied.
4. The method according to claim 3, wherein the NLP synchronization process specifically comprises: if the NLP service in the idle state exists, starting to synchronize a single NLP service and report the state, reporting the state after the single NLP service is synchronized, then judging whether all the NLP services are synchronously completed, if so, emptying a training result buffer area, otherwise, starting the next synchronization process; if no NLP service is available, the next synchronization process is started.
5. The method of claim 2, wherein the NLP service registers a current state in the NLP gateway, and actively synchronizes its own state into the NLP gateway every 60 seconds.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624878.8A CN111753552B (en) | 2020-07-02 | NLP-based training mode and recognition mode dynamic switching method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624878.8A CN111753552B (en) | 2020-07-02 | NLP-based training mode and recognition mode dynamic switching method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111753552A true CN111753552A (en) | 2020-10-09 |
| CN111753552B CN111753552B (en) | 2024-04-19 |
Family
ID=
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1726488A (en) * | 2002-05-07 | 2006-01-25 | 国际商业机器公司 | Integrated development tool for building a natural language understanding application |
| US20190213354A1 (en) * | 2018-01-09 | 2019-07-11 | Accenture Global Solutions Limited | Automated secure identification of personal information |
| WO2019153522A1 (en) * | 2018-02-09 | 2019-08-15 | 卫盈联信息技术（深圳）有限公司 | Intelligent interaction method, electronic device, and storage medium |
| CN110176228A (en) * | 2019-05-29 | 2019-08-27 | 广州伟宏智能科技有限公司 | A kind of small corpus audio recognition method and system |
| US20190286697A1 (en) * | 2018-03-15 | 2019-09-19 | International Business Machines Corporation | Unsupervised on-the-fly named entity resolution in dynamic corpora |
| CN110427627A (en) * | 2019-08-02 | 2019-11-08 | 北京百度网讯科技有限公司 | Task processing method and device based on semantic expressiveness model |
| CN110688488A (en) * | 2018-06-20 | 2020-01-14 | 南京网感至察信息科技有限公司 | Universal knowledge graph cloud service system |
| CN110717017A (en) * | 2019-10-17 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Method for processing corpus |
| CN110795529A (en) * | 2019-09-05 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Model management method, model management device, storage medium and electronic equipment |
| CN110851519A (en) * | 2019-11-18 | 2020-02-28 | 上海新炬网络信息技术股份有限公司 | Method for processing data through ETL tool based on NLP natural language |
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1726488A (en) * | 2002-05-07 | 2006-01-25 | 国际商业机器公司 | Integrated development tool for building a natural language understanding application |
| US20190213354A1 (en) * | 2018-01-09 | 2019-07-11 | Accenture Global Solutions Limited | Automated secure identification of personal information |
| WO2019153522A1 (en) * | 2018-02-09 | 2019-08-15 | 卫盈联信息技术（深圳）有限公司 | Intelligent interaction method, electronic device, and storage medium |
| US20190286697A1 (en) * | 2018-03-15 | 2019-09-19 | International Business Machines Corporation | Unsupervised on-the-fly named entity resolution in dynamic corpora |
| CN110688488A (en) * | 2018-06-20 | 2020-01-14 | 南京网感至察信息科技有限公司 | Universal knowledge graph cloud service system |
| CN110176228A (en) * | 2019-05-29 | 2019-08-27 | 广州伟宏智能科技有限公司 | A kind of small corpus audio recognition method and system |
| CN110427627A (en) * | 2019-08-02 | 2019-11-08 | 北京百度网讯科技有限公司 | Task processing method and device based on semantic expressiveness model |
| CN110795529A (en) * | 2019-09-05 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Model management method, model management device, storage medium and electronic equipment |
| CN110717017A (en) * | 2019-10-17 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Method for processing corpus |
| CN110851519A (en) * | 2019-11-18 | 2020-02-28 | 上海新炬网络信息技术股份有限公司 | Method for processing data through ETL tool based on NLP natural language |
Non-Patent Citations (2)
| Title |
|---|
| PETER IZSAK: ""Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models"", 《2019 FIFTH WORKSHOP ON ENERGY EFFICIENT MACHINE LEARNING AND COGNITIVE COMPUTING - NEURIPS EDITION》, 31 December 2019 (2019-12-31) * |
| 唐黎哲: ""以LDA为例的大规模分布式机器学习系统分析"", 《计算机应用》, 10 March 2017 (2017-03-10) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11393476B2 (en) | Automatically determining language for speech recognition of spoken utterance received via an automated assistant interface | |
| KR102535338B1 (en) | Speaker diarization using speaker embedding(s) and trained generative model | |
| US11493992B2 (en) | Invoking automated assistant function(s) based on detected gesture and gaze | |
| US20210193146A1 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| US11295092B2 (en) | Automatic post-editing model for neural machine translation | |
| US11347801B2 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| CN114207710A (en) | Detecting and/or registering a thermal command to trigger a response action by an automated assistant | |
| US11200893B2 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| KR20210008521A (en) | Dynamic and/or context-specific hot words to invoke automated assistants | |
| KR102508338B1 (en) | Determining whether to automatically resume the first automated assistant session when the second session interrupts | |
| US11392213B2 (en) | Selective detection of visual cues for automated assistants | |
| US11567980B2 (en) | Determining responsive content for a compound query based on a set of generated sub-queries | |
| CN111753552B (en) | NLP-based training mode and recognition mode dynamic switching method | |
| CN111753552A (en) | Training mode and recognition mode dynamic switching method based on NLP | |
| US20200004768A1 (en) | Method for processing language information and electronic device therefor | |
| JP2022544428A (en) | Search item rewriting method, device, device and storage medium | |
| CN109255131A (en) | Interpretation method, device, terminal and storage medium | |
| US20230343336A1 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| US20220293109A1 (en) | Device arbitration for local execution of automatic speech recognition | |
| CN117909557A (en) | Man-machine interaction method, system, equipment and storage medium based on large language model | |
| JP2021521497A (en) | Adaptation of automated assistants based on detected mouth movements and / or gaze |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |