CN111753552B - NLP-based training mode and recognition mode dynamic switching method - Google Patents
NLP-based training mode and recognition mode dynamic switching method Download PDFInfo
- Publication number
- CN111753552B CN111753552B CN202010624878.8A CN202010624878A CN111753552B CN 111753552 B CN111753552 B CN 111753552B CN 202010624878 A CN202010624878 A CN 202010624878A CN 111753552 B CN111753552 B CN 111753552B
- Authority
- CN
- China
- Prior art keywords
- nlp
- request
- recognition
- service
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 54
- 238000000034 method Methods 0.000 title claims abstract description 26
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 238000003058 natural language processing Methods 0.000 description 70
- 230000006870 function Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
Abstract
The invention relates to a training mode and recognition mode dynamic switching method based on NLP, which comprises the following steps: s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type; s2: judging the request type, if the request is a semantic recognition request, entering a recognition mode, selecting the NLP service in idle or in recognition, outputting a semantic recognition result and returning a recognition result, and if the NLP service in idle or in recognition is not available, continuing to try to select the NLP service until the request is overtime; s3: judging the request type, if the request is a request of the corpus publishing type, entering a training model, adding the corpus into a publishing queue, and directly returning the result. The invention can display whether the NLP service currently provides the recognition mode or the training mode, and switch the flow according to the state of the current NLP service, thereby solving the problem of inaccurate semantic recognition caused by the NLP service of which the recognition request is mistakenly input into the training mode.
Description
Technical Field
The invention relates to the field of data processing, in particular to a training mode and recognition mode dynamic switching method based on NLP.
Background
NLP is an important direction in the fields of computer science and artificial intelligence. It is studying various theories and methods that enable effective communication between a person and a computer in natural language. Natural language processing is a discipline that integrates linguistics, computer science, and mathematics. NLP is composed of two main technical areas: natural language understanding and natural language generation.
The main objective of natural language understanding direction is to help the machine better understand the language of people, including semantic understanding of basic lexical and syntactic, and high-level understanding of requirements, chapters and emotion levels.
The direction of natural language generation is mainly aimed at helping a machine generate a language which can be understood by a person, such as text generation, automatic digest and the like.
NLP technology is based on big data, knowledge graph, machine learning, linguistics and other technologies and resources, and can form a specific application system of machine translation, deep question-answering and dialogue system, thereby serving various actual services and products
At present, most intentions and semantic recognition adopt NLP service clusters, and the NLP service occupies a large amount of CPU resources in the training mode process, so that no idle resources are needed to process the semantic recognition request, but if the entry of the semantic recognition request is not closed, the semantic recognition result cannot be processed if the recognition request enters, and the recognition rate is greatly reduced;
When NLP service is processing recognition request, if corpus is issued to NLP service, the request processing semantic recognition is directly interrupted, NLP service is triggered to perform model training, and training is not performed after recognition request processing is finished, so that recognition rate is reduced
When a problem occurs at present, since the NLP service does not have an elegant switching mode, the overall recognition rate is disturbed, so that the recognition rate is unstable and suddenly low, and meanwhile, the false impression of inaccurate recognition of a model obtained by NLP service training is also caused.
Disclosure of Invention
In order to solve the problems, the invention provides a method for dynamically switching between a training mode and an identification mode based on NLP, which can display whether the NLP service currently provides the identification mode or the training mode, and switch the flow according to the state of the current NLP service, thereby solving the problem of inaccurate semantic identification caused by the fact that the identification request is mistakenly input into the NLP service of the training mode.
The technical scheme of the invention is as follows:
A method for dynamically switching training modes and recognition modes based on NLP, comprising the following steps:
s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type;
S2: judging the request type, if the request is a semantic recognition request, entering a recognition mode, selecting the NLP service in idle or in recognition, outputting a semantic recognition result and returning a recognition result, and if the NLP service in idle or in recognition is not available, continuing to try to select the NLP service until the request is overtime;
s3: judging the request type, if the request is a request of the corpus publishing type, entering a training model, adding the corpus into a publishing queue, and directly returning the result.
Preferably, the NLP service includes 4 states, specifically: in idle, it means any type of request can be accepted; in the synchronization, the service is in a locked state and does not accept any new request when the service is in data synchronization; in training, the service is shown to be in corpus training, and is in a locked state at the moment and does not accept any new request; in recognition, the function of semantic recognition is provided by the representation service, and the request of the semantic recognition type can be accepted.
Preferably, the step S3 asynchronously performs the following procedures: and adding the corpus into a release queue, periodically checking the task queue by a corpus task processor, pushing the corpus data into the NLP service to perform model training when the corpus task processor has data and has idle NLP service, actively reporting the state of the NLP service, automatically pushing the data of the training result to a training result buffer area after the model training is finished, and sequentially synchronizing the data into the NLP service until all the NLP service completes the data synchronization of the latest model, and emptying the training result buffer area.
Preferably, the synchronization process of the NLP specifically includes: if the idle NLP service exists, starting to synchronize a single NLP service and reporting the state, reporting the state after the single NLP service is synchronized, judging whether all NLP services are synchronized, if so, emptying a training result buffer area, otherwise, starting the next synchronization process; if no NLP service exists in the idle state, the next synchronization process is started.
Preferably, the NLP service registers the current state into the NLP gateway, and actively synchronizes the state of the NLP service into the NLP gateway every 60 seconds.
The beneficial effects of the invention are as follows: the method comprises the steps that whether an NLP service currently provides an identification mode or a training mode is displayed in real time, and the NLP gateway can switch flow according to the state of the current NLP service, so that the problem of inaccurate semantic identification caused by the fact that an identification request is mistakenly input into the NLP service of the training mode is solved; the problem of inaccurate semantic recognition caused by the fact that a request for semantic recognition is forcedly interrupted and an error result is returned in the switching process is solved; the NLP gateway can be used for adjusting the request flow control of recognition and training, ensuring the voice recognition request flow and improving the resource utilization rate of the machine.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the invention.
Detailed Description
Embodiments of the present invention will be further described below with reference to the accompanying drawings.
The invention provides a training mode and recognition mode dynamic switching method based on NLP, firstly NLP service registers the current state into NLP gateway, and actively synchronizes the state of NLP into NLP gateway every 60 s.
Among them, NLP services have four states:
in idle, it means any type of request can be accepted;
In the synchronization, the service is in a locked state and does not accept any new request when the service is in data synchronization;
In training, the service is shown to be in corpus training, and is in a locked state at the moment and does not accept any new request;
in recognition, the function of semantic recognition is provided by the representation service, and the request of the semantic recognition type can be accepted.
The NLP gateway has functions of request flow control, request type routing, service state synchronization and service heartbeat detection, the NLP gateway can be used for adjusting the request flow control for recognition and training, 90% NLP service nodes are switched into a recognition mode when busy, voice recognition request flow is guaranteed, a training mode is started when idle, NLP training functions are provided, and the resource utilization rate of the machine is improved.
As shown in fig. 1, the specific implementation flow of the present invention is:
When an NLP request passes through an NLP gateway, the gateway carries out routing through a request type, at the moment, a request of a semantic recognition type is input, NLP service in idle or in recognition is automatically selected, if no NLP service in idle or in recognition at the moment is available, the NLP service continues to be tried to be selected until the request is overtime, if the NLP service is selected, voice recognition is started, meanwhile, the state of the NLP service is actively reported to the NLP gateway, the NLP service is displayed as a recognition mode at the moment, and a result is returned after the recognition is finished.
When a request of corpus release type (namely training request) is input, firstly, corpus is added into a release queue, a result is directly returned, then a corpus task queue processor continuously checks a task queue every few seconds, if data exists and idle NLP services exist, corpus data is pushed into the NLP services for model training, meanwhile, the state of the NLP services is actively reported, namely, a training mode is entered, the situation that semantic recognition cannot be processed due to the fact that a request of semantic recognition enters the NLP services at the moment is avoided, an error result is returned, after model training is finished, the corpus is automatically pushed into a training result buffer area, if the idle NLP services are found, the corpus task queue processor actively and sequentially synchronizes to the NLP services until all the NLP services complete data synchronization of the latest model, and then the training result buffer area is emptied.
The training request and the identification request are not simultaneously input into the NPL service, and when any type of request is input, the NPL gateway decides whether to execute the request according to the real-time state of the NPL service.
Finally, it should be noted that: the above examples are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention, but it should be understood by those skilled in the art that the present invention is not limited thereto, and that the present invention is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the corresponding technical solutions. Are intended to be encompassed within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (5)
1.A method for dynamically switching between a training mode and an identification mode based on NLP, comprising the steps of:
s1: when the NLP request passes through the NLP gateway, routing is carried out according to the request type;
S2: judging the request type, if the request is a semantic recognition request, entering a recognition mode, selecting the NLP service in idle or in recognition, outputting a semantic recognition result and returning a recognition result, and if the NLP service in idle or in recognition is not available, continuing to try to select the NLP service until the request is overtime;
S3: judging the request type, entering a training model if the request type is a corpus publishing type request and NLP service is idle, adding the corpus into a publishing queue, and directly returning a result.
2. The method for dynamically switching between training mode and recognition mode based on NLP according to claim 1, wherein the NLP service comprises 4 states, specifically: in idle, it means any type of request can be accepted; in the synchronization, the service is in a locked state and does not accept any new request when the service is in data synchronization; in training, the service is shown to be in corpus training, and is in a locked state at the moment and does not accept any new request; in recognition, the function of semantic recognition is provided by the representation service, and the request of the semantic recognition type can be accepted.
3. The method for dynamically switching between the training mode and the recognition mode based on the NLP according to claim 1, wherein the step S3 asynchronously performs the following procedures: and adding the corpus into a release queue, periodically checking the task queue by a corpus task processor, pushing the corpus data into the NLP service to perform model training when the corpus task processor has data and has idle NLP service, actively reporting the state of the NLP service, automatically pushing the data of the training result to a training result buffer area after the model training is finished, and sequentially synchronizing the data into the NLP service until all the NLP service completes the data synchronization of the latest model, and emptying the training result buffer area.
4. A method for dynamically switching between a training mode and an identification mode based on NLP according to claim 3, wherein the synchronization process of NLP is specifically: if the idle NLP service exists, starting to synchronize a single NLP service and reporting the state, reporting the state after the single NLP service is synchronized, judging whether all NLP services are synchronized, if so, emptying a training result buffer area, otherwise, starting the next synchronization process; if no NLP service exists in the idle state, the next synchronization process is started.
5. The method for dynamically switching between training mode and recognition mode based on NLP according to claim 2, wherein the NLP service registers the current state into the NLP gateway, and the NLP service actively synchronizes the own state into the NLP gateway every 60 seconds.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624878.8A CN111753552B (en) | 2020-07-02 | 2020-07-02 | NLP-based training mode and recognition mode dynamic switching method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010624878.8A CN111753552B (en) | 2020-07-02 | 2020-07-02 | NLP-based training mode and recognition mode dynamic switching method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111753552A CN111753552A (en) | 2020-10-09 |
| CN111753552B true CN111753552B (en) | 2024-04-19 |
Family
ID=72678548
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010624878.8A Active CN111753552B (en) | 2020-07-02 | 2020-07-02 | NLP-based training mode and recognition mode dynamic switching method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111753552B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1726488A (en) * | 2002-05-07 | 2006-01-25 | 国际商业机器公司 | Integrated development tool for building a natural language understanding application |
| WO2019153522A1 (en) * | 2018-02-09 | 2019-08-15 | 卫盈联信息技术（深圳）有限公司 | Intelligent interaction method, electronic device, and storage medium |
| CN110176228A (en) * | 2019-05-29 | 2019-08-27 | 广州伟宏智能科技有限公司 | A kind of small corpus audio recognition method and system |
| CN110427627A (en) * | 2019-08-02 | 2019-11-08 | 北京百度网讯科技有限公司 | Task processing method and device based on semantic expressiveness model |
| CN110688488A (en) * | 2018-06-20 | 2020-01-14 | 南京网感至察信息科技有限公司 | Universal knowledge graph cloud service system |
| CN110717017A (en) * | 2019-10-17 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Method for processing corpus |
| CN110795529A (en) * | 2019-09-05 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Model management method, model management device, storage medium and electronic equipment |
| CN110851519A (en) * | 2019-11-18 | 2020-02-28 | 上海新炬网络信息技术股份有限公司 | Method for processing data through ETL tool based on NLP natural language |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10521608B2 (en) * | 2018-01-09 | 2019-12-31 | Accenture Global Solutions Limited | Automated secure identification of personal information |
| US11194967B2 (en) * | 2018-03-15 | 2021-12-07 | International Business Machines Corporation | Unsupervised on-the-fly named entity resolution in dynamic corpora |
-
2020
- 2020-07-02 CN CN202010624878.8A patent/CN111753552B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1726488A (en) * | 2002-05-07 | 2006-01-25 | 国际商业机器公司 | Integrated development tool for building a natural language understanding application |
| WO2019153522A1 (en) * | 2018-02-09 | 2019-08-15 | 卫盈联信息技术（深圳）有限公司 | Intelligent interaction method, electronic device, and storage medium |
| CN110688488A (en) * | 2018-06-20 | 2020-01-14 | 南京网感至察信息科技有限公司 | Universal knowledge graph cloud service system |
| CN110176228A (en) * | 2019-05-29 | 2019-08-27 | 广州伟宏智能科技有限公司 | A kind of small corpus audio recognition method and system |
| CN110427627A (en) * | 2019-08-02 | 2019-11-08 | 北京百度网讯科技有限公司 | Task processing method and device based on semantic expressiveness model |
| CN110795529A (en) * | 2019-09-05 | 2020-02-14 | 腾讯科技（深圳）有限公司 | Model management method, model management device, storage medium and electronic equipment |
| CN110717017A (en) * | 2019-10-17 | 2020-01-21 | 腾讯科技（深圳）有限公司 | Method for processing corpus |
| CN110851519A (en) * | 2019-11-18 | 2020-02-28 | 上海新炬网络信息技术股份有限公司 | Method for processing data through ETL tool based on NLP natural language |
Non-Patent Citations (2)
| Title |
|---|
| "Training Compact Models for Low Resource Entity Tagging using Pre-trained Language Models";Peter Izsak;《2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition》;20191231;全文 * |
| "以LDA为例的大规模分布式机器学习系统分析";唐黎哲;《计算机应用》;20170310;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111753552A (en) | 2020-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7064018B2 (en) | Automated assistant dealing with multiple age groups and / or vocabulary levels | |
| KR102461920B1 (en) | Automated assistants with conference capabilities | |
| US11735182B2 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| Reitter et al. | Alignment and task success in spoken dialogue | |
| CN110770694B (en) | Obtaining response information from multiple corpora | |
| US11295092B2 (en) | Automatic post-editing model for neural machine translation | |
| US11347801B2 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| KR20190099061A (en) | Context-aware human-to-computer conversation | |
| US11200893B2 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| US11392213B2 (en) | Selective detection of visual cues for automated assistants | |
| EP4130975A1 (en) | Determining whether to automatically resume first automated assistant session upon cessation of interrupting second session | |
| US11854533B2 (en) | Speaker awareness using speaker dependent speech model(s) | |
| EP3835993A2 (en) | Keyword extraction method, apparatus and medium | |
| CN111753552B (en) | NLP-based training mode and recognition mode dynamic switching method | |
| KR20190074508A (en) | Method for crowdsourcing data of chat model for chatbot | |
| Qoriah | Ideational meaning of HIV AIDS slogans: A systemic functional linguistic study | |
| JP2022544428A (en) | Search item rewriting method, device, device and storage medium | |
| CN113767379B (en) | Rendering content using content agents and/or stored content parameters | |
| US20230343336A1 (en) | Multi-modal interaction between users, automated assistants, and other computing services | |
| CN116795957A (en) | Dialogue information acquisition method and device, storage medium and electronic device | |
| CN114020153A (en) | Multi-mode man-machine interaction method and device | |
| CN116762126A (en) | Locally performed device arbitration for automatic speech recognition | |
| CN117253478A (en) | Voice interaction method and related device | |
| Brick | TIDE: A timing-sensitive incremental discourse engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |