CN117273151A

CN117273151A - Scientific instrument use analysis method, device and system based on large language model

Info

Publication number: CN117273151A
Application number: CN202311557340.XA
Authority: CN
Inventors: 侯金磊; 马良; 钟巧勇; 谢迪; 浦世亮
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2023-11-21
Filing date: 2023-11-21
Publication date: 2023-12-22
Anticipated expiration: 2043-11-21
Also published as: CN117273151B

Abstract

The application provides a scientific instrument use analysis method, device and system based on a large language model, and relates to the field of artificial intelligence, wherein the method comprises the following steps: acquiring information of a related query problem of the use of a scientific instrument input by a user; extracting visual feature vectors of pictures in the query problem information through a visual coding model; encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through a large language model to obtain analysis information corresponding to the query problem information; the analysis information is presented to the user. The method is suitable for the experiment process by using scientific instruments and is used for reducing the professional level requirements of operators.

Description

Scientific instrument use analysis method, device and system based on large language model

Technical Field

The application relates to the field of artificial intelligence, in particular to a scientific instrument use analysis method, device and system based on a large language model.

Background

Scientific instruments are mainly instruments and devices for detecting substance components, and mainly comprise instruments and devices for performing chromatography, spectrum and mass spectrometry detection.

In the use process of the scientific instrument, an operator is required to understand information related to the scientific instrument (such as the scientific instrument, substances detected by the scientific instrument, experimental conditions when the scientific instrument is used, and the like), and the requirement on the professional level of the operator is high.

Disclosure of Invention

Based on the technical problems, the application provides a scientific instrument use analysis method, device and system based on a large language model, which can guide the use of the scientific instrument by using the large language model, thereby reducing the professional level requirement on operators.

In a first aspect, the present application provides a method for analyzing usage of scientific instruments based on a large language model, the method comprising: acquiring information of a related query problem of the use of a scientific instrument input by a user; the scientific instrument is an instrument device for detecting the components of substances, and the query problem information comprises spectrograms, query problem voices and/or query problem texts; extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram; encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through a large language model to obtain analysis information corresponding to the query problem information; the analysis information is presented to the user.

Optionally, the visual coding model comprises a visual encoder and a linear mapping layer; the visual encoder is used for encoding spectrograms in the query problem information to obtain an initial visual feature vector; the linear mapping layer is used to align the initial visual feature vector with the text feature vector.

Optionally, in the case that the query question information is a query question text, and the query question text is used for querying experimental conditions of the analysis target compound set, the method further includes: identifying target compounds in a target compound set in the query question information through a large language model; searching a preset experimental condition library according to the target compound to obtain one or more target experimental conditions corresponding to the target compound; the experimental condition library comprises correspondence between a plurality of compounds and experimental conditions; displaying target experiment conditions to a user; the large language model has the function of calling an informatization tool; the information tool includes: a chemistry information tool and a computational chemistry tool; the method further comprises the steps of: if the experimental condition library does not comprise the target compound, identifying the molecular structure of the target compound by using a chemical information tool; predicting the compound property of the target compound from the molecular structure of the target compound by calculating a chemical tool; the compound properties include polarity, ionization constant, and oil-water partition coefficient; reasoning the compound property of the target compound through a large language model to obtain a reference experimental condition corresponding to the target compound; and displaying the reference experimental conditions to the user through at least one of natural language, pictures and animation and video in the GIF format.

Optionally, reasoning the target feature vector through the large language model to obtain analysis information corresponding to the query problem information, including: calling a rdkit library of Python through a large language model, and analyzing the molecular structure of the target compound according to the Smiles structural formula of the target compound; the molecular structure includes the number of cyclic structures, the number of aromatic rings, and the number of hydroxyl groups; the chemical nature of the target compound is predicted from the molecular structure of the target compound by means of a graphometer model.

Optionally, the graphometer model includes center coding, spatial coding, and edge coding; center coding is used to describe the importance of each vertex in the molecular structure; spatial coding is used to represent the distance of two vertices; edge coding is used to describe the type of chemical bond between two vertices.

Optionally, in the case that the query issue information includes a file and a query issue text, the method further includes: converting the file in the query problem information into a file text; dividing a file text into a plurality of file text blocks according to a preset word number threshold; dividing a query question text in the query question information into a plurality of query text blocks according to a preset word number threshold; determining a target text block from the plurality of file text blocks according to the plurality of query text blocks; the target text block is a text block with similarity with the query text block being greater than a similarity threshold; and reasoning is carried out based on the prompt of the target text block through the large language model, and answer information corresponding to the query question information is determined.

Optionally, determining the target text block from the plurality of file text blocks according to the plurality of query text blocks includes: vectorizing a plurality of file text blocks by using an encoder in the Bert model to obtain a feature vector of each file text block; vectorizing a plurality of query text blocks by using an encoder in the Bert model to obtain a feature vector of each query text block; and determining the file text blocks with the similarity between the feature vector and the feature vector of the query text block being greater than a similarity threshold value from a plurality of file text blocks as target text blocks according to the feature vector of each query text block.

Optionally, the method further comprises: acquiring a first training sample set; the first training sample set comprises a plurality of first training samples, and each first training sample comprises a picture and a description text corresponding to the picture; training the linear mapping layer based on the first training sample set so that the linear mapping layer has a function of converting visual features of the picture into semantic features; acquiring a second training sample set; the second training sample set comprises a plurality of second training samples, and each second training sample comprises a training spectrogram and a description text corresponding to the training spectrogram; training the linear mapping layer based on the second training sample set so that the linear mapping layer has a function of converting visual features of the spectrogram into semantic features; acquiring a third training sample set; the third training sample set comprises a plurality of third training samples, and each third training sample comprises a training spectrogram and question-answer sentence pairs corresponding to the training spectrogram; the question-answer sentence pair comprises a pair of question sentences and answer sentences; training the linear mapping layer based on the third training sample set so that the linear mapping layer has the function of converting the visual features of the spectrogram into the semantic features of the problem statement and the semantic features of the answer statement; and obtaining a visual coding model according to the linear mapping layer trained based on the third training sample set.

Optionally, the large language model includes a word segmenter and a decoder; the word segmentation device is used for converting the text into feature vectors; the decoder is composed of stacked transducer modules; each transducer module consists of a multi-headed self-attention layer, a feed-forward neural network layer, residual connections, and layer normalization.

According to the analysis method for the use of the scientific instrument based on the large language model, analysis information corresponding to the query problem information can be determined according to the related query problem information of the use of the scientific instrument input by the user and the preset large language learning model, namely, the analysis information corresponding to the query problem information can be understood and analyzed by using the large language model to the use of the scientific instrument input by the user, the analysis information corresponding to the query problem information is fed back, the user can use the scientific instrument to perform experiments without having a higher professional level, and the requirement of the professional level of operators of the scientific instrument is reduced.

In addition, the analysis method for the use of the scientific instrument based on the large language model can utilize the large language model obtained through corpus training based on the scientific instrument field to understand and analyze query problem information input by a user, compared with the large language model in the current general field, the analysis method for the use of the scientific instrument based on the large language model obtained through corpus training in the vertical field of the scientific instrument field has the advantages that the study on the query problem information related to the use of the scientific instrument is more complete, the response speed is faster, and the efficiency of experiments conducted by using the scientific instrument is improved.

In addition, in the large language model-based scientific instrument use analysis method, query problem texts input by users and analysis information displayed by item users are natural languages, so that the use threshold of a deep learning model in the field of scientific instruments is reduced, the user can understand conveniently, and the use experience of the user is improved.

In a second aspect, the present application provides a large language model based scientific instrument usage analysis device comprising the functional modules for the method of the first aspect above.

In a third aspect, the present application provides a scientific instrument usage analysis system based on a large language model, the system comprising: a front end and a rear end; the front end is used for acquiring information of a query problem related to the use of the scientific instrument input by a user; scientific instruments are instruments and equipment for detecting the components of substances; the query issue information includes one or more of the following: files, spectrograms, and query question text; the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram, or a mass spectrogram; the query problem text is natural language; the back end is used for determining analysis information corresponding to the query problem information according to the query problem information and a preset large language model; under the condition that the query problem information comprises a spectrogram and a query problem text, the back end is specifically used for extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training spectrograms and descriptive texts corresponding to the training spectrograms; encoding a query question text in the query question information through a large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through a large language model to obtain analysis information; the front end is also used for displaying analysis information to a user; the analysis information is natural language.

The advantageous effects of the second aspect to the third aspect described above may be described with reference to the first aspect, and will not be repeated.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a large language model based scientific instrument usage analysis system according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of another method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 6 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a graphometer model provided in an embodiment of the present application;

FIG. 8 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 9 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a large language model based scientific instrument usage analysis system according to an embodiment of the present application;

FIG. 11 is a logic diagram of a large language model based scientific instrument usage analysis system provided by an embodiment of the present application;

fig. 12 is a schematic diagram of a composition of a scientific instrument usage analysis device based on a large language model according to an embodiment of the present application.

Detailed Description

Hereinafter, the terms "file," "query," and "third," etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "file," "query," or "third," etc., may explicitly or implicitly include one or more of such features.

Chromatographic detection is a separation and analysis technique in which the components of a mixture are separated by a chromatographic column and the separated components are then analyzed and measured.

Chromatographic detection may include liquid chromatographic detection, gas chromatographic detection, and the like.

Liquid chromatography detection uses liquid as mobile phase, carries the mixture through chromatographic column, and utilizes different substances with different adsorption or dissolution capacities on the chromatographic column to separate different components on the chromatographic column. Liquid chromatography detection can generally be performed using a liquid chromatograph.

The gas chromatography detection uses gas as a mobile phase, carries the mixture through a chromatographic column, and utilizes the adsorption or dissolution capability of different substances on the chromatographic column to separate different components on the chromatographic column. Gas chromatography detection can typically be performed using a gas chromatograph.

The spectroscopic detection may include ultraviolet-visible spectroscopic detection, infrared light detection, X-ray diffraction detection, and the like.

Ultraviolet-visible spectrum detection by measuring the absorption spectrum of a substance in ultraviolet-visible light, the molecular structure and chemical constitution of the substance can be studied. Ultraviolet-visible spectrum detection can typically be performed using an ultraviolet-visible spectrometer.

Infrared spectrum detection uses electromagnetic radiation in the infrared region to measure vibration and rotational energy levels of molecules of a substance, thereby determining chemical composition and structural information of the substance. Infrared spectroscopy can be generally performed using an infrared spectrometer.

X-ray diffraction detection determines the crystal structure and chemical composition of a substance by measuring the angle of diffraction of X-rays in the substance. X-ray diffraction detection can be generally performed using an X-ray diffractometer.

Mass spectrometry detection may include conventional mass spectrometry detection, tandem mass spectrometry detection, time-of-flight mass spectrometry detection, and the like.

Conventional mass spectrometry detection is performed by a conventional mass spectrometer. Conventional mass spectrometers include an ionization source, a mass analyzer, a detector, and the like. Conventional mass spectrometry detection can obtain a mass spectrum of a sample by ionizing the sample by an ionization source, separating ionized ions according to mass-to-charge ratios by a mass analyzer, and detecting ion signals by a detector.

Tandem mass spectrometry detection was performed by a tandem mass spectrometer. Tandem mass spectrometers include an ionization source, a mass analyzer, a detector, additional ionization sources, and additional mass analyzers, among others. The tandem mass spectrum detection can be used for carrying out ionization and separation processes on samples at two poles or more, so that richer particle information and higher detection sensitivity are obtained.

Time-of-flight mass spectrometry detection is performed by a time-of-flight mass spectrometer. The time-of-flight mass spectrometer includes an ionization source, a flight tube, a mass analyzer, and the like. The time-of-flight mass spectrum detection can ionize a sample through an ionization source, and the mass and speed information of the ions can be obtained by measuring the time of flight of the ionized ions in a flight tube.

In the use of scientific instruments (such as the above-mentioned various mass spectrometers), an operator is required to understand information related to the scientific instruments (such as the scientific instruments themselves, substances detected by the scientific instruments, experimental conditions when the scientific instruments are used, etc.), and the requirement on the expertise level of the operator is high.

Based on the above, the embodiment of the application provides a method, a device and a system for analyzing the usage of a scientific instrument based on a large language model, which can guide the usage of the scientific instrument by using a deep learning model in the vertical field obtained by corpus training related to the field of the scientific instrument, thereby reducing the professional level requirement on operators.

The following description is made with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of a composition of a large language model based scientific instrument usage analysis system according to an embodiment of the present application. As shown in fig. 1, the system includes an input 100, a processor 200, and a display 300.

The input 100 may be a mouse, a keyboard, a touch display, a microphone, a camera, or the like.

The input 100 may be used to obtain information about the use of scientific instruments by a user.

The processor 200 is configured to determine corresponding analysis information according to the query issue information, and the specific process may be described in the following embodiments, which are not repeated here.

Display 300 may be used to present analysis information to a user.

The following describes a method for analyzing the usage of a scientific instrument based on a large language model provided in the embodiment of the present application.

Fig. 2 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application, and optionally, the method may be performed by a guidance system having the components shown in fig. 1 and shown in fig. 2, and the method includes S101 to S103.

S101, acquiring related query problem information of scientific instrument use input by a user.

Wherein, scientific instrument is the instrument equipment for detecting the material composition. The query question information includes a spectrogram and query question speech and/or query question text, the spectrogram includes at least one of a chromatographic spectrogram, a spectral spectrogram, or a mass spectrogram, and the query question text is natural language.

Alternatively, as described above, the input 100 may be a mouse, a keyboard, a touch-sensitive display screen, a microphone, a camera, or the like. The guidance system may receive query question information entered by the user via a mouse, keyboard, touch display, microphone, or camera.

Alternatively, the input/output interface of the terminal device may also be a microphone. The front end 100 may receive query question voices input by a user through a microphone and convert the query question voices into query question text. The specific conversion process may be described in the related art, and will not be described herein.

S102, determining analysis information corresponding to the query problem information according to the query problem information and a preset large language model.

The deep learning model comprises a large language model, wherein the large language model is obtained based on corpus training in the field of scientific instruments;

s102 may be described in the following embodiments, and will not be described herein.

S103, displaying analysis information to a user.

Wherein the analysis information is natural language.

For example, a front end in the guidance system may receive the analysis information sent by the back end and display the analysis information via a question-answer interface.

According to the analysis method for the use of the scientific instrument based on the large language model, analysis information corresponding to the query problem information can be determined according to the related query problem information of the use of the scientific instrument input by the user and the preset large language learning model, namely, the analysis information corresponding to the query problem information can be understood and analyzed by using the large language model to the use of the related query problem information of the scientific instrument input by the user, the analysis information corresponding to the query problem information is fed back, the user can use the scientific instrument to perform experiments without having a higher professional level, and the requirements of the professional level of operators of the scientific instrument are reduced.

In addition, the method for analyzing the use of the scientific instrument based on the large language model can utilize the large language model obtained through corpus training based on the scientific instrument field to understand and analyze query problem information input by a user, and compared with the large language model in the current general field, the method for analyzing the use of the scientific instrument based on the large language model obtained through corpus training in the vertical field of the scientific instrument field has the advantages that the study on the query problem information related to the use of the scientific instrument is more complete, the response speed is faster, and the efficiency of experiments conducted by using the scientific instrument is improved.

In addition, in the method for analyzing the use of the scientific instrument based on the large language model, query problem text input by a user and analysis information displayed to the user are both natural languages, so that the use threshold of a deep learning model in the field of the scientific instrument is reduced, the user can understand the query problem text conveniently, and the use experience of the user is improved.

In some embodiments, the aforementioned guidance system (or back-end in the guidance system) has several functions: scientific instrument local knowledge base questions and answers, scientific instrument spectrogram analysis, scientific instrument experiment initial condition recommendation, scientific instrument related tool call, and scientific instrument common sense knowledge questions and answers.

The following describes these functions:

1. scientific instrument local knowledge base questions and answers.

As described above, the query question information entered by the user may include a file, and the guidance system may use the file entered by the user as a local knowledge base and guide (or reply) questions posed in the query question text based on the local knowledge base. Based on the above understanding, fig. 3 is another flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application. As shown in fig. 3, in the case where the query issue information includes a file and a query issue text, S102 may specifically include S1021a to S1025a.

S1021a, converting the file in the query question information into a file text.

Alternatively, when it is detected that the query issue information includes a file, the guidance system may call a base tool to convert the file into file text.

For example, if the file in the query question information is a word file or txt file, the guidance system may directly convert the word file or txt file into a file text using Python (which may be understood as the basic tool described above).

For another example, if the file in the query issue information is a portable document format (portable document format, PDF) file, the guidance system may analyze the PDF file using a PDF analysis tool (i.e., the basic tool described above) to obtain a file text.

Alternatively, the PDF parsing tool may use a nougat model, which is a transducer architecture of an encoder-decoder trained end-to-end, and may process not only PDF files, but also scanned version PDF files, and in particular, has a better effect on the recognition of a table (in the relevant literature of scientific instruments, much information is presented in the form of a table).

And S1022a, dividing the file text into a plurality of file text blocks according to a preset word number threshold.

The preset word number threshold may be preset in the guidance system (or back end 200) by a manager. For example, the preset word count threshold may be set to 200 or 300, or the like. The specific numerical value of the preset word number threshold is not limited in the embodiment of the application.

Alternatively, the guidance system may use a preset separation symbol to separate different text blocks of a document or text in the same text block of a document.

For example, the preset separation symbol may be. "," \n ", or" \n ", etc. The embodiment of the application does not limit the specific types of the preset separation symbols.

S1023a, dividing the query question text in the query question information into a plurality of query text blocks according to a preset word number threshold.

S1023a may be described with reference to S1022a, and is not described herein.

It should be understood that the input of the large language model has a length limitation, and the scientific instrument based on the large language model provided in the embodiment of the application uses the analysis method to divide the text into text blocks according to the preset word number threshold before matching the file text and the query problem text so as to meet the input length limitation of the large language model.

S1024a, determining a target text block from the plurality of file text blocks according to the plurality of query text blocks.

The target text block is a text block with similarity with the query text block being greater than a similarity threshold.

In one possible implementation, to facilitate retrieval of the target text block, the guidance system may vector the file text block and the query text block prior to retrieving the target text block. In this case, the step S1024a may specifically include the following steps:

a1, vectorizing a plurality of file text blocks to obtain feature vectors of each file text block.

For example, the guidance system may use an encoder in the Bert model to vector the text blocks of the file to obtain feature vectors for each text block of the file.

Optionally, the guidance system may also fine tune the Bert model using a large number of scientific instrument documents before vectorizing the file text blocks using encoders in the Bert model, so that the Bert model learns the features of the scientific instrument field.

A2, vectorizing the plurality of query text blocks to obtain the feature vector of each query text block.

a2 may be described with reference to a1 above, and will not be described here again.

a3, according to the feature vector of each query text block, determining a file text block with similarity between the feature vector and the feature vector of the query text block being greater than a similarity threshold value from a plurality of file text blocks as a target text block.

Alternatively, the guidance system may use the feature vector of each query text block to retrieve the most similar document text block as the target text block in the vector database. The manner in which the similarity is calculated may specifically be cosine similarity.

Alternatively, the guidance system may utilize multi-threaded processing of a central processing unit (central processing unit, CPU) to retrieve multiple similar blocks of document text at the same time.

S1025a, reasoning is carried out based on the prompt of the target text block through the large language model, and answer information corresponding to the query question information is determined.

Alternatively, the guidance system may enter the retrieved target text block, query question text, and prompt word into a large language model, which may infer the input and then return answer information.

For example, the prompt word may be "please answer the question of the user (query question text) in combination with the above text (target text block).

Illustratively, the guidance system may provide an interface to the outside in the form of a chat robot, the user may input a question text to be consulted in an input box of the interface, the user may then provide a local knowledge base (e.g., txt, pdf, or a word-like format file) to the guidance system, the guidance system may understand the question text input by the user and match related information (target text block) in the local knowledge base, then analyze in combination with the matched related information, and then return an answer to the user's question in the form of a natural language. If no relevant information is retrieved, the guidance system may return "no relevant information retrieved" and answer the question text entered by the user in combination with the common sense knowledge trained.

For example, a user may upload one or more files of liquid chromatography in an input box of the interface and then ask questions about the file, such as "summarize this file, if there is a compound to analyze, please complain about me analyzed compound and chromatographic conditions employed". The guidance system can understand the question text input by the user, then extracts the related information in the document, analyzes the user question according to the extracted related information, and finally returns the answer of the user question in a natural language form.

2. And (5) analyzing spectrograms of scientific instruments.

As described above, the query question information entered by the user may include a spectrogram, and the guidance system may analyze the spectrogram entered by the user. Fig. 4 is a schematic flow chart of a method for analyzing usage of a scientific instrument based on a large language model according to an embodiment of the present application. As shown in fig. 4, in the case where the query issue information includes a spectrogram and a query issue text, S102 described above may specifically include S1021b to S1024b.

S1021b, extracting visual feature vectors of pictures in the query problem information through a visual coding model.

The visual coding model is obtained based on training of a training spectrogram and a descriptive text corresponding to the training spectrogram, and can comprise a visual encoder and a linear mapping layer. The visual encoder is used for encoding spectrograms in the query problem information to obtain an initial visual feature vector. For example, the visual encoder may employ a visual encoder in the BLIP-2 model, which may include an image encoder (Vision Transformers, viT) model and a Q-former model. ViT the model is Vision Transformers architecture, and the model weight is a Pre-Training weight obtained by Training a large number of Image-text pair data in a CLIP (Contrastive Language-Image Pre-Training) Training mode; the Q-former model consists of two transducer modules, an image transducer module for interacting visual features with the image encoder and a text encoding-decoding bi-directional text transducer module. The visual encoder uses pre-training weights in the BLIP-2 model, without participating in the training of the entire visual encoding model. The linear mapping layer is used to align the initial visual feature vector with the text feature vector. The training process of the visual coding model may be described in the following embodiments, and will not be described in detail here.

S1022b, encoding the query question text in the query question information through the large language model to obtain a first text feature vector.

For example, a word segmentation device (token) may be included in the large language model, and the guidance system may specifically encode the query question text by using the word segmentation device in the large language model to obtain the first text feature vector.

S1023b, splicing the visual feature vector and the first text feature vector to obtain a target feature vector.

Alternatively, the splicing mode may be simple series connection or splicing, or may be complex cross-modal conversion. The embodiment of the application does not limit the specific mode of splicing.

S1024b, reasoning the target feature vector through the large language model to obtain analysis information.

For example, the large language model may include a decoder, and the guidance system may specifically perform decoding reasoning on the target feature vector through the decoder in the large language model to obtain analysis information.

Optionally, the guidance system may also be trained to obtain a visual coding model in several stages:

the first stage:

step 1, acquiring a first training sample set.

The first training sample set comprises a plurality of first training samples, and each first training sample comprises a picture (natural image) and descriptive text corresponding to the picture.

And step 2, training the linear mapping layer based on the first training sample set so that the linear mapping layer has a function of converting visual features of the picture into semantic features.

For example, the guidance system may train the linear mapping layer in a first stage with a first number of first training samples. The first number may be set to 1000 ten thousand.

And a second stage:

and step 3, acquiring a second training sample set.

The second training sample set comprises a plurality of second training samples, and each second training sample comprises a training spectrogram and descriptive text corresponding to the training spectrogram.

And 4, training the linear mapping layer based on the second training sample set so that the linear mapping layer has a function of converting visual features of the spectrogram into semantic features.

For example, the guidance system may train the linear mapping layer with a second number of second training samples in a second stage. The second number may be set to 1 ten thousand.

And a third stage:

and 5, acquiring a third training sample set.

The third training sample set comprises a plurality of third training samples, each third training sample comprises a training spectrogram and a question-answer sentence pair corresponding to the training spectrogram, and the question-answer sentence pair comprises a pair of question sentences and answer sentences.

And step 6, training the linear mapping layer based on the third training sample set so that the linear mapping layer has the function of converting the visual features of the spectrogram into the semantic features of the problem statement and the semantic features of the answer statement.

For example, the guidance system may train the linear mapping layer in a third stage with a third number of third training samples. The third number may be set to 1000.

And 7, obtaining a visual coding model according to the linear mapping layer trained based on the third training sample set.

Alternatively, as described above, the visual coding model may include a visual encoder and a linear mapping layer. In order to improve the training efficiency of the visual coding model, parameters in the visual coder can be fixed in the training process, namely the visual coder does not participate in training, wherein the parameters adopt pre-training weights in the BLIP-2 model, and the visual coder which does not participate in training is combined with a linear mapping layer trained in three stages to obtain the visual coding model.

3. And recommending initial conditions of scientific instrument experiments.

In the case where the query question information is a query question text and the query question text is used to query the experimental conditions for analyzing the set of target compounds, fig. 5 is a schematic flow chart of a method for analyzing the usage of a scientific instrument based on a large language model according to an embodiment of the present application. As shown in fig. 5, the S102 may specifically include S1021c to S1023c.

S1021c, identifying target compounds in the target compound set in the query question information through the large language model.

The training data of the large language model may include compound name entity recognition task data, and in this case, the large language model may have a function of recognizing a compound.

And S1022c, searching a preset experimental condition library according to the target compound to obtain one or more target experimental conditions corresponding to the target compound.

Wherein the library of experimental conditions includes correspondence between a plurality of compounds and experimental conditions. The experimental conditions may be liquid phase, gas phase, or experimental conditions for mass spectrometry detection.

Illustratively, the experimental library may be specifically as shown in table 1 below:

TABLE 1

As shown in table 1, compound 1 may correspond to experimental condition 1 and experimental condition 2. Compound 2 may correspond to experimental condition 3 and experimental condition 4.

Alternatively, the guidance system may take the target compound as an index, search through a library of experimental conditions, and take the experimental condition corresponding to the target compound in the library of experimental conditions as the target experimental condition.

S1023c, displaying target experiment conditions to a user.

In one possible implementation, the guidance system may present all of the target experimental conditions to the user.

In another possible implementation, the library of experimental conditions further includes a source of each correspondence. The step S1021c may specifically include the following steps:

and step 1, under the condition that the target experimental conditions comprise a plurality of target experimental conditions, determining the first M target experimental conditions from the plurality of target experimental conditions according to a preset source priority order, and displaying the first M target experimental conditions to a user.

Wherein M is an integer greater than 0. M may be preset in the guidance system by a manager. For example, M may be set to 3 or 5, etc. The specific value of M is not limited in the embodiment of the present application.

Illustratively, the preset source priority order may be, in order from high to low: scientific instrument manufacturers, standards, academic papers, others.

Optionally, the guidance system may further obtain a compound property of the target compound, and after determining the target experimental condition, input the compound property of the target compound, the target experimental condition, and the prompt word into a large language model, and the large language model may infer the input, analyze the explanation of the target experimental condition in combination with the compound property of the target compound, and display the result to the user.

For example, the term may be set to "please bind to the nature of the compound, and the experimental conditions described above are interpreted.

The procedure for directing the system to obtain the nature of the compound of interest may be described in the examples below and will not be described in detail herein.

Optionally, the large language model also has a function of calling an informatization tool; the information tool includes: the chemical information tool and the calculation chemical tool are specifically described with reference to the following examples. In this case, for the case where the target compound is not included in the experimental condition library, fig. 6 is a schematic flow chart of a further analysis method for use of the scientific instrument based on the large language model according to the embodiment of the present application. As shown in fig. 6, the method may further include S201 to S204.

S201, if the experimental condition library does not comprise the target compound, the molecular structure of the target compound is identified by using a chemical information tool.

Alternatively, the guidance system may include a chemical information tool having the ability to query the underlying chemical information. The guidance system may query the molecular results of the target compound through the semiochemical tool.

For example, the semiochemical tool may be implemented primarily by an elastisearch database. The guidance system may invoke a chemical information tool to convert the chinese or english name of the target compound into a unique chemical abstract (Chemical Abstracts Service, CAS) number and Smiles structural formula for the target compound. To support compound chinese name to CAS retrieval, a compound chinese name database may be created that contains a plurality of data sources, which may include data from a chinese database and an english database, which may be translated into chinese using translation tools for compound names in the english database, thereby improving hit rate and accuracy of compound chinese name conversion to CAS number through mutual verification of the plurality of data sources.

The guidance system may then call the rdkit library of Python to calculate the molecular weight of the target compound, analyze the molecular structure of the target compound (e.g., number of cyclic structures, number of aromatic rings, number of hydroxyl groups, etc.) based on the Smiles structural formula of the target compound

S202, predicting the compound property of the target compound according to the molecular structure of the target compound by calculating a chemical tool.

Among them, the computational chemistry tools have the function of calculating the properties of compounds, including polarity, ionization constant (pKa), oil-water partition coefficient (log p, log d), etc. The computational chemistry tool is obtained by training a Graphomer model with the molecular structure of the training compound, and the pKa experimental value and log D experimental value of the training compound obtained in the published literature, and the Loss function during training can be a minimum mean square error Loss (mean squared error Loss, MSE Loss) function of a predicted value and a true value.

Alternatively, the computational chemistry tool may be implemented primarily by a graphometer model, and the guidance system may predict the chemical nature of the target compound from the molecular structure of the target compound, specifically by the graphometer model.

Fig. 7 is a schematic structural diagram of a graphometer model according to an embodiment of the present application. As shown in fig. 7, on the basis of the left standard transducer model, the graphometer model adds three codes related to the graph structure information to compensate that the attention mechanism of the transducer model cannot code the graph structure information.

Among them, centrality Encoding (center code) is used to describe the importance of each node (vertex in molecular structure) in the graph (molecular structure), where the importance of different nodes is different, and common attention mechanisms cannot take this information into account. Centrality Encoding are weighted fused with the input features. Spatial Encoding is used to describe the distance of two nodes, and for each pair of nodes, the shortest distance of the two nodes on the graph is used as Spatial Encoding between the two nodes. Edge Encoding (Edge Encoding) is used to describe the type of Edge (chemical bond in molecular structure) between two nodes, and is used to fuse the Edge representation information of the shortest path between two nodes, as a part of reference information. Both Spatial Encoding and Edge Encoding are used to calibrate the attention score in the transducer, allowing the model to take into account the distance between nodes, the Edge type when calculating the attention score between nodes. The final degree score calculation can be expressed as the following equation (1):

；

in the formula (1),representing the original attention score in the left transducer. Representing Spatial Encoding. />Representing Edge Encoding.

Alternatively, the guidance system may train the graphometer model with a PCQM4N dataset comprising 380 ten thousand molecular structures, so that the graphometer model can learn rich chemical knowledge, and can better understand the molecular structure, thereby effectively improving accuracy of pKa and logP predictions.

S203, reasoning the compound property of the target compound through a large language model to obtain a reference experimental condition corresponding to the target compound.

S204, displaying reference experiment conditions to a user through at least one of natural language, pictures, animation in a GIF format and video.

Optionally, after determining the reference experimental condition, the guidance system may further input the compound property of the target compound, the reference experimental condition, and the prompt word into a large language model, and the large language model may infer the input, interpret and analyze the reference experimental condition in combination with the compound property of the target compound, and display the same to the user.

For example, the guidance system may provide an interface to the outside in the travel of dialogue questions, the user may input questions to be consulted in an input box of the interface, such as acquiring initial experimental conditions of liquid chromatography for which compounds are to be analyzed, and the user may input what liquid chromatography conditions should be used for "i want to analyze vitamin a, vitamin D, and vitamin E". Or vitamin A, vitamin D and vitamin E, the guiding system can automatically recognize the input intention of the user, then the self-built database is queried whether the experimental conditions reported by the open literature exist or not through the steps, and if the experimental conditions exist, the guiding system can analyze and explain each sub-item in the queried chromatographic conditions by utilizing the chemical properties, so that the user can judge whether the experimental conditions are available or not. If the natural language is not available, the guidance system can analyze the properties of the compound according to the knowledge obtained by training, recommend a reasonable experimental condition and finally return the experimental condition and corresponding explanation in a natural language form.

4. And recommending initial conditions of scientific instrument experiments.

The guidance system may include an informatization tool, which may include: chemical information tools, computational chemical tools, scientific instrument tools, and base tools.

The chemical information tool has the function of inquiring basic chemical information, wherein the basic chemical information comprises CAS number, molecular structure and molecular weight of chemical abstract society of the compound. The details of the chemical information tool may be described above with reference to S201, and will not be described here again.

The chemical calculation tool has the function of calculating the properties of the compound, wherein the properties of the compound comprise polarity, ionization constant and oil-water distribution coefficient. The details of the calculation chemical means may be described with reference to S202 above, and will not be described here again.

The scientific instrument tool has the function of calculating relevant parameters related to the use of the scientific instrument, wherein the relevant parameters comprise chromatographic column similarity.

The basic tool has the functions of converting files into texts and recognizing the texts. The details of the basic tool may be described in S1021a above, and will not be described here again.

Alternatively, the large language model may be provided with functionality to invoke the informatization tool.

Optionally, in the case that the query issue information is a query issue text, fig. 8 is a schematic flow chart of another analysis method for use of a scientific instrument based on a large language model according to an embodiment of the present application. As shown in fig. 8, S102 may specifically include S1021d to S1022d.

S1021d, determining a target tool from the informationized tools according to the query problem text.

In one possible implementation, the guidance system may be pre-configured with a correspondence between trigger words and an informatization tool. If the query question text comprises the target trigger word, the guidance system can take the informationized tool corresponding to the target trigger word as the target tool in the corresponding relation between the trigger word and the informationized tool.

In another possible implementation manner, the guiding system may vectorize the query issue text to obtain the query text feature, and may preset a description text of the informationized tool in the guiding system, and the guiding system may vectorize the description text, perform similarity matching on the vectorized query issue text and the description text, and use the informationized tool corresponding to the description text most similar to the query issue text as the target tool.

S1022d, generating a tool calling instruction through the large language model to call the target tool to obtain analysis information corresponding to the query problem text.

For example, the guidance system may provide an interface to the outside in the form of a chat robot, and the user may input the questions that he wants to consult in the input box of the interface, and for some objective questions, the guidance system may automatically understand the questions and determine whether to invoke the informatization tool. For example, the user may ask what the CAS number for vitamin A is. "instruction system may generate tool call instruction through large language model" tool call: CAS number of vitamin a: the guide system may perform regular pattern matching, and after matching to "# result=get_cas (" vitamin a ")", the guide system may call the function get_cas ("vitamin a") to obtain a result, and then the guide system may reorganize the prompt word "the problem of user input is: what the CAS number of vitamin a is. And (3) calling an n tool: CAS number of vitamin a: results returned by # result=get_cas ("vitamin a") \n: 68-26-8\n please answer the user's question "in connection with the above procedure and input the prompt words into the large language model summary and output.

5. Knowledge and question-answering of scientific instruments.

In the case that the query question information is a query question text, and the query question text is used to query the basic knowledge related to the usage of the scientific instrument, fig. 9 is a schematic flow chart of a method for analyzing the usage of the scientific instrument based on the large language model according to the embodiment of the present application. As shown in fig. 9, S102 may specifically include S1021e.

S1021e, analyzing the query problem information through the large language model to obtain analysis information.

Illustratively, the guidance system may provide an interface to the outside in the form of a chat robot, and the user may input a question that he wants to consult, such as a question related to liquid chromatography, in an input box of the interface, and the user may ask "how to suppress the tailing of the alkaline compound". ". The large language model may automatically understand the user's input intent and then return answers to the user's questions in natural language. In addition, the function can be not in conflict with the functions of scientific instrument initial experiment condition recommendation, scientific instrument spectrogram analysis and scientific instrument local knowledge base question-answering, and the effect of multi-round dialogue can be realized.

Alternatively, the large language model may employ a decoder structure, i.e., the large language model may include a word segmenter (tokenizer) and a decoder (decoder).

Wherein the word segmentation device is used for converting the text into feature vectors which can participate in calculation. The decoder consists of stacked transducer modules. Each transducer module consists of a Multi-Head Self-Attention Layer (Multi-Head Self-Attention Layer), a Feed-Forward neural network Layer (Feed-Forward Layer), residual connections, and Layer normalization.

In the pre-training stage of the large language model, the large language model can be pre-trained by adopting universal knowledge text and scientific instruments and chemically related text data.

In the instruction fine tuning stage of the large language model, question and answer data of professionals such as professionals in the manufacturing and application industries of the scientific instruments, experimenters and the like and knowledge question and answer data of the scientific instruments disclosed on the network can be collected to conduct instruction fine tuning on the pre-trained large language model, and the knowledge question and answer capability of the large language model is endowed.

Alternatively, the penalty functions of both the pre-training phase and the instruction fine-tuning phase may employ predicting the penalty of the next word.

Alternatively, the loss function may be specifically shown in the following formula (2):

；

in the formula (2),indicate->Personal word (s)/(s)>Before->The word(s) of the word,the representation is +.>Predicting +. >Probability of individual words.

Optionally, in the training process of the large language model, scoring data of experts can be collected, and the reinforced learning strategy is adopted to perform preference alignment on the large language model.

Based on the understanding of the above embodiments, fig. 10 is a schematic structural diagram of a scientific instrument usage analysis system based on a large language model according to an embodiment of the present application. As shown in fig. 10, the guidance system may include five application capabilities of scientific instrument knowledge questions and answers, scientific instrument local knowledge base questions and answers, scientific instrument related tool calls, scientific instrument experiment initial condition recommendations, and scientific instrument spectrogram analysis. The system may include two algorithmic models, a visual coding model and a scientific instrument large language model (i.e., the large language model described above). The system may also include four tools, a chemical information tool, a computational chemical tool, a scientific instrument tool, and other base tools (i.e., the base tools described above). The guidance system can receive dialogue question and answer input of the user, apply the five application capabilities, and call the two algorithm models and the four tools to feed back analysis information to the user.

Based on the understanding of the above embodiments, fig. 11 is a logic diagram of a large language model based scientific instrument usage analysis system according to an embodiment of the present application. As shown in FIG. 11, the guidance system may analyze the input to determine if there is a file, if there is a picture, or if it is a plain text input. If the file is input, the local knowledge base question-answering function is called (scientific instrument) to conduct guidance. If the picture is input, a spectrogram analysis function is called (scientific instrument) to conduct guidance. If the text is input, analyzing the problem of the text expression by using a large model of the scientific instrument (namely the large language model), guiding by using a question and answer function of common knowledge of the scientific instrument, (initial) experimental condition recommending function of the scientific instrument and a tool calling function of the scientific instrument, and finally outputting analysis information by the large model of the scientific instrument.

The scientific instrument usage analysis system based on the large language model provided by the embodiment of the application can provide a method comprising the following steps: acquiring information of a related query problem of the use of a scientific instrument input by a user; the scientific instrument is an instrument device for detecting the components of substances, and the query problem information comprises spectrograms, query problem voices and/or query problem texts; extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram; encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through a large language model to obtain analysis information corresponding to the query problem information; the analysis information is presented to the user.

The foregoing description of the solution provided in the embodiments of the present application has been mainly presented in terms of a method. To achieve the above functions, it includes corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. The technical aim may be to use different methods to implement the described functions for each particular application, but such implementation should not be considered beyond the scope of the present application.

In an exemplary embodiment, the present application further provides a scientific instrument usage analysis device based on the large language model. Fig. 12 is a schematic diagram of a composition of a scientific instrument usage analysis device based on a large language model according to an embodiment of the present application. As shown in fig. 12, the apparatus includes: an acquisition module 1201 and a processing module 1202.

An obtaining module 1201, configured to obtain information about a query problem related to use of a scientific instrument input by a user; the scientific instrument is an instrument device for detecting the components of substances, and the query problem information comprises spectrograms, query problem voices and/or query problem texts;

a processing module 1202, configured to extract, by using a visual coding model, a visual feature vector of a picture in the query question information; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram; encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through a large language model to obtain analysis information corresponding to the query problem information; the analysis information is presented to the user.

Optionally, in the case that the query issue information is a query issue text, and the query issue text is used for querying experimental conditions of the analysis target compound set, the target compound set includes one or more target compounds, and the processing module 1202 is specifically configured to identify, through a large language model, the target compounds in the target compound set in the query issue information; searching a preset experimental condition library according to the target compound to obtain one or more target experimental conditions corresponding to the target compound; the experimental condition library comprises correspondence between a plurality of compounds and experimental conditions; displaying target experiment conditions to a user; the large language model has the function of calling an informatization tool; the information tool includes: a chemistry information tool and a computational chemistry tool; the processing module 1202 is further configured to identify a molecular structure of the target compound using the chemical information tool if the target compound is not included in the experimental condition library; predicting the compound property of the target compound from the molecular structure of the target compound by calculating a chemical tool; the compound properties include polarity, ionization constant, and oil-water partition coefficient; reasoning the compound property of the target compound through a large language model to obtain a reference experimental condition corresponding to the target compound; the reference experimental conditions are presented to the user through natural language.

Optionally, the processing module 1202 is specifically configured to call the rdkit library of Python, and analyze the molecular structure of the target compound according to the Smiles structural formula of the target compound; the molecular structure includes the number of cyclic structures, the number of aromatic rings, and the number of hydroxyl groups; the chemical nature of the target compound is predicted from the molecular structure of the target compound by means of a graphometer model.

Optionally, in the case that the query issue information includes a file and a query issue text, the processing module 1202 is further configured to convert the file in the query issue information into a file text; dividing a file text into a plurality of file text blocks according to a preset word number threshold; dividing a query question text in the query question information into a plurality of query text blocks according to a preset word number threshold; determining a target text block from the plurality of file text blocks according to the plurality of query text blocks; the target text block is a text block with similarity with the query text block being greater than a similarity threshold; analysis information is determined based on the hinting of the target text block through the large language model.

Optionally, the processing module 1202 is specifically configured to vectorize a plurality of document text blocks by using an encoder in the Bert model to obtain a feature vector of each document text block; vectorizing a plurality of query text blocks by using an encoder in the Bert model to obtain a feature vector of each query text block; and determining the file text blocks with the similarity between the feature vector and the feature vector of the query text block being greater than a similarity threshold value from a plurality of file text blocks as target text blocks according to the feature vector of each query text block.

Optionally, in the case that the query issue information is a query issue text, and the query issue text is used to query scientific instruments using related basic knowledge, the processing module 1202 is specifically configured to analyze the query issue information through a large language model, so as to obtain analysis information.

Optionally, the obtaining module 1201 is further configured to obtain a first training sample set; the first training sample set comprises a plurality of first training samples, and each first training sample comprises a picture and a description text corresponding to the picture; the processing module 1202 is further configured to train the linear mapping layer based on the first training sample set, so that the linear mapping layer has a function of converting visual features of the picture into semantic features; the obtaining module 1201 is further configured to obtain a second training sample set; the second training sample set comprises a plurality of second training samples, and each second training sample comprises a training spectrogram and a description text corresponding to the training spectrogram; the processing module 1202 is further configured to train the linear mapping layer based on the second training sample set, so that the linear mapping layer has a function of converting visual features of the spectrogram into semantic features; the obtaining module 1201 is further configured to obtain a third training sample set; the third training sample set comprises a plurality of third training samples, and each third training sample comprises a training spectrogram and question-answer sentence pairs corresponding to the training spectrogram; the question-answer sentence pair comprises a pair of question sentences and answer sentences; the processing module 1202 is further configured to train the linear mapping layer based on the third training sample set, so that the linear mapping layer has a function of converting the visual features of the spectrogram into semantic features of the question sentence and semantic features of the answer sentence; and obtaining a visual coding model according to the linear mapping layer trained based on the third training sample set.

It should be noted that the division of the modules in fig. 12 is illustrative, and is merely a logic function division, and other division manners may be actually implemented. For example, two or more functions may also be integrated in one processing module. The integrated modules may be implemented in hardware or in software functional units.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the processes or functions in accordance with embodiments of the present application are fully or partially produced. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer-executable instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, from one website, computer, server, or data center by wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). Computer readable storage media can be any available media that can be accessed by a computer or data storage devices including one or more servers, data centers, etc. that can be integrated with the media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), a solid state disk, etc.

Although the present application has been described herein in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a review of the figures, the disclosure, and the appended claims. In the claims, the word "Comprising" does not exclude other elements or steps, and the "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Although the present application has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present application. It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.

The foregoing is merely a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for analyzing usage of scientific instruments based on a large language model, the method comprising:

acquiring information of a related query problem of the use of a scientific instrument input by a user; the scientific instrument is an instrument device for detecting the substance components, and the query problem information comprises spectrograms, query problem voices and/or query problem texts;

extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram;

encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through the large language model to obtain analysis information corresponding to the query problem information;

And displaying the analysis information to the user.

2. The method of claim 1, wherein the visual coding model comprises a visual encoder and a linear mapping layer; the visual encoder is used for encoding spectrograms in the query problem information to obtain an initial visual feature vector; the linear mapping layer is to align the initial visual feature vector with a text feature vector.

3. The method according to claim 1, wherein in case the query question information is a query question text and the query question text is used for querying experimental conditions of analysis of a set of target compounds, the method further comprises:

identifying target compounds in a target compound set in the query question information through the large language model;

searching a preset experimental condition library according to the target compound to obtain one or more target experimental conditions corresponding to the target compound; the experimental condition library comprises correspondence between a plurality of compounds and experimental conditions;

displaying the target experimental conditions to the user;

the large language model has the function of calling an informatization tool; the information tool includes: a chemistry information tool and a computational chemistry tool; the method further comprises the steps of:

If the target compound is not included in the experiment condition library, identifying the molecular structure of the target compound by using the chemical information tool;

predicting, by the computational chemistry tool, a compound property of the target compound from a molecular structure of the target compound; the compound properties include polarity, ionization constant, and oil-water partition coefficient;

reasoning the compound property of the target compound through the large language model to obtain a reference experimental condition corresponding to the target compound;

the reference experimental conditions are displayed to the user through at least one of natural language, pictures and animation and video in a GIF format.

4. The method of claim 1, wherein reasoning the target feature vector through the large language model to obtain analysis information corresponding to the query problem information comprises:

calling a rdkit library of Python through the large language model, and analyzing the molecular structure of the target compound according to the Smiles structural formula of the target compound; the molecular structure includes the number of cyclic structures, the number of aromatic rings, and the number of hydroxyl groups;

Predicting the chemical nature of the target compound according to the molecular structure of the target compound through a Graphomer model.

5. The method according to claim 4, wherein the graphometer model includes center coding, spatial coding, and edge coding; the center code is used for describing the importance of each vertex in the molecular structure; the spatial coding is used for representing the distance between two vertexes; edge coding is used to describe the type of chemical bond between two vertices.

6. The method of claim 1, wherein in the case where the query issue information includes a file and query issue text, the method further comprises:

converting the file in the query problem information into a file text;

dividing the file text into a plurality of file text blocks according to a preset word number threshold;

dividing the query question text in the query question information into a plurality of query text blocks according to the preset word number threshold;

determining a target text block from the plurality of file text blocks according to the plurality of query text blocks; the target text block is a text block with similarity with the query text block being greater than a similarity threshold;

And reasoning is carried out based on the prompt of the target text block through the large language model, and answer information corresponding to the query question information is determined.

7. The method of claim 6, wherein determining a target text block from the plurality of file text blocks based on the plurality of query text blocks comprises:

vectorizing the plurality of file text blocks by using an encoder in the Bert model to obtain a feature vector of each file text block;

vectorizing the plurality of query text blocks by using an encoder in the Bert model to obtain a feature vector of each query text block;

and determining a file text block with similarity between the feature vector and the feature vector of the query text block being greater than the similarity threshold value from the plurality of file text blocks as the target text block according to the feature vector of each query text block.

8. The method according to claim 2, wherein the method further comprises:

acquiring a first training sample set; the first training sample set comprises a plurality of first training samples, and each first training sample comprises a picture and a description text corresponding to the picture;

Training the linear mapping layer based on the first training sample set so that the linear mapping layer has a function of converting visual features of pictures into semantic features;

acquiring a second training sample set; the second training sample set comprises a plurality of second training samples, and each second training sample comprises a training spectrogram and a description text corresponding to the training spectrogram;

training the linear mapping layer based on the second training sample set so that the linear mapping layer has a function of converting visual features of a spectrogram into semantic features;

acquiring a third training sample set; the third training sample set comprises a plurality of third training samples, and each third training sample comprises a training spectrogram and question-answer sentence pairs corresponding to the training spectrogram; the question-answer sentence pair comprises a pair of question sentences and answer sentences;

training the linear mapping layer based on the third training sample set so that the linear mapping layer has the function of converting the visual features of the spectrogram into the semantic features of the problem statement and the semantic features of the answer statement;

and obtaining the visual coding model according to the linear mapping layer trained based on the third training sample set.

9. The method of any of claims 1-8, wherein the large language model comprises a word segmenter and a decoder; the word segmentation device is used for converting the text into feature vectors; the decoder is composed of stacked Transformer modules; each transducer module consists of a multi-headed self-attention layer, a feed-forward neural network layer, residual connections, and layer normalization.

10. A scientific instrument usage analysis device based on a large language model, the device comprising: the device comprises an acquisition module and a processing module;

the acquisition module is used for acquiring information of related query questions of the scientific instrument use input by a user; the scientific instrument is an instrument device for detecting the substance components, and the query problem information comprises spectrograms, query problem voices and/or query problem texts; acquiring information of a related query problem of the use of a scientific instrument input by a user; the scientific instrument is an instrument device for detecting the components of the substances; the query issue information includes one or more of the following: files, spectrograms, and query question text; the spectrum comprises at least one of a chromatogram, a spectroscope, or a mass spectrogram; the query question text is natural language; the processing module is used for extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram; encoding query question voice and/or query question text in the query question information through a preset large language model to obtain a first text feature vector; splicing the visual feature vector and the first text feature vector to obtain a target feature vector; reasoning the target feature vector through the large language model to obtain analysis information corresponding to the query problem information; and displaying the analysis information to the user.

11. A large language model based scientific instrument usage analysis system, the system comprising: an input, a processor, and a display;

the input end is used for acquiring information of a related query problem of the scientific instrument use input by a user; the scientific instrument is an instrument device for detecting the substance components, and the query problem information comprises spectrograms, query problem voices and/or query problem texts;

the processor is used for extracting visual feature vectors of pictures in the query problem information through a visual coding model; the visual coding model is obtained based on training a training spectrogram and a descriptive text corresponding to the training spectrogram, wherein the spectrogram comprises at least one of a chromatographic spectrogram, a spectral spectrogram or a mass spectrogram;

The display is used for displaying the analysis information to the user.