TW202131352A - System of generating training data by questions and answers and method thereof - Google Patents

Abstract

A system of generating training data by questions and answers and a method thereof are provided by the disclosure. A platform-end provides the browse of a target image, receives the operation to adjust a display view of the target image, shows a question used to analyze the display view, gets a response answer, and associates the display view with the question and the response answer to generating the training data used to automatically analyze the target image. The disclosure can automatically prompt the region of interest, and significantly reduce the burden to collect the training data to the user.

Description

System and method for generating training data through question and answer

The present invention relates to a system and method for generating training data, in particular to a system and method for generating training data through question and answer.

For pathology interpretation, experts (such as pathologists) currently perform manual interpretation. In the process of manual interpretation, experts must use a microscope to view the solid glass slide, and use a recording pen, paper and pen to manually record the viewing results. Finally, Write a pathology report based on the viewing results.

However, the above method has the following problems: 1. Expert interpretation is inconsistent: different experts usually have different reading angles for the same physical slide (that is, the identified areas of interest are different), which will cause different experts to have different ideas about the same physical slide, and ultimately lead to the content of the pathology report. Inconsistent. 2. It is not easy for multiple people to read together: limited to the physical slide and microscope allowing only one person to view at the same time, manual interpretation cannot achieve multiple people watching together, and it cannot ensure that multiple people can focus on the same feeling when viewing in turn. Area of interest. 3. Diversity of pathological features: With the advancement of medical technology, more and more pathological features have been discovered. For the rapidly increasing pathological features, experts must spend a lot of time to learn new pathological features in addition to heavy work, so as to correctly focus on the region of interest where the pathological features are located, which causes a serious burden on the experts.

Therefore, the existing methods for manual interpretation of physical slides have the aforementioned problems, and more effective solutions are urgently needed.

The present invention provides a system and method for generating training data through question and answer, which can prompt a region of interest when a user browses an image, and collect training data of the region of interest through question and answer.

The present invention provides a method for generating training data through question and answer, which includes the following steps: providing a target image for browsing on a platform side; accepting an operation to adjust a display field of view of the target image; presenting a problem for analyzing the display field of view, And obtain a response answer; and, associate the display field of view with a corresponding question and answer set to generate a training data for automatically analyzing the target image, wherein the question and answer set includes the question and the response answer.

The present invention also provides a system for generating training data through question and answer, including a database and a platform, the database is used to store a target image, the platform is connected to the database, and connected to a client via a network, the The platform is configured to provide a browsing interface on the client to browse the target image, accept operations from the client to adjust a display field of view of the target image, present a problem for analyzing the display field of view, and obtain a response Answer, and correlate the display field of view with a corresponding question and answer set to generate a training data for automatically analyzing the target image, wherein the question and answer set includes the question and the response answer.

The invention can automatically prompt the region of interest, and greatly reduces the burden on the user for collecting training data.

The technical scheme of the present invention will be described in detail below in conjunction with the drawings and specific embodiments to further understand the purpose, scheme and effect of the present invention, but it is not intended as a limitation of the scope of the attached patent application of the present invention.

The automatic interpretation of artificial intelligence (AI) images must use a large amount of training data to train the learning model.

During the browsing of the images, the expert can manually interpret and label a large number of images one by one, and then generate training data based on these marks, but this makes the expert must interrupt the browsing and spend a lot of time marking and annotating.

Take the interpretation of medical images (such as digital pathology images) as an example. When a doctor is interpreting medical images (such as judging whether there are cancer cells), when a cancer cell is found, the interpretation must be interrupted to mark and annotate the cancer cells. It will cause great inconvenience when explaining or teaching with patients.

In this regard, the present invention mainly provides a system and method for generating training data through question and answer. The main principle of the present invention is that one or more regions of interest (i.e., the display field of view described below) of an image to be analyzed (i.e., the target image described later) are set in advance (using artificial or AI technology) to problems related to this area.

Moreover, in the process of interpreting the target image by the user (for example, a physician, expert or examiner, etc.), when browsing to a region of interest for which a problem has been set, the present invention will automatically present the problem, and the user can know that the current display field of view is of interest Area, and can be more focused on the observation of the current display field of view. In addition, users only need to simply answer questions related to the current area to complete the input of the training parameters of the area of interest. Since the interpretation will not be interrupted, it will not take too much time, which will increase the user’s willingness to give feedback and improve training. The accuracy of the data.

In addition, since the present invention eliminates the need for manual marking and annotation by the user, the burden on the user can be greatly reduced, so that the user can concentrate more on the interpretation and improve the accuracy of the interpretation.

Moreover, the present invention raises questions related to the current browsing area, so that the user can pay attention to the observation points related to the question when answering the question, thereby improving the accuracy of interpretation.

Please refer to FIG. 1, which is an architecture diagram of a system according to an embodiment of the present invention. The system for generating training data through question and answer of the present invention may include a platform terminal 10 (for example, a server, a cloud service platform, a desktop computer, a notebook computer, or any combination thereof), a database 12 (for example, a network database , Local database, relational database and other databases or their combination) and AI module 13.

The platform 10 may include a storage module 101, a communication module 102, a human-machine interface 103, and a processing module 100.

The storage module 101 (for example, a storage device such as RAM, EEPROM, solid-state hard disk, magnetic disk hard disk, flash memory, or any combination thereof) is used for storing data. The communication module 102 (such as a network interface card, NIC) is used to connect to a network (such as the Internet), and through the network and external equipment (such as database 12, AI module 13, and/or client 11) communication. The man-machine interface 103 (including an input interface and an output interface, such as a mouse, a keyboard, various keys, a touch panel, a display, a touch screen, a projection module, etc.) is used to interact with the user. The processing module 100 (which can be a processor such as a CPU, GPU, TPU, MCU, or any combination thereof) is used to control the platform side and implement the functions proposed by the present invention.

The AI module 13 can be built on a server or a cloud service platform (for example, Amazon Web Service, Google Cloud Platform, Microsoft Azure, etc.), or on the platform side 10. The AI module 13 includes a learning model 130. The training data generated by the present invention is used to establish and train the learning model 130 to improve the accuracy of the learning model 130.

The learning model 130 is established and trained based on machine learning technology, and can automatically analyze the input image (such as the target image described later) to generate predictive knowledge information of a specific display field of the input image. The aforementioned predictive knowledge information can be used as reference information when the user analyzes the input image.

In one embodiment, the learning model 130 includes a VQA (Visual Question Answering, visual question answering) architecture model, which can train text features (question and answer sets, target information, etc.) based on the image features (display field of view) of the input image.

In another embodiment, the learning model 130 includes a DQN (Deep Q-Learning, deep Q-Learning) architecture model, which can refer to the user's image browsing process. After each display field of view is adjusted, the target quantitative value (target information) Make adjustments and finally generate predictive target information for AI automatic browsing images. Specifically, the DQN architecture model can be used to first generate a score (Q-Value) for each browsing action of the input image (which can be a browsing action input by the user or a predicted browsing action automatically generated), and simulate the browsing action with the highest score. Browse the input image to change the display field of view, and input the converted image of the display field of view into the DQN architecture model again to obtain the next browsing action and its display field of view, and so on, until the browsing action is stopped, and can be based on the display field of view (It can be either of the converted display field of view or the last display field of view) image analysis predicts target information.

For example, please refer to FIGS. 3 to 5. FIG. 3 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention, FIG. 4 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention, and FIG. 5 is A schematic diagram of the input and output of the learning model according to an embodiment of the invention.

In one embodiment, as shown in FIG. 3, when a display field image of the target image (that is, a specific sub-image of the target image) is input to the learning model 130, the learning model 130 can automatically analyze and generate prediction target information ( Such as through the DQN architecture model). The aforementioned target information is the analysis result generated after analyzing the image characteristics of the display field, and is used to describe the characteristics of the image characteristics, such as the type of the image characteristics (for example, the type of cell lesions), the range (such as the proportion) or the degree (For example, severe, minor, no impact), etc.

In one embodiment, as shown in FIG. 4, when an image of a display field of view of the target image and a question corresponding to the display field are input to the learning model 130, the learning model 130 can automatically analyze and generate a predicted answer (such as through VQA architecture model). Furthermore, the learning model 130 can also generate new questions about the display field of view, so that when the user browses the same display field of view next time, the user can answer questions (including new questions) to strengthen the training of the learning model 130 for the display field of view. Spend.

In one embodiment, as shown in FIG. 5, when an image of a display field of view of the target image is input to the learning model 130, the learning model 130 can automatically analyze and generate a predicted action course (for example, through a DQN architecture model). Specifically, the learning model 130 may be able to summarize the action history of one or more users on the target image (that is, the combination of multiple browsing actions on the target image) added during the training process of the learning model 130 It is suitable for the browsing method of this target image and provides inexperienced users as a reference.

Please refer to FIGS. 1, 2 and 6 together. FIG. 2 is a partial architecture diagram of a platform side according to an embodiment of the present invention, and FIG. 6 is a flowchart of a method according to an embodiment of the present invention.

In one embodiment, the processing module 100 may include modules 104-105, 20-22, 200-202, 210-213, 220-221, and the AI module 13 may include modules 131-132. These modules are configured to perform different functions.

The aforementioned modules are mutually connected (electrical connection and information connection), and can be hardware modules (such as electronic circuit modules, integrated circuit modules, SoC, etc.), software modules (such as firmware System, operating system, or application) or a mix and match of software and hardware modules, which are not limited.

It is worth mentioning that when the aforementioned modules are software modules (for example, firmware, operating systems, or applications), the storage module 101 and the AI module 13 may include non-transitory computer-readable recording media. The non-transitory computer readable recording medium stores a computer program, and the computer program records a computer executable program code. When the processing module 100 and the AI module 13 execute the aforementioned program code, the function of the corresponding module can be realized.

The method of generating training data through question and answer in this embodiment includes the following steps.

Step S10: Provide browsing of the target image through the image browsing and recording module 20 on the platform side 10. The aforementioned target image can be stored in the database 12 or uploaded by the user.

Specifically, the platform terminal 10 may provide a browsing interface (such as a GUI), and display the target image in the browsing interface.

In one embodiment, the user can operate the man-machine interface 110 (similar to the man-machine interface 103) of the user terminal 11 (for example, a desktop computer, a notebook computer, a tablet, a smart phone, etc.) To connect to the platform terminal 10 to browse the target image using the browsing service provided by the platform terminal 10.

In one embodiment, the user can directly operate the man-machine interface 103 of the platform terminal 10 to use the browsing service.

Step S11: The platform terminal 10 accepts the user's browsing operation (which can be a local operation or a remote operation) through the image browsing and recording module 20 to adjust the display field of view of the target image, such as zooming in, zooming out, translation, and rotation.

Step S12: After changing the display field of view, the platform terminal 10 obtains the problem corresponding to the current display field of view through the knowledge acquisition module 21 and presents the problem. The aforementioned question is used to analyze the current display field of view.

Then, after reading the question, the user can refer to the current display field of view and past experience, and quickly input the response answer to answer the question, so that the platform terminal 10 obtains the response answer to the question. In this way, the present invention can obtain a set of question and answer sets (including questions and response answers) of the current display field of view, that is, obtain the training parameters of the current display field of view.

Step S13: The platform side uses the knowledge record and processing module 22 to display the field of view and its corresponding question and answer set in association with each other to generate training data. The aforementioned training data is used to provide the learning model 13 for training of automatically analyzing the target image.

In this way, the user only needs to simply answer the questions during the browsing process of the target image, and the present invention can automatically create the training data of the target image.

Please refer to FIGS. 1-2, FIGS. 7A-7B, and FIGS. 9-11 together. FIG. 7A is a flowchart of the first part of a method according to an embodiment of the invention, and FIG. 7B is a flowchart of the second part of a method according to an embodiment of the invention 9 is a schematic diagram of a display field of view of a target image according to an embodiment of the present invention, FIG. 10 is a schematic diagram of another display field of view of FIG. 9, and FIG. 11 is a schematic diagram of another display field of view of FIG. 9.

In this embodiment, the platform 10 provides a web browsing service through a web module 104 (such as a web server module) to interact with the client 11.

The method of generating training data through question and answer in this embodiment includes the following steps.

Step S20: The platform terminal 10 provides a browsing interface on the client terminal 11 through the web module 104 to provide a web service for browsing the target image.

In one embodiment, step S20 may include the following steps S30-S34.

Step S30: The client 11 logs in to the platform 10 through a webpage.

In one embodiment, the database 12 may include a user database 123, and the user database 123 is used to store the registration information of different users (or the client 11) (for example, account password, network address, hardware address, device Identification code, digital signature, Dongle Key and other identifiable information). The user can operate the client terminal 11 to provide login information (for example, account password, network address, hardware address, device identification code, digital signature, Dongle Key and other identifiable information) to the platform terminal 10.

Then, the platform terminal 10 compares the login information with each registered information through the management module 105, and determines that the authentication is passed and the login is allowed when the login information matches any of the registered information.

Step S31: After successfully logging in, the platform terminal 10 provides a browsing interface (such as a web program) to the client terminal 11 through the web module 104 to present the browsing interface.

In one embodiment, the database 12 includes an image library 120, and the image library 120 stores multiple images, such as electronic component images, medical images, monitor images, or other images that need to be interpreted.

Step S32: The client 11 can select one of the multiple images from the image library 120 as the target image (ie, the image to be interpreted next) through the webpage, or upload the image to the platform 10 as the target image.

Then, the platform terminal 10 imports the target image through the image viewing module 200.

Step S33: The platform terminal 10 uses the image information processing module 202 to determine the display field of view of the target image to be presented according to preset field of view parameters (such as a preset display reference point, zoom level, and/or rotation angle), that is, determine the desired field of view. The range of the target image presented.

The aforementioned display reference point of the display field of view may be an image coordinate (such as a pixel position), which represents the position of the reference point (such as the image center point or any image boundary point) of the display field of view in the entire target image. The aforementioned zoom level of the display field of view refers to the current magnification or reduction level of the display field (this value will correspond to the actual magnification or reduction in proportion to images of different resolutions). The aforementioned rotation angle of the display field of view refers to the angle difference of the display field of view with respect to the target image.

For example, the platform end 10 can be set to: when the zoom level is 1, the actual magnification is 5 times; when the zoom level is 5, the actual magnification is 40 times.

In another example, the platform end 10 can be set to: when the zoom level is 1, the actual magnification is 0.25 times (such as zooming out so that the entire target image enters the display field of view); when the zoom level is 5, the actual magnification is 1 Times (e.g. browsing in equal proportions).

The relationship (such as the ratio) between the zoom level and the actual magnification ratio of the present invention can be determined according to requirements and is not limited.

Thereby, based on the display reference point and zoom level, or the display reference point, zoom level, and rotation angle, the image information processing module 202 can determine the field of view, such as calculating the field of view with a bounding box.

Step S34: The platform terminal 10 transmits the determined image of the display field of view to the client 11 through the network, so as to present the display field of view on the browsing interface.

Step S21: The client 11 adjusts the display field of view of the target image through the browsing interface.

In one embodiment, step S21 may include the following steps S40-S42.

Step S40: The user terminal 11 inputs a browsing operation through the browsing interface to adjust the display field of view, and the platform side 10 collects the user's browsing operation through the image operation module 201 and converts it into the adjustment amount of each field of view parameter (that is, determines the adjusted field of view parameter) .

In one embodiment, the browsing interface can display various operation buttons (such as move left, move right, move up, move down, zoom in, zoom out, rotate clockwise, rotate counterclockwise, etc.), and the image operation module 201 monitors these operation buttons To collect the user's browsing operations.

In one embodiment, the image operation module 201 can monitor the keyboard or mouse actions of the client 11 to collect the user's browsing operations (such as monitoring mouse or keyboard signals to trigger adjustment of the zoom level (at zoom ratio), panning display Datum point and rotating image).

Step S41: The platform terminal 10 determines a new display field of view based on the adjusted field of view parameters through the image information processing module 202, that is, determines the new range of the target image to be presented.

Step S42: The platform terminal 10 outputs an image of the adjusted display field of view through the network and transmits it to the client terminal 11 to present the adjusted display field of view on the browsing interface.

In one embodiment, the platform 10 may perform image compression (especially distorted image compression) before transmitting the image of the display field of view to the client 11 to appropriately reduce the image details and reduce the resolution, thereby appropriately reducing the size of the image It is conducive to network transmission.

It is worth mentioning that when the target image is too large (for example, the resolution is too high), it takes a lot of transmission time to send the entire target image to the client 11, and the client 11 may not be able to perform smoothly due to insufficient hardware performance. Browse the entire target image.

In this regard, the present invention only transmits the image of the display field of view (and can be appropriately compressed) to the client 11, which can ensure that the client 11 can browse the entire target image smoothly. Moreover, when the user terminal 11 wants to view the details of the image, it can perform an enlarged browsing operation, so that the platform terminal 10 can return the enlarged image of the area for smooth browsing, which can improve the user experience.

In one embodiment, the target image can be divided into multiple tiles, and the platform end 10 uses the tiles as the smallest transmission unit.

For example, as shown in FIG. 9, the display reference point is the image center, the target image 30 is divided into 48 tiles, and each tile has a corresponding identification code (such as coordinates).

In addition, the display field of view of the preset field of view parameters may be a full range, that is, the display reference point 40 is (4, 3), the magnification level is 1, and the rotation angle is 0 degrees. The platform terminal 10 transmits all the tiles (48 in total) in the display field of view (full range) to the client terminal 11 so as to display the complete target image 30 on the browsing interface.

Then, when the user wants to adjust the display field of view to the range 50, he can input a browsing operation. As shown in FIG. 10, the adjusted display reference point 41 is (2.5, 3.5), the magnification level is 2, and the rotation angle is 0 degrees. The platform end 10 transmits all the tiles (15 in total) in the adjusted display field of view to the user end 11, so that the adjusted display field of view is displayed on the browsing interface.

Then, when the user wants to adjust the display field of view to the range 51, he can input a browsing operation. As shown in FIG. 11, the adjusted display reference point 42 is (1.5, 4.5), the magnification level is 3, and the rotation angle is 0 degrees. The platform terminal 10 transmits all the tiles (6 in total) in the adjusted display field of view to the user terminal 11, so that the adjusted display field of view is displayed on the browsing interface.

Through the above method, the present invention can effectively provide the browsing service of the target image.

Then step S22 is performed: the platform terminal 10 judges through the knowledge acquisition module 21 whether there is a corresponding question for the current display field to be answered by the user, if there is a question and answer function, if there is no corresponding question, no question and answer is performed.

In one embodiment, the database 12 includes a knowledge base 121. The knowledge base 121 can store knowledge information corresponding to different display fields of the target image. The aforementioned knowledge information may include multiple questions to be answered, multiple question and answer sets (answered questions), multiple target information and/or action information, etc.

The aforementioned questions are pre-set for the corresponding image characteristics of the display field of view (such as the shape, type, color, change, etc. characteristics of the object in the display field of view), or automatically generated through machine learning (Figure 4), and It is used to analyze image characteristics.

In one embodiment, each display field of view is associated with a set of identification codes (which can be determined based on the identification codes of the covered tiles, determined based on the corresponding field of view parameters, or serially numbered, etc., which are not limited). Moreover, the knowledge information related to the display field of view is all corresponding to the same identification code.

In one embodiment, step S21 may include the following steps S40-S42.

Step S40: The platform side 10 obtains the corresponding question from the knowledge base 121 based on the identification code of the current display field through the question and answer module 210.

Step S41: The platform 10 displays the obtained questions on the browsing interface of the client 11 through the question and answer module 210, and can provide an answer interface for the user to answer.

Step S42: The platform terminal 10 receives the response answer input by the user through the answer interface through the question and answer module 210.

For example, referring to FIG. 10, when the user browses to the display field of view of FIG. 10, the question and answer module 210 can display the question and answer interface 60 or 61.

Take the question and answer interface 60 as an example. The question is an open question (such as a question and answer). The question and answer interface 60 includes a text input area, and the user enters text in the text input area as a response answer.

Other open-ended questions and answers can be, for example: "Q: Which organ system is abnormal in the image? A: Cardiovascular", "Q: What are the bile duct cells and Hexagonal tubules stained with cytokeratin 7 here? A: Immunohistochemical staining ", "Question: How to connect the electronic components in the picture? Answer: Soldering", etc., there is no limitation.

Taking the question and answer interface 61 as an example, the question is a closed question (such as a multiple-choice question), the question and answer interface 60 includes multiple answer options, and the user selects one or more answer options as a response answer.

Other closed question and answer can be for example: "Q: Is the tissue protein subunits positively charged to make the negatively charged DNA more compact and stable? Option: Yes; No", "Q: Is this the cerebral embolism area? Option: Yes; No ", "Question: How many electronic components are there in the picture? Options: 1; 2; 3", etc., which are not limited.

In one embodiment, the present invention further provides a target setting function. Specifically, in any display field of view, step S23: the platform terminal 10 can input the interface through the target module 212 to accept the user terminal 11 after viewing the current display field of view The set target information. The aforementioned target information is used as a result of the user's manual analysis of the quantitative features of the full image, and used as the training parameter of the target image.

Take the judgment of the proportion of cancer cells as an example. When experts watch pathological images at low magnification (wider field of view), they may think that the proportion of cancer cells is high (e.g. 80%), and set the target information of this pathological image as "cancer cells". The ratio is 80%". After adjusting the display field of view, such as viewing the pathological image at medium magnification (narrower field of view), it may be found that some of the cells are not cancer cells (e.g. 60%). At this time, the target information of this pathological image can be corrected as "Proportion of Cancer Cells" 60%".

In one embodiment, the present invention further provides a knowledge information prompt function. Specifically, in any display field of view, step S24: the knowledge processing and providing module 213 determines whether the current display field of view corresponds to the knowledge base 121 Knowledge information. If so, the knowledge information is displayed on the browsing interface to explain the display field of view.

In one embodiment, each time the image information processing module 202 determines the display field of view, it can generate image browsing information (such as identification parameters or identification codes) of the display field of view, and the knowledge acquisition module 21 can search for knowledge based on the image browsing information. The library 121 detects whether there is knowledge information related to the display field of view (such as the previously input question and answer set or the target information of the entire target image, or the content automatically interpreted by AI).

For example, please refer to Fig. 11. When the user browses to the display field of view in Fig. 11, the knowledge acquisition module 21 finds that the current display field of view has corresponding knowledge information in the knowledge base (abnormal ratio 70%), and can display the knowledge information于标’s interface 62. In addition, if the user judges that the knowledge information is incorrect, he can also modify it through the input interface (for example, modify it to 90%) to enable the target module 212 to set new target information for learning.

In one embodiment, the present invention further provides a browsing action judgment function. Specifically, each time the display field of view is switched, the action module 211 can obtain the previous display reference point, the previous zoom level or the previous display field of view. A rotation angle, which is compared with the display reference point, zoom level or rotation angle of the current display field of view to determine the browsing action of switching from the previous display field to the current display field of view (e.g. pan direction, pan amount, magnification level, Rotation direction, rotation angle, etc.), and the browsing action is added to the action information, that is, the content of the browsing action is determined based on the changes of the two sets of field of view parameters.

Then, step S25 is performed: the platform terminal 10 uses the training data generation module 221 to associate the display field of view with the corresponding knowledge information (for example, a question and answer set completed by the user, target information and/or action information) to generate training data.

In one embodiment, the present invention further provides a recording function (steps S26-S27).

Step S26: During the browsing process, the platform terminal 10 judges whether the preset recording condition is satisfied through the knowledge recording and processing module 22. The aforementioned recording conditions can be every time the display field of view is changed, after every operation, or timing (such as 10 seconds), and is not limited.

If the recording conditions are not met, continue monitoring.

If the recording conditions are met, step S27 is performed: the platform terminal 10 uses the image capturing module 220 to capture the image of the display field of view based on the aforementioned image browsing information, and the knowledge recording and processing module 22 records the image of the display field of view and knowledge information (such as Questions and answers completed by the user, subject information and/or action information), and associated.

In one embodiment, the database 12 includes browsing records 122. The knowledge recording and processing module 22 associates the visual field parameters and knowledge information of the display field of view, and records them in the browse record 122 as a browsing history (which may include the action history, all the questionnaires and target information showing the field of view).

Please also refer to FIG. 8, which is a flowchart of training and automatic analysis of the method according to an embodiment of the present invention. The present invention further provides a training function (step S60) and an automatic analysis function (step S61).

Step S60: The AI module 13 trains the learning model 130 according to the training data. In one embodiment, step S60 includes steps S70-S73.

Step S70: The AI module 13 loads and obtains the learning model 130.

Step S71: The AI module 13 analyzes the training data through the training module 132.

In one embodiment, the training data includes an image showing the field of view and its corresponding knowledge information (for example, question-and-answer set, action information, target information, etc.). The training module 132 analyzes the image characteristics of the display field of view, and generates text characteristics based on knowledge information (such as questions in the question and answer set and the target information input by the user), that is, the knowledge information is converted to the semantic network.

In one embodiment, the AI module 13 may first convert the training data into a format acceptable to the training module 132 and/or the learning model 130 through the data conversion module 131, and then perform analysis and training.

Step S72: The training module 132 inputs image features and text features to the learning model 130 to generate predictive knowledge information (for example, predictive answers to questions, predictive target information, or predictive action information, etc.).

Step S73: The training module 132 compares the predicted knowledge information with the knowledge information of the training data to train the learning model. For example, comparing the predicted answer with the user's response answer, comparing the predicted target information with the target information input by the user, or comparing the predicted action information with the user's actual browsing action, etc., there is no limitation.

Step S61: The AI module 13 receives image input to generate corresponding prediction knowledge information. In one embodiment, step S61 includes steps S80-S81.

Step S80: The AI module 13 accepts (for example, the client 11 or the platform 10) operation to select an image (another target image).

Step S81: The AI module 13 inputs the target image to the learning model 130 to obtain the predictive knowledge information of the target image (which may include the predictive action history of one or more display fields of the target image, predictive questions, and predictive target information) , And can be stored in the knowledge base 121.

Please refer to FIG. 12, which is a schematic diagram of the action history of an embodiment of the present invention. After completing the history recording of the target image, the platform terminal 10 of the present invention can provide a playback function, displaying the action history 70 of the target image, the display field of view 71 and the thumbnail interface 72 in the browsing interface. The user terminal can select any step of the action history 70 to watch afterwards, or continuously play multiple steps of the action history 70 to reproduce the browsing path during analysis.

Please refer to FIG. 13, which is a schematic diagram of a heat map according to an embodiment of the present invention.

After analyzing the target image, the platform terminal 10 of the present invention can provide a heat map function, which can display which part of the target image is the area of interest of the model during the algorithm analysis process.

As shown in the figure, the platform terminal 10 displays the field of view interface 80 and the thumbnail interface 81 of the target image in the browsing interface. In the display field of view interface 80, areas with a higher degree of confidence in algorithm analysis will be represented by lighter hot areas 801, and areas with lower confidence in algorithm analysis will be represented by darker hot areas 802. In addition, The white area 800 refers to the background area.

In this way, the user or other users can directly understand the importance of each position of the target image, and can directly ignore the browsing of the white hot zone 800 and dark hot zone 802 (unimportant areas) to save time and focus more on light colors Browse hot zone 801 (important area) to improve accuracy.

In the present invention, by prompting questions or knowledge information in a designated display field of view, experts can know that the current display field of view is a region of interest, and focus more on the analysis of the current display field of view.

The present invention provides a browsing service (web module 104) of target images through web pages, allowing experts from all over the world to view the same target images through web browsers, and browse the same target images (including answering questions and setting target images). Information), and different knowledge information (ie training data) of different experts for the same target image can be collected.

In addition, the subsequent collection of these knowledge information is provided to the AI module 13 for training as training data for the learning model 130, so that an expert-verified and highly reliable learning model 130 can be generated. In the future, when other experts (such as pathologists) perform image interpretation, they can first refer to the results of AI module 13 interpretation (such as predictive knowledge information) as the basis for subsequent pathology report writing. Since the reference sources are consistent, the final interpretation results of the experts can be improved Consistency.

The present invention provides the browsing service of the target image through the webpage and can also reduce the transportation time and damage risk of the physical slide. In addition, by prompting the knowledge of the current display field of view and accepting the input of the target information, the expert’s thoughts during the review process can be fully recorded, and can be shared with other experts for viewing, which helps different experts understand each other’s analysis ideas and strategies .

Of course, the present invention can also have various other embodiments. Without departing from the spirit and essence of the present invention, those with ordinary knowledge in the technical field to which the present invention belongs can make various corresponding changes and modifications according to the present invention, but these Corresponding changes and modifications should belong to the scope of the patent application attached to the present invention.

10: Platform side

100: Processing module

101: storage module

102: Communication module

103, 110: Human-machine interface

104: web module

105: Management Module

11: User side

12: Database

120: Image library

121: Knowledge Base

122: browsing history

123: User database

13: AI module

130: learning model

131: Data Conversion Module

132: Training Module

20: Image browsing and recording module

200: Image viewing module

201: Image operation module

202: Image Information Processing Module

21: Knowledge Acquisition Module

210: Q&A Module

211: Action Module

212: target module

213: Knowledge Processing and Providing Module

22: Knowledge Record and Processing Module

220: Image capture module

221: Training data generation module

30-32: Target image

40-42: Display reference point

50-51: range

60-62, 70-72, 80-81: interface

800-802: area

S10-S13: The first training data generation step

S20-S27: Second training data generation step

S30-S34: Display steps

S40-S42: Browse steps

S50-S52: Q&A Steps

S60, S70-S73: training steps

S61, S80-S81: automatic analysis steps

FIG. 1 is an architecture diagram of a system according to an embodiment of the present invention;

2 is a partial architecture diagram of the platform side according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of the input and output of a learning model according to an embodiment of the present invention;

4 is a schematic diagram of the input and output of a learning model according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of the input and output of a learning model according to an embodiment of the present invention;

Figure 6 is a flowchart of a method according to an embodiment of the present invention;

FIG. 7A is a flowchart of the first part of a method according to an embodiment of the present invention;

FIG. 7B is a flowchart of the second part of the method according to an embodiment of the present invention;

FIG. 8 is a flowchart of training and automatic analysis of a method according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a display field of view of a target image according to an embodiment of the present invention;

FIG. 10 is another schematic diagram showing the field of view of FIG. 9;

FIG. 11 is another schematic diagram showing the field of view of FIG. 9;

FIG. 12 is a schematic diagram of the action history of an embodiment of the present invention; and

FIG. 13 is a schematic diagram of a heat map according to an embodiment of the present invention.

S10-S13: The first training data generation step

Claims (20)

A method of generating training data through question and answer, including: a) Provide browsing of a target image on a platform; b) Accept the operation to adjust a display field of view of the target image; c) Present a question used to analyze the display field of view and obtain a response answer; and d) Associating the display field of view with a corresponding question and answer set to generate a training data for automatically analyzing the target image, wherein the question and answer set includes the question and the response answer. The method for generating training data through question and answer as described in claim 1, wherein the step a) includes: a1) After logging in to the platform, select one of a plurality of images from an image library as the target image, or upload an image to the platform as the target image; a2) Determine the display field of view of the target image on the platform side based on at least one of a display reference point, a zoom level and a rotation angle; and a3) Output the image of the display field of view to present the display field of view. The method for generating training data through question and answer as described in claim 1, wherein the target image is divided into a plurality of tiles; the step b) includes: b1) Accept an operation to adjust a field of view parameter, where the field of view parameter includes at least one of a display reference point, a zoom level, and a rotation angle; b2) Determine all the tiles that enter the display field of view based on the adjusted field of view parameters as the adjusted image of the display field of view; and b3) Output the adjusted image of the display field of view to present the adjusted display field of view. The method for generating training data through question and answer as described in claim 1, wherein the step c) includes: c1) Obtain the corresponding question from a database based on an identification code of the display field of view, where the question is preset for an image feature of the display field of view or automatically generated through machine learning, and is used to analyze the Image feature c2) Show the problem; and c3) Receive the response answer. The method for generating training data through question and answer as described in claim 4, wherein the question is a closed question; the step c3) is to display a plurality of answer options, and use the selected answer option as the response answer. The method for generating training data through question and answer as described in claim 4, wherein the question is an open-ended question; the step c3) is to display a text input area, and use the text inputted in the text input area as the response answer. The method of generating training data through question and answer as described in claim 1 further includes: e) When a recording condition is met, capture the image of the display field of view and record a corresponding piece of knowledge information, where the knowledge information includes the question and answer collection and a browsing action; Wherein, the step e) when recording the browsing action is to obtain the previous display reference point, the previous zoom level or the previous rotation angle of the previous display field of view, and the previous display reference point, the previous zoom level or the previous display reference point The previous rotation angle is compared with a display reference point, a zoom level or a rotation angle of the current display field of view to determine the browsing action of switching from the previous display field of view to the current display field of view; The step d) is to associate the display field of view and the knowledge information with an identification code for correlation. The method for generating training data through question and answer as described in claim 1, further includes at least one of the following steps: f) Accept the operation to set a target information, where the target information is an analysis result of an image feature of the target image; and g) When judging that the current display field of view has a corresponding knowledge information in a database, display the knowledge information, wherein the knowledge information is used to explain the display field of view. The method of generating training data through question and answer as described in claim 1, further includes: h1) Obtain a learning model; h2) Analyze an image feature of the display field of view of the training data, and generate a text feature based on the question and answer set; h3) Input the image feature and the text feature to the learning model to generate a predicted answer; and h4) Train the learning model by comparing the predicted answer with the response answer. The method of generating training data through question and answer as described in claim 1, further includes: i1) Accept the operation to select an input image; and i2) Input the input image to a learning model to obtain a predictive knowledge information of the input image. A system for generating training data through question and answer, including: A database for storing a target image; and A platform, connected to the database, and connected to a client via the network, the platform is configured to provide a browsing interface on the client to browse the target image, and accept operations from the client to adjust the target image A display field of view presents a question for analyzing the display field of view, obtains a response answer, and associates the display field of view with a corresponding question and answer set to generate a training data for automatically analyzing the target image, wherein the question set Include the answer to the question and the response. The system for generating training data through question and answer as described in claim 11, wherein the database includes: A user database for storing multiple registration data; and An image library for storing multiple images; Among them, the platform side includes: A web module configured to interact with the client through a web page to receive a login data of the client, and select one of the images in the image library or upload an image from the client as the target Image; and A management module is configured to accept login when the login information matches any of the registration information. As described in claim 11, the system for generating training data through question and answer, wherein the platform side includes: An image viewing module is set to obtain the target image, wherein the target image is divided into a plurality of tiles; An image operation module configured to adjust a field of view parameter according to the operation of the user terminal, wherein the field of view parameter includes at least one of a display reference point, a zoom level, and a rotation angle; and An image information processing module is set to determine all the tiles that enter the display field of view based on the adjusted field of view parameters as the adjusted image of the display field of view, and output the adjusted image of the display field of view to the client , To present the adjusted display field of view on the user side. The system for generating training data through question and answer as described in claim 11, wherein the database includes a knowledge base for storing a plurality of questions corresponding to the different display fields of the target image, and the questions are specific to the display field An image feature is preset or automatically generated through machine learning, and is used to analyze the image feature, and the display field of view and the corresponding question are associated with an identification code; Among them, the platform side includes: A question and answer module is configured to obtain the corresponding question based on the identification code of the current display field of view, display the question on the client, and receive the response answer from the client. The system for generating training data through question and answer as described in claim 14, wherein the question is a closed question or an open question; Among them, the question and answer module is set to display multiple answer options on the client when the question is a closed question, and use the selected answer option as the response answer, and when the question is a closed question , Display a text input area, and use the text content entered in the text input area as the response answer. As described in claim 11, the system for generating training data through question and answer, wherein the platform side includes: An action module is set to obtain the previous display reference point, the previous zoom level or the previous rotation angle of the previous display field of view, and the previous display reference point, the previous zoom level or the previous rotation angle and Compare a display reference point, a zoom level or a rotation angle of the current display field of view to determine a browsing action of switching from the previous display field of view to the current display field of view; and A knowledge recording and processing module is set to capture an image of the display field of view and record a corresponding knowledge information when a recording condition is met, wherein the knowledge information includes the question and answer set and the browsing action, the display field of view and The knowledge information is corresponding to an identification code for association. As described in claim 11, the system for generating training data through question and answer further includes an AI module, and the AI module includes: A learning model for automatically generating at least one predicted browsing action of an input image, simulating browsing of the input image based on the at least one predicted browsing action to transform the display field of view, and image analysis based on the transformed display field of view A prediction target information of the input image; Wherein, the learning model includes a DQN architecture model; Among them, the information of the prediction target is associated with the input image. According to claim 11, the system for generating training data through question and answer, wherein the database includes a knowledge base for storing a plurality of knowledge information corresponding to different display fields, and the knowledge information is used to explain the display field ； Among them, the platform end includes: A target module is configured to accept operations from the client to set target information, where the target information is an analysis result of an image feature of the target image; and A knowledge processing and providing module is set to display the knowledge information on the client when it is determined that the current display field of view has corresponding knowledge information; Wherein, the platform is configured to associate the display field of view with the knowledge information to generate the training data, and the knowledge information includes the question and answer set and the set target information. As described in claim 11, the system for generating training data through question and answer further includes an AI module, and the AI module includes: A learning model for automatically analyzing an input image to automatically generate predictive knowledge information of at least one display field of the input image, wherein the predictive knowledge information includes a predictive action history, a predictive target information, and any display At least one of a predicted question of the field of view and a predicted answer of any one of the displayed fields of view. The system for generating training data through question and answer as described in claim 19, wherein the learning model includes a VQA framework model; Wherein, the AI module further includes a training module configured to analyze an image feature of the display field of view of the training data, and generate a text feature based on a knowledge information of the same display field of view of the training data, and input the image The feature and the text feature are sent to the learning model to generate the predicted answer, and the learning model is adjusted by comparing the predicted answer with the response answer.

Images