TWI832032B - System of generating training data by questions and answers and method thereof - Google Patents

Abstract

A system of generating training data by questions and answers and a method thereof are provided by the disclosure. A platform-end provides the browse of a target image, receives the operation to adjust a display view of the target image, shows a question used to analyze the display view, gets a response answer, and associates the display view with the question and the response answer to generating the training data used to automatically analyze the target image. The disclosure can automatically prompt the region of interest, and significantly reduce the burden to collect the training data to the user.

Description

System and method for generating training data through question and answer

The present invention relates to a system and method for generating training data, and in particular to a system and method for generating training data through question and answer.

For pathological interpretation, manual interpretation is currently performed by experts (such as pathologists). During the manual interpretation process, the experts must use a microscope to view the physical slides, and at the same time use recording pens, paper and pen and other tools to manually record the viewing results. Finally, Write a pathology report based on the viewing results.

However, the above method has the following problems:

1. Inconsistent interpretations by experts: Different experts usually have different reading angles for the same physical slide (that is, the identified areas of interest are different). This will lead to different experts having different ideas about the same physical slide, and ultimately lead to differences in the content of the pathology report. Inconsistent.

2. It is not easy for multiple people to read together: Due to the limitation that the physical slides and microscopes only allow one person to watch at the same time, manual interpretation cannot achieve multiple people watching at the same time, nor can it ensure that multiple people can focus on the same feeling during the viewing process. area of interest.

3. Diversity of pathological characteristics: With the advancement of medical technology, more and more pathological characteristics have been discovered. For the rapidly increasing number of pathological features, experts must allocate a lot of time to learn new pathological features in addition to heavy work in order to correctly focus on the area of interest where the pathological features are located, which causes a serious burden on the experts.

Therefore, the existing manual interpretation of physical slides has the aforementioned problems, and a more effective solution is urgently needed.

The present invention provides a system and method for generating training data through question and answer, which can prompt a region of interest to a user during image browsing, and collect training data for the region of interest through question and answer.

The present invention proposes a method for generating training data through question and answer, which includes the following steps: providing browsing of a target image on a platform; accepting operations to adjust a display field of view of the target image; presenting a question for analyzing the display field of view, and obtain a response answer; and, associate the display field of view with a corresponding question and answer set to generate training data for automatically analyzing the target image, wherein the question and answer set includes the question and the response answer.

The present invention also proposes a system for generating training data through question and answer, including a database and a platform. The database is used to store a target image. The platform is connected to the database and connected to a client through the network. The platform is configured to provide a browsing interface for the client to browse the target image, accept operations from the client to adjust a display field of view of the target image, present a question for analyzing the display field of view, and obtain a response. The answer is generated by associating the display field of view with a corresponding question and answer set to generate training data for automatically analyzing the target image, wherein the question and answer set includes the question and the response answer.

The invention can automatically prompt areas of interest and greatly reduce the burden on users in collecting training data.

10:Platform side

100: Processing module

101:Storage module

102:Communication module

103, 110: Human-computer interface

104:web module

105: Management module

11: Client

12:Database

120:Image library

121:Knowledge Base

122: Browsing history

123:User database

13:AI module

130: Learning model

131:Data conversion module

132:Training module

20:Image browsing and recording module

200:Image viewing module

201:Image operation module

202:Image information processing module

21: Knowledge acquisition module

210: Q&A module

211:Action Module

212:Target module

213: Knowledge processing and provision module

22:Knowledge recording and processing module

220:Image capture module

221: Training data generation module

30-32: Target image

40-42: Display reference point

50-51: Range

60-62, 70-72, 80-81: Interface

800-802: Area

S10-S13: First training data generation step

S20-S27: Second training data generation step

S30-S34: Display steps

S40-S42: Browsing steps

S50-S52: Question and Answer Steps

S60, S70-S73: training steps

S61, S80-S81: Automatic analysis steps

Figure 1 is an architecture diagram of the system according to an embodiment of the present invention; Figure 2 is a partial architecture diagram of the platform side according to an embodiment of the present invention; Figure 3 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention; Figure 4 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention. Figure 5 is a schematic diagram of the input and output of the learning model according to one embodiment of the present invention; Figure 6 is a flow chart of the method according to one embodiment of the present invention; Figure 7A is a schematic diagram of the input and output of the learning model according to one embodiment of the present invention; The first part of the flow chart of the method of the embodiment; Figure 7B is the second part of the flow chart of the method of an embodiment of the present invention; Figure 8 is a flow chart of the training and automatic analysis of the method of an embodiment of the present invention; Figure 9 is a flow chart of the method of the present invention. Figure 10 is a schematic diagram of another display field of view of Figure 9; Figure 11 is a schematic diagram of another display field of view of Figure 9; Figure 12 is an action process of an embodiment of the present invention. schematic diagram; and FIG. 13 is a schematic diagram of a heat map according to an embodiment of the present invention.

The technical solution of the present invention will be described in detail below in conjunction with the drawings and specific embodiments to further understand the purpose, solutions and effects of the present invention, but this is not intended to limit the scope of the patent application attached to the present invention.

Automatic image interpretation by artificial intelligence (AI) must use a large amount of training data to train the learning model.

During image browsing, experts can manually interpret and label a large number of images one by one, and then generate training data based on these labels. However, this requires experts to interrupt browsing and spend a lot of time labeling and annotating.

Take the interpretation of medical images (such as digital pathology images) as an example. During the process of interpreting medical images (such as determining whether there are cancer cells), when a doctor finds cancer cells, he must first interrupt the interpretation to mark and annotate the cancer cells. If at the same time This can cause great inconvenience when explaining or teaching to patients.

In this regard, the present invention mainly provides a system and method for generating training data through question and answer. The main principle of the present invention is to (using manual or AI technology) preset one or more areas of interest (ie, the display field of view to be described later) of the image to be analyzed (ie, the target image to be described later) and set questions related to this area.

Moreover, when the user (such as a doctor, expert or examiner) interprets the target image, when browsing to the area of interest for which the question has been set, the present invention will automatically present the question, and the user can know that the current displayed field of view is the area of interest. area, and you can focus more on the current display field of view. Moreover, the user only needs to simply answer questions related to the current area to complete the input of training parameters for this area of interest. Since the interpretation will not be interrupted, it will not take too much time, thus increasing the user's willingness to give feedback and improving training. Accuracy of data.

Moreover, since the present invention eliminates the need for manual marking and annotation by the user, it can significantly reduce the user's burden, allowing the user to concentrate more on interpretation, thereby improving the accuracy of interpretation.

Furthermore, by raising questions related to the current browsing area, the present invention allows the user to pay attention to the observation points related to the question when answering the question, thereby improving the accuracy of interpretation.

Please refer to FIG. 1 , which is an architectural diagram of a system according to an embodiment of the present invention. The system for generating training data through question and answer of the present invention may include a platform 10 (for example, a server, a cloud service platform, a desktop General-purpose computers such as computers and laptops or any combination thereof), database 12 (for example, a network database, a local database, a relational database and other databases or a combination thereof) and the AI module 13.

The platform 10 may include a storage module 101, a communication module 102, a human-machine interface 103 and a processing module 100.

The storage module 101 (for example, a storage device such as RAM, EEPROM, solid state drive, magnetic hard drive, flash memory or any combination thereof) is used to store data. The communication module 102 (such as a network interface card, NIC) is used to connect to a network (such as the Internet) and communicate with external devices (such as the database 12, AI module 13 and/or client 11) through the network Communication. The human-machine interface 103 (including input interface and output interface, such as mouse, keyboard, various buttons, touch pad, display, touch screen, projection module, etc.) is used to interact with the user. The processing module 100 (which can be a processor such as CPU, GPU, TPU, MCU or any combination thereof) is used to control the platform and implement the functions proposed by the present invention.

The AI module 13 can be built on a server or cloud service platform (such as Amazon Web Service, Google Cloud Platform or Microsoft Azure, etc.), or on the platform side 10 . AI module 13 includes learning model 130 . The training data generated by the present invention is used to establish and train the learning model 130 to improve the accuracy of the learning model 130.

The learning model 130 is established and trained based on machine learning technology, and can automatically analyze the input image (such as the target image described below) to generate predictive knowledge information for a specific display field of view of the input image. The aforementioned prediction knowledge information can be used as reference information when the user analyzes the input image.

In one embodiment, the learning model 130 includes a VQA (Visual Question Answering, visual question answering) architecture model, which can train text features (question and answer sets, target information, etc.) based on the image features (display field of view) of the input image.

In another embodiment, the learning model 130 includes a DQN (Deep Q-Learning, Deep Q Learning) architecture model, which can refer to the user's image browsing process and obtain the quantified value of the target (target information) after each display field of view adjustment. Make adjustments and finally generate an AI automatically browsed image Forecast target information. Specifically, the DQN architecture model can first be used to generate a score (Q-Value) for each browsing action of the input image (which can be a browsing action input by the user or an automatically generated predicted browsing action), and the simulation is based on the browsing action with the highest score. Browse the input image to change the display field of view, and re-input the transformed display field of view image to the DQN architecture model to obtain the next browsing action and its display field of view, and so on until the browsing action is stopped and the display field of view can be based on (It can be any converted display field of view or the last display field of view) image analysis prediction target information.

For example, please refer to Figures 3 to 5. Figure 3 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention. Figure 4 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention. Figure 5 is a schematic diagram of the input and output of the learning model according to an embodiment of the present invention. A schematic diagram of the input and output of a learning model according to an embodiment of the invention.

In one embodiment, as shown in FIG. 3 , after an image displaying the field of view of the target image (ie, a specific sub-image of the target image) is input to the learning model 130 , the learning model 130 can automatically analyze and generate predicted target information ( Such as through the DQN architecture model). The aforementioned target information is the analysis result generated by analyzing the image features of the display field of view, and is used to describe the characteristics of this image feature, such as the type of image feature (such as the type of cell lesions), range (such as proportion) or degree (For example, serious, minor, no impact), etc.

In one embodiment, as shown in FIG. 4 , after an image of a display field of view of the target image and a question corresponding to the display field of view are input to the learning model 130 , the learning model 130 can automatically analyze and generate a predicted answer (such as through VQA architecture model). Furthermore, the learning model 130 can also generate new questions about this display field of view, so that when the user browses the same display field of view next time, he can answer questions (including new questions) to strengthen the training of the learning model 130 for this display field of view. Spend.

In one embodiment, as shown in FIG. 5 , after an image of a display field of view of the target image is input to the learning model 130 , the learning model 130 can automatically analyze and generate a predicted action process (such as through a DQN architecture model). Specifically, the action history of one or more users on the target image (that is, the combination of multiple browsing actions on the target image) added during the training process of the learning model 130 can be used to obtain The learning model 130 can summarize the browsing method suitable for this target image and provide inexperienced users as a reference.

Please refer to Figures 1, 2 and 6 together. Figure 2 is a partial architecture diagram of the platform side according to an embodiment of the present invention, and Figure 6 is a flow chart of a method according to an embodiment of the present invention.

In one embodiment, the processing module 100 may include modules 104-105, 20-22, 200-202, 210-213, and 220-221, and the AI module 13 may include modules 131-132. These modules are each configured to perform different functions.

The aforementioned modules are interconnected (can be electrical connections and information connections), and can be hardware modules (such as electronic circuit modules, integrated circuit modules, SoC, etc.), software modules (such as firmware modules, etc.) body, operating system or application) or a mix and match of software and hardware modules, without limitation.

It is worth mentioning that when the aforementioned module is a software module (such as firmware, operating system or application), the storage module 101 and the AI module 13 may include a non-transitory computer-readable recording medium, as mentioned above The non-transitory computer-readable recording medium stores computer programs, and the computer programs record computer-executable program codes. When the processing module 100 and the AI module 13 execute the aforementioned program codes, the functions of the corresponding modules can be realized.

The method of generating training data through question and answer in this embodiment includes the following steps.

Step S10: Provide browsing of the target image through the image browsing and recording module 20 on the platform end 10. The aforementioned target images may be stored in the database 12 or uploaded by the user.

Specifically, the platform 10 can provide a browsing interface (such as a GUI) and display the target image in the browsing interface.

In one embodiment, the user can operate the human-machine interface 110 (similar to the human-machine interface 103 and will not be described again here) of the client 11 (for example, a desktop computer, a notebook computer, a tablet, a smartphone, or other general-purpose computers). To connect to the platform terminal 10, to use the browsing service provided by the platform terminal 10 to browse the target image.

In one embodiment, the user can directly operate the human-machine interface 103 of the platform 10 to use the browsing service.

Step S11: The platform 10 accepts the user's browsing operation (which can be a local operation or a remote operation) through the image browsing and recording module 20 to adjust the display field of view of the target image, such as zooming in, zooming out, panning, rotating, etc.

Step S12: After changing the display field of view, the platform 10 obtains the problem corresponding to the current display field of view through the knowledge acquisition module 21, and presents the problem. The aforementioned questions are used to analyze the current display field of view.

Then, after the user has finished reading the question, he or she can quickly input a response answer to answer the question by referring to the current display field of view and past experience, so that the platform terminal 10 obtains the response answer to the question. Through this, the present invention can obtain a set of question and answer sets (including questions and responses) of the current display field of view, that is, obtain the training parameters of the current display field of view.

Step S13: The platform associates the display field of view with its corresponding question and answer set through the knowledge recording and processing module 22 to generate training data. The aforementioned training data is used to provide the learning model 130 for training in automatically analyzing target images.

In this way, the user only needs to simply answer questions during the browsing process of the target image, and the present invention can automatically create training data for the target image.

Please refer to Figures 1-2, Figures 7A-7B, and Figures 9-11 together. Figure 7A is the first part of the flow chart of the method according to one embodiment of the present invention, and Figure 7B is the second part of the flow chart of the method according to one embodiment of the present invention. FIG. 9 is a schematic diagram of a display field of view of a target image according to an embodiment of the present invention. FIG. 10 is a schematic diagram of another display field of view of FIG. 9 . FIG. 11 is a schematic diagram of another display field of view of FIG. 9 .

In this embodiment, the platform 10 provides web browsing services through a web module 104 (such as a web server module) to interact with the client 11 .

The method of generating training data through question and answer in this embodiment includes the following steps.

Step S20: The platform 10 provides a browsing interface to the client 11 through the web module 104 to provide a web service for browsing target images.

In one embodiment, step S20 may include the following steps S30-S34.

Step S30: The client 11 logs into the platform 10 through the web page.

In one embodiment, the database 12 may include a user database 123. The user database 123 is used to store registration information (such as account passwords, network addresses, hardware addresses, devices) of different users (or clients 11). Identification code, digital signature, Dongle Key and other identifiable information). The user can operate the client 11 to provide login information (such as account password, network address, hardware address, device identification code, digital signature, Dongle Key and other identifiable information) to the platform 10 .

Then, the platform end 10 compares the login information with each registration information through the management module 105, and determines that the verification is passed and login is allowed when the login information matches any registration information.

Step S31: After successful login, the platform 10 provides a browsing interface (such as a web program) to the client 11 through the web module 104 to present the browsing interface.

In one embodiment, the database 12 includes an image library 120 that stores multiple images, such as electronic component images, medical images, monitor images, or other images that need to be interpreted.

Step S32: The client 11 can select one of the multiple images from the image library 120 as the target image (ie, the image to be interpreted next) through the web page, or upload the image to the platform 10 as the target image.

Then, the platform 10 imports the target image through the image viewing module 200 .

Step S33: The platform 10 uses the image information processing module 202 to determine the display field of view of the target image to be presented based on the preset field of view parameters (such as the preset display reference point, zoom level and/or rotation angle), that is, determines the display field of view to be displayed. The range of target images presented.

The display reference point of the aforementioned display field of view can be an image coordinate (such as a pixel position), which represents the position of the reference point of this display field of view (such as the image center point or any image boundary point) in the entire target image. Set. The aforementioned zoom level of the display field of view represents the current magnification or reduction level of the display field of view (this value will correspond to the actual magnification or reduction ratio in proportion to the images of different resolutions). The aforementioned rotation angle of the display field of view represents the angle difference between the display field of view and the target image.

For example, the platform 10 can be set as follows: when the zoom level is 1, the actual magnification is 5 times; when the zoom level is 5, the actual magnification is 40 times.

In another example, the platform 10 can be set as follows: when the zoom level is 1, the actual magnification is 0.25 times (such as zooming out so that the entire target image enters the display field of view); when the zoom level is 5, the actual magnification is 1 times (such as equal-proportion browsing).

The relationship (such as the ratio) between the zoom level and the actual magnification of the present invention can be determined according to the requirements and is not limited.

Thereby, based on the display reference point and zoom level, or the display reference point, zoom level and rotation angle, the image information processing module 202 can determine the field of view range, such as calculating the field of view range using a bounding box.

Step S34: The platform 10 transmits the image of the determined display field of view to the client 11 through the network, so as to present the display field of view on the browsing interface.

Step S21: The client 11 adjusts the display field of view of the target image through the browsing interface.

In one embodiment, step S21 may include the following steps S40-S42.

Step S40: The user terminal 11 inputs browsing operations through the browsing interface to adjust the display field of view. The platform terminal 10 collects the user's browsing operations through the image operation module 201 and converts them into adjustment amounts of each field of view parameter (i.e., determines the adjusted field of view parameters). .

In one embodiment, the browsing interface can display various operation buttons (such as left move, right move, up move, down move, zoom in, zoom out, clockwise rotation, counterclockwise rotation, etc.), and the image operation module 201 monitors these operation buttons. To collect user browsing operations.

In one embodiment, the image operation module 201 can monitor the keyboard or mouse movements of the client 11 to collect the user's browsing operations (such as monitoring the mouse or keyboard signals to adjust the magnification level (in zoom ratio), panning display, etc. datum point and rotated image).

Step S41: The platform 10 determines a new display field of view based on the adjusted field of view parameters through the image information processing module 202, that is, determines a new range of the target image to be presented.

Step S42: The platform terminal 10 outputs the image of the adjusted display field of view through the network and sends it to the client 11, so as to present the adjusted display field of view on the browsing interface.

In one embodiment, before transmitting the image of the display field of view to the user end 11, the platform 10 may first perform image compression (especially distortion image compression) to appropriately reduce image details and resolution, thereby appropriately reducing the size of the image. And it is beneficial to network transmission.

It is worth mentioning that when the target image is too large (such as the resolution is too high), it will take a lot of transmission time to send the entire target image to the client 11, and the client 11 may not be able to smoothly transmit the target image due to insufficient hardware performance. Browse the entire target image.

In this regard, the present invention only transmits the image of the display field of view (and can be moderately compressed) to the user terminal 11, which can ensure that the user terminal 11 can browse the entire target image smoothly. Moreover, when the user terminal 11 wishes to view the details of the image, it can perform a magnified browsing operation, so that the platform terminal 10 returns the enlarged image of the area for smooth browsing, which can improve the user experience.

In one embodiment, the target image can be divided into multiple tiles, and the platform end 10 uses tiles as the smallest transmission unit.

For example, as shown in Figure 9, the display reference point is the center of the image, the target image 30 is divided into 48 tiles, and each tile has a corresponding identification code (such as coordinates).

Moreover, the display field of view of the preset field of view parameter may be the full range, that is, the display reference point 40 is (4,3), the magnification level is 1, and the rotation angle is 0 degrees. Platform 10 will display field of view (full range) All the tiles (48 in total) are sent to the client 11, so that the complete target image 30 is displayed in the browsing interface.

Then, when the user wants to adjust the display field of view to the range 50, he or she can input a browsing operation. As shown in Figure 10, the adjusted display reference point 41 is (2.5, 3.5), the magnification level is 2, and the rotation angle is 0 degrees. The platform terminal 10 will transmit all the tiles (a total of 15 tiles) in the adjusted display field of view to the client 11, so as to display the adjusted display field of view in the browsing interface.

Next, when the user wishes to adjust the display field of view to the range 51, a browsing operation can be input. As shown in Figure 11, the adjusted display reference point 42 is (1.5, 4.5), the magnification level is 3, and the rotation angle is 0 degrees. The platform terminal 10 will transmit all the tiles (a total of 6) in the adjusted display field of view to the client 11, so as to display the adjusted display field of view in the browsing interface.

Through the above method, the present invention can effectively provide browsing services of target images.

Next, step S22 is performed: the platform 10 determines through the knowledge acquisition module 21 whether there are corresponding questions for the user to answer in the current display field of view. If so, the question and answer function is executed. If there is no corresponding question, the question and answer function is not executed.

In one embodiment, the database 12 includes a knowledge base 121 . The knowledge base 121 may store knowledge information corresponding to different display views of the target image. The aforementioned knowledge information may include multiple questions to be answered, multiple question and answer sets (answered questions), multiple target information and/or action information, etc.

Each of the aforementioned questions is preset for the image characteristics of the corresponding display field of view (such as the shape, type, color, change and other characteristics of objects in the display field of view), or is automatically generated through machine learning (as shown in Figure 4), and It is used to analyze image features.

In one embodiment, each display field of view is associated with a set of identification codes (which can be determined based on the identification codes of the covered tiles, determined based on the corresponding field of view parameters, or sequentially numbered, etc., without limitation). Moreover, the knowledge information related to the display field of view is all mapped to the same identification code.

In one embodiment, step S21 may include the following steps S40-S42.

Step S40: The platform 10 uses the question and answer module 210 to obtain the corresponding question from the knowledge base 121 based on the identification code of the current display field of view.

Step S41: The platform 10 displays the obtained questions on the browsing interface of the client 11 through the question and answer module 210, and provides an answer interface for the user to answer.

Step S42: The platform 10 receives the response answer input by the user through the answer interface through the question and answer module 210.

For example, please refer to Figure 10. When the user browses to the display view of Figure 10, the question and answer module 210 can display the question and answer interface 60 or 61.

Taking the question and answer interface 60 as an example, the question is an open-ended question (such as a question and answer question). The question and answer interface 60 includes a text input area, and the user inputs text content in the text input area as a response answer.

Other open-ended questions and answers could be, for example: "Q: Which organ system is abnormal in the image? A: Cardiovascular", "Q: What are the bile duct cells and Heschmann's tubules stained with cytokeratin 7 here? A: Immunohistochemistry staining ”, “Q: How are the electronic components in the picture connected? Answer: Soldering”, etc., are not limited.

Taking the question and answer interface 61 as an example, the question is a closed question (such as a multiple-choice question). The question and answer interface 60 includes multiple answer options, and the user selects one or more answer options as a response answer.

Other closed-ended questions and answers may be, for example: "Q: Do histone subunits have positive charges that make negatively charged DNA more compact and stable? Options: Yes; No", "Q: Is this a cerebral embolism area? Options: Yes; No ", "Q: How many electronic components are there in the picture? Options: 1; 2; 3", etc., without limitation.

In one embodiment, the present invention further provides a target setting function. Specifically, in any display field of view, step S23: the platform 10 can input the interface through the target module 212 to accept the user terminal 11 after viewing the current display field of view. The target information set. The aforementioned target information is the result of the user's manual analysis of the quantitative features of the full image and serves as the training parameter for this target image.

Take judging the proportion of cancer cells as an example. When experts view pathological images at low magnification (wide field of view), they may think that the proportion of cancer cells is high (such as an estimated 80%), and set the target information of this pathological image as "cancer cells." The ratio is 80%.” After adjusting the display field of view, if you view the pathological image at medium magnification (narrow field of view), you may find that some cells are not cancer cells (e.g., an estimated 60%). At this time, you can modify the target information of this pathological image to "proportion of cancer cells." 60%".

In one embodiment, the present invention further provides a knowledge information prompting function. Specifically, in any display field of view, step S24: the knowledge processing and providing module 213 determines whether the current display field of view has a corresponding information in the knowledge base 121. Knowledge information. If so, the knowledge information is displayed in the browsing interface to explain the display field of view.

In one embodiment, the image information processing module 202 can generate image browsing information (such as identification parameters or identification codes) of the display field of view each time after determining the display field of view, and the knowledge acquisition module 21 can search for knowledge based on the image browsing information. The library 121 is used to detect whether there is knowledge information related to the displayed field of view (such as the previously input question and answer set or the target information of the entire target image, or the content automatically interpreted by AI).

For example, please refer to Figure 11. When the user browses to the display field of view in Figure 11, the knowledge acquisition module 21 finds that the current display field of view has corresponding knowledge information in the knowledge base (abnormal ratio 70%), and can display the knowledge information. In the target interface 62. In addition, if the user determines that the knowledge information is incorrect, he can also modify it through the input interface (for example, modify it to 90%), so that the target module 212 sets new target information for learning.

In one embodiment, the present invention further provides a browsing action judgment function. Specifically, each time the display field of view is switched, the action module 211 can obtain the previous display reference point, the previous zoom level or the previous display field of view. A rotation angle and compared with the display reference point, zoom level or rotation angle of the current display field of view to determine the browsing action of switching from the previous display field of view to the current display field of view (such as the translation direction, translation amount, magnification level, Rotation direction, rotation angle degree, etc.), and adds the browsing action to the action information, that is, judging the content of the browsing action based on the changes in the two sets of visual field parameters.

Next, step S25 is executed: the platform 10 generates training data by associating the display field of view with corresponding knowledge information (such as a question and answer set completed by the user, target information and/or action information) through the training data generation module 221 .

In one embodiment, the present invention further provides a recording function (steps S26-S27).

Step S26: During the browsing process, the platform 10 determines whether the preset recording conditions are met through the knowledge recording and processing module 22. The aforementioned recording conditions can be every time the display field of view is changed, after every operation, or at a fixed time (such as 10 seconds), and are not limited.

If the recording conditions are not met, monitoring will continue.

If the recording conditions are met, step S27 is executed: the platform 10 uses the image capture module 220 to capture the image of the display field of view based on the aforementioned image browsing information, and the knowledge recording and processing module 22 records the image of the display field of view and knowledge information (such as Questions and answers completed by the user, subject information and/or action information), and related.

In one embodiment, database 12 includes browsing records 122 . The knowledge recording and processing module 22 associates the visual field parameters and knowledge information of the displayed visual field, and records it in the browsing record 122 as a browsing history (which may include action history, question and answer sets and target information of all displayed visual fields).

Please also refer to FIG. 8 , which is a flow chart of training and automatic analysis of the method according to an embodiment of the present invention. The present invention further provides a training function (step S60) and an automatic analysis function (step S61).

Step S60: The AI module 13 trains the learning model 130 based on the training data. In one embodiment, step S60 includes steps S70-S73.

Step S70: The AI module 13 loads and obtains the learning model 130.

Step S71: The AI module 13 analyzes the training data through the training module 132.

In one embodiment, the training data includes images showing visual fields and corresponding knowledge information (such as question and answer sets, action information, target information, etc.). The training module 132 analyzes the image features of the display field of view and generates text features based on knowledge information (such as questions in the question and answer set and target information input by the user), that is, converts the knowledge information into the semantic network.

In one embodiment, the AI module 13 may first convert the training data into a format acceptable to the training module 132 and/or the learning model 130 through the data conversion module 131, and then perform analysis and training.

Step S72: The training module 132 inputs image features and text features to the learning model 130 to generate predictive knowledge information (such as predicted answers to questions, predicted target information, or predicted action information, etc.).

Step S73: The training module 132 compares the predicted knowledge information with the knowledge information of the training data to train the learning model. For example, comparing the predicted answer with the user's response answer, comparing the predicted target information with the target information input by the user, or comparing the predicted action information with the user's actual browsing action, etc., etc., without limitation.

Step S61: The AI module 13 receives image input to generate corresponding predictive knowledge information. In one embodiment, step S61 includes steps S80-S81.

Step S80: The AI module 13 accepts an operation (for example, the client 11 or the platform 10) to select an image (another target image).

Step S81: The AI module 13 inputs the target image to the learning model 130 to obtain the prediction knowledge information of the target image (which may include the prediction action process, prediction questions and prediction target information of one or more display fields of the target image). , and can be stored in the knowledge base 121.

Please refer to FIG. 12 , which is a schematic diagram of the action process of an embodiment of the present invention. After completing the history recording of the target image, the platform terminal 10 of the present invention can provide a playback function and display the action history 70 of the target image, the display field of view 71 and the thumbnail interface 72 in the browsing interface. The client can later choose to act View any step of the action process 70, or continuously play multiple steps of the action process 70 to reproduce the browsing path during analysis.

Please refer to FIG. 13 , which is a schematic diagram of a heat map according to an embodiment of the present invention.

After analyzing the target image, the platform terminal 10 of the present invention can provide a heat map function, which can display which parts of the target image are the areas of concern to the model during the algorithm analysis process.

As shown in the figure, the platform terminal 10 displays a display view interface 80 and a thumbnail interface 81 of the target image in the browsing interface. In the display field of view interface 80, areas with higher confidence in algorithm analysis will be represented by lighter-colored hot areas 801, and areas with lower confidence in algorithm analysis will be represented by darker hot areas 802. In addition, The white area 800 refers to the background area.

In this way, the user or other users can directly understand the importance of each position of the target image, and can directly ignore the browsing of the white hot area 800 and the dark hot area 802 (unimportant area) to save time and focus more on the light color. Browse hot area 801 (important area) to improve accuracy.

By prompting questions or knowledge information in a designated display field of view, the present invention can allow experts to know that the current display field of view is an area of interest and focus more on the analysis of the current display field of view.

The present invention provides a target image browsing service (web module 104) through a web page, allowing experts around the world to view the same target image through a web browser and perform browsing operations on the same target image (including answering questions and setting targets). information), and different knowledge information (i.e., training data) of different experts on the same target image can be collected.

Moreover, by subsequently collecting this knowledge information and providing it to the AI module 13 for training as training data for the learning model 130, an expert-verified and highly reliable learning model 130 can be generated. When other experts (such as pathologists) perform image interpretation in the future, they can first refer to the results of AI module 13 interpretation (such as predictive knowledge information) as the basis for writing subsequent pathology reports. Since the reference sources are consistent, the final interpretation results of the experts can be improved. consistency.

The present invention provides a browsing service of target images through a web page and can also reduce the time and damage risk of physical slide transportation. In addition, by prompting the knowledge of the currently displayed field of view and accepting the input of target information, the expert's thoughts during the image reading process can be completely recorded and shared with other experts for viewing, which helps different experts understand each other's analysis ideas and strategies. .

Of course, the present invention can also have various other embodiments. Without departing from the spirit and essence of the present invention, those with ordinary knowledge in the technical field to which the present invention belongs can make various corresponding changes and modifications according to the present invention. However, these Corresponding changes and deformations shall fall within the scope of the patent application attached to the present invention.

S10-S13: First training data generation step

Claims (20)

A method for generating training data through question and answer, including: a) a platform providing browsing of a target image; b) the platform accepting operations to adjust a display field of view of the target image; c) the platform presenting to analyze the Display a question of the field of view and obtain a response answer; and d) the platform associates a question and answer set with a browsing action to generate training data for automatically analyzing the target image; wherein the question and answer set corresponds to the display field of view , the question and answer set includes the question and the response answer; wherein the decision of the browsing action is based on obtaining the previous display reference point, the previous zoom level or the previous rotation angle of the previous display field of view, and converting the previous display reference point , the previous zoom level or the previous rotation angle is compared with a display reference point, a zoom level or a rotation angle of the current display field of view to determine the time to switch from the previous display field of view to the current display field of view. The browsing action. The method for generating training data through question and answer as described in claim 1, wherein step a) includes: a1) after logging in to the platform, the platform receives an operation to select one of multiple images from an image library as the target image, or upload the image to the platform as the target image; a2) the platform determines the display field of view of the target image based on at least one of the display reference point, the zoom level and the rotation angle; and a3 ) The platform terminal outputs an image of the display field of view to present the display field of view. The method of generating training data through question and answer as described in claim 1, wherein the target image is divided into a plurality of tiles; the step b) includes: b1) The platform accepts an operation to adjust a field of view parameter, wherein the field of view parameter includes at least one of the display reference point, the zoom level and the rotation angle; b2) the platform decides to enter based on the adjusted field of view parameter All the tiles of the display field of view are used as images of the adjusted display field of view; and b3) the platform outputs the adjusted image of the display field of view to present the adjusted display field of view. The method for generating training data through question and answer as described in claim 1, wherein step c) includes: c1) the platform obtains the corresponding question from a database based on an identification code of the display field of view, wherein the question is for An image feature of the display field of view is preset or automatically generated through machine learning, and is used to analyze the image feature; c2) the platform displays the question; and c3) the platform receives the response answer. The method of generating training data through question and answer as described in claim 4, wherein the question is a closed question; the step c3) is to display multiple answer options and use the selected answer option as the response answer. The method of generating training data through question and answer as described in claim 4, wherein the question is an open-ended question; the step c3) is to display a text input area and use the text content input into the text input area as the response answer. The method of generating training data through question and answer as described in claim 1 further includes: e1) accepting an operation through the platform to select an input image; and e2) inputting the input image into a learning model through the platform to obtain the A predicted motion course of the input image. The method for generating training data through question and answer as described in claim 1 further includes at least one of the following steps: f) The platform accepts an operation to set a target information, wherein the target information is an image as the target image An analysis result of the characteristics; and g) when the platform determines that the current display field of view has corresponding knowledge information in a database, the knowledge information is displayed, where the knowledge information is used to explain the display field of view. The method of generating training data through question and answer as described in claim 1 further includes: h1) obtaining a learning model through an AI module; h2) analyzing an image feature of the display field of view of the training data through the AI module, and generate a text feature based on the question and answer set; h3) input the image feature and the text feature to the learning model through the AI module to generate a predicted answer; and h4) use the AI module to compare the predicted answer with the response answers to train the learning model. The method of generating training data through question and answer as described in claim 1 further includes: i1) accepting an operation through the platform to select an input image; and i2) inputting the input image into a learning model through the platform to obtain the Input a prediction knowledge information of the image. A system for generating training data through question and answer, including: a database for storing a target image; and a platform connected to the database and connected to a client via a network, the platform being configured to The client provides a browsing interface to browse the target image, accepts operations from the client to adjust a display field of view of the target image, presents a question for analyzing the display field of view, obtains a response answer, and associates a question and answer set with A browse action is used to generate a training set for automatic analysis of the target image. Training data, wherein the question and answer set corresponds to the display field of view, and the question and answer set includes the question and the response answer; wherein, the platform includes: an action module configured to obtain the previous display reference point of the previous display field of view , the previous zoom level or the previous rotation angle, the previous display reference point, the previous zoom level or the previous rotation angle and the current display reference point, a zoom level or a rotation angle of the display field of view Comparison is performed to determine a browsing action of switching from the previous display view to the current display view. The system for generating training data through question and answer as described in request 11, wherein the database includes: a user database for storing multiple registration information; and an image database for storing multiple images; wherein, the platform The client includes: a web module configured to interact with the client through a web page to receive a login information of the client, and select one of the multiple images in the image library or upload an image from the client as the target image; and a management module configured to accept login when the login information matches any of the registration information. The system for generating training data through question and answer as described in claim 11, wherein the platform includes: an image viewing module configured to obtain the target image, wherein the target image is divided into a plurality of tiles; an image operation A module configured to adjust a field of view parameter according to the operation of the user terminal, wherein the field of view parameter includes at least one of the display reference point, the zoom level and the rotation angle; and An image information processing module configured to determine all the tiles entering the display field of view as images of the adjusted display field of view based on the adjusted field of view parameter, and output the adjusted image of the display field of view to the client. , to present the adjusted display field of view on the client. The system for generating training data through question and answer as described in claim 11, wherein the database includes a knowledge base for storing a plurality of questions corresponding to different display fields of view of the target image, and the questions are directed to the display field of view. An image feature is preset or automatically generated through machine learning, and is used to analyze the image feature. The display field of view and the corresponding question are associated with an identification code; wherein, the platform includes: a question and answer module , is configured to obtain the corresponding question based on the identification code of the current display field of view, display the question on the client, and receive the response answer from the client. The system for generating training data through question and answer as described in claim 14, wherein the question is a closed question or an open question; wherein the question and answer module is configured to display on the client when the question is a closed question There are multiple answer options, and the selected answer option is used as the response answer. When the question is a closed question, a text input area is displayed, and the text content entered into the text input area is used as the response answer. The system for generating training data through question and answer as described in claim 11, wherein the platform includes: a knowledge recording and processing module, which is configured to capture the image of the displayed field of view and record the corresponding when a recording condition is met. A piece of knowledge information, wherein the knowledge information includes the question and answer set and the browsing action, and the display field of view and the knowledge information are mapped to an identification code for association. The system for generating training data through question and answer as described in claim 11 further includes an AI module, and the AI module includes: A learning model for automatically generating at least one predicted browsing action for an input image, simulating browsing of the input image based on the at least one predicted browsing action to transform the display field of view, and performing image analysis based on the transformed display field of view A prediction target information of the input image; wherein the learning model includes a DQN architecture model; wherein the prediction target information is associated with the input image. The system for generating training data through question and answer as described in claim 11, wherein the database includes a knowledge base for storing a plurality of knowledge information corresponding to different display fields of view, and the knowledge information is used to describe the display field of view. ; Among them, the platform further includes: a target module configured to accept the operation of the client to set a target information, wherein the target information is an analysis result of an image feature of the target image; and a knowledge processing The providing module is configured to display the knowledge information on the client when it is determined that the current display field of view has corresponding knowledge information; wherein, the platform is configured to associate the display field of view with the knowledge information to generate The training data and the knowledge information include the question and answer set and the set target information. The system for generating training data through question and answer as described in claim 11 further includes an AI module. The AI module includes: a learning model for automatically analyzing an input image to automatically generate at least one element of the input image. Predictive knowledge information of a displayed field of view, wherein the predicted knowledge information includes at least one of a predicted action process, a predicted target information, a predicted question of any displayed field of view, and a predicted answer of any displayed field of view. The system for generating training data through question and answer as described in claim 19, wherein the learning model includes a VQA architecture model; Wherein, the AI module further includes a training module, which is configured to analyze an image feature of the display field of view of the training data, and generate a text feature based on knowledge information of the same display field of view of the training data, and input the image Features and the text features are applied to the learning model to generate the predicted answer, and the learning model is adjusted by comparing the predicted answer with the response answer.

Images