WO2023037413A1 - Data acquisition system and data acquisition method - Google Patents

Abstract

Provided are a data acquisition system and a data acquisition method which make it possible to acquire rare data that is input to an inference device. An inference model for inferring and displaying a lesion from an image is used to perform inference (S1), and when the relation between the result of the inference and an action of a doctor who diagnoses the lesion is not an expected relation (S3 Yes, S5), image data at the time of display is acquired (S7). Further, the acquired image data is regarded as a training data candidate, an annotation is given, and training data is created (S7).

Description

Data acquisition system and data acquisition method

The present invention relates to a data acquisition system and data acquisition method for collecting image data for inference model generation in medical equipment.

An inference model is generated by collecting a large amount of data, creating teacher data by annotating this data, and performing machine learning such as deep learning using this teacher data. It is known to make various inferences by inputting data into an inference unit having this inference model.

For example, Patent Document 1 discloses an inference device that inputs sensor data provided on a target machine into an inference model and performs inference for controlling the target machine. This inference device includes a diagnosis target machine to be diagnosed and a database DB for acquiring and storing sensor data of machines similar to this machine, and a first feature amount created from the sensor data acquired for the similar machine. By creating an inference model using , determines the state of the machine to be diagnosed. With this configuration, the reasoning device of Patent Document 1 can correctly determine the state of the target machine even when the sensor data of the target machine is abnormal data.

JP 2020-187516 A

In the inference device in Patent Document 1, normal diagnosis can be performed even when abnormal data is input. However, the inference device of Patent Document 1 does not describe acquisition of rare data in order to generate an inference model for properly inferring even rare data. That is, the inference device in Patent Literature 1 cannot create an inference model in consideration of rare data such as when an abnormality occurs.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a data acquisition system and a data acquisition method that can acquire rare data to be input to an inference device.

In order to achieve the above object, a data acquisition system according to a first invention comprises an inference unit having an inference model for inferring and displaying a lesion from an image, an inference result in the inference unit, and a diagnosis of the lesion. and an acquisition unit that acquires image data when the display is performed when the relationship of actions of the doctor who performs the display is not in the expected relationship.

A data acquisition system according to a second invention is the data acquisition system according to the first invention, wherein the image data acquired by the acquisition unit when the display is performed is a moving image including image frames at the time of the display, and the image frames are displayed. Data containing identifiable information.
A data acquisition system according to a third aspect of the invention is the data acquisition system according to the first aspect of the invention, further comprising a recording unit that adds information about the rarity of the image to the image acquired by the acquisition unit and records the image.

According to a fourth aspect of the present invention, there is provided a data acquisition system according to the first aspect, further comprising a teacher data candidate selection unit that selects the image data acquired by the acquisition unit as a teacher data candidate.
A data acquisition system according to a fifth aspect is the data acquisition system according to the first aspect, wherein the image data acquired by the acquisition unit when the display is performed is a time-series image group including image frames at the time of the display, and This is data containing information that can specify an image frame.

A data acquisition system according to a sixth aspect of the present invention has a database that records an expected relationship between an inference result obtained using the inference model in the first aspect and an action of the doctor who makes the diagnosis. .
A data acquisition system according to a seventh invention is the data acquisition system according to the first invention, according to the similarity between the image when the lesion is displayed and the teacher data group when the inference model is learned, the teacher data A candidate image similarity determination unit is provided, and the acquisition unit acquires the image determined as the teacher data candidate by the image similarity determination unit.
A data acquisition system according to an eighth invention is the data acquisition system according to the first invention, in which the inference results of the inference unit are not displayed, and based on the action taken by the doctor, whether or not the expected relationship is met. judge.

A data acquisition method according to a ninth aspect of the invention uses an inference model for inferring and displaying a lesion from an image, and the relationship between the result of the inference and the action of a doctor who diagnoses the lesion is established. If the relationship is not in the expected relationship, the image data when the above display is performed is acquired.
A data acquisition method according to a tenth aspect of the invention makes inferences using an inference model for displaying an operation guide for an endoscope, and determines the relationship between the operation guide display and the actions of a doctor who operates the endoscope. If the relationship is not in the expected relationship, the operation-related data at the time when the operation guide display is performed is acquired.

A data processing method according to an eleventh invention inputs sensor data provided in a medical device or operation data of a doctor to the medical device into an inference model to make an inference, and uses the result of the inference and the medical device. When the doctor's actions are not in the expected relationship, an identification code can be given to the sensor data or the operation data when the expected relationship is not met.
A data processing system according to a twelfth aspect of the present invention comprises an inference unit that inputs sensor data provided in a medical device or operation data of a doctor on the medical device into an inference model to make an inference; a result of the inference; a data processing unit capable of assigning an identification code to the sensor data or the operation data when the action relationship of the doctor using the device does not meet the expected relationship. have.

According to the present invention, it is possible to provide a data acquisition system and a data acquisition method for acquiring rare data to be input to an inference device.

1 is a block diagram mainly showing an electrical configuration of a data acquisition system according to an embodiment of the present invention; FIG. 1 is a block diagram mainly showing an electrical configuration of a data acquisition system according to an embodiment of the present invention; FIG. 4 is a flowchart showing image acquisition operations in the data acquisition system according to one embodiment of the present invention. 4 is a flow chart showing the operation of acquiring important images in the data acquisition system according to one embodiment of the present invention; It is a figure which shows the recording content of the database in the data acquisition system which concerns on one Embodiment of this invention. 4 is a flow chart showing the operation of an in-hospital system in the data acquisition system according to one embodiment of the present invention. 4 is a flow chart showing the operation of the device in the data acquisition system according to one embodiment of the present invention. In the data acquisition system according to one embodiment of the present invention, (a) shows acquisition of data for generating an inference model, and (b) shows inference in an inference model generated using the acquired data. .

An example in which the present invention is applied to a data acquisition system will be described below as an embodiment of the present invention. This data acquisition system includes a device 10 such as an endoscope, an in-hospital system 20 that can cooperate with this device, and a management server 30 . Each device or the like has each unit for realizing various functions, but the device or the like that realizes each function may be changed as appropriate. Further, in the present embodiment, the device 10 is described as an endoscope, but the device 10 is not limited to an endoscope as long as it outputs data such as image data and performs inference using this data. Also, the part described as the in-hospital system 20 need not be limited to what is provided in the hospital, such as a cloud-based management tool, as long as the information of the device 10 can be shared. A person who can access a cloud-based management tool or the like to determine data is described as a doctor, but it is also possible to write an expert.

FIG. 1A shows a configuration of a device 10 such as an endoscope and an in-hospital system 20 that cooperates with the endoscope 10. FIG. FIG. 1B also shows a management server 30 that cooperates with the hospital system 20. This management server 30 has a recording unit that records existing teacher data and the like used to create an inference model, an annotation/learning unit, and the like. The hospital system 20 and the management server 30 can communicate for data exchange and the like through a communication network such as the Internet. The arrangement of each block in the device 10, the hospital system 20, and the management server 30 may be appropriately changed other than the arrangement shown in FIGS. 1A and 1B. In FIGS. 1A and 1B, image data and the like acquired by the equipment 10 are transmitted to the management server 30 through the hospital system 20. Of course, communication may be performed directly by

In FIG. 1A, a device 10 such as an endoscope has a control unit 11, a display unit 12, an image acquisition unit 13, an inference unit 14, and an operation result reflection unit 15. The image acquisition unit 13 includes various circuits such as an imaging lens, an imaging device, an imaging device control circuit, and an imaging data processing circuit. When the device 10 is an endoscope, the image acquisition unit 13 acquires an image inside the body of a patient or the like. The input image P1 is an example of image data. This input image P1 has information (tag data) such as category 1 attached to the image data P1d. be. Note that FIG. 1A shows only P1 as an input image, but of course, many images are input at the time of inspection, and not only still images but also moving images may be used. The image acquisition unit 13 may input a plurality of images for each examination, for example, as indicated by examinations Ex1 to Ex3 in FIG.

The operation result reflection unit 15 has an operation unit for operating the device 10, a user interface, and the like. For example, when the device 10 is an endoscope, the operation unit includes a release operation unit for instructing imaging, a light source, an operation unit for supplying air, water, and suction, and mechanisms such as an angle mechanism and a biopsy mechanism. have. As an interface, it may have a user interface such as a touch screen. The operation result reflection unit 15 implements the operation of each operation unit according to the operation of the operation unit described above. For example, when the release operation unit instructs acquisition of a still image, the image acquisition unit 13 acquires the still image, and when the release operation unit instructs acquisition of a motion, the image acquisition unit 13 acquires the motion. .

The inference unit 14 has an inference engine equipped with an existing inference model 14a, a category determination unit 14b, and an image similarity determination unit 14c. The existing inference model 14a is an inference model generated by the annotation/learning unit 32 using existing teacher data 33a recorded in the recording unit 33 of the management server 30, which will be described later. The existing inference model 14a functions as an inference model for inferring and displaying a lesion from an image, and the inference unit 14 functions as an inference unit equipped with this existing inference model 14a (for example, see S1 in FIG. 2). ).

Also, the existing inference model 14a is not limited to inferring a lesion, and may be, for example, an inference model for displaying an operation guide for an endoscope. The image obtained by the image obtaining unit 13 and/or the operation state information obtained by the operation result reflecting unit 15 are input to this inference model, the operation guide is inferred, and the operation guide is displayed based on the inference result. The inference unit 14 functions as an inference unit equipped with this existing inference model 14a.

Also, the existing inference model 14a may be an inference model other than an inference model that infers a lesion or an operation guide. The medical device may be provided with other sensors in addition to acquiring image data by the imaging device. Of course, the operation data of the doctor's medical equipment may be input to the existing inference model 14a. The inference unit 14 functions as an inference unit that inputs sensor data provided in a medical device or operation data of a doctor on the medical device to an inference model and makes an inference.

Even if the existing inference model itself is not generated by the annotation/learning unit 32 in the management server 30, the annotation/learning unit 32 requests an external learning device to generate an inference model, and the generated inference model may be Also, the inference engine may be configured by hardware, may be configured by software (program), or may be a combination of hardware and software. Further, the inference unit 14 may be provided in the hospital system 20 or the management server 30 and perform inference using an existing inference model in the hospital system 20 or the management server 30 .

The existing inference model 14a can infer whether or not there is a tumor or the like in the input image, and if there is a tumor or the like, it can display its position. The existing inference model 14a may infer lesions and the like in addition to tumors. Further, the existing inference model 14 a may perform inference for guiding the doctor or the like when operating the device 10 and display the inference result on the display unit 12 . The image data P1d acquired by the image acquisition unit 13 is input to the inference engine having the existing inference model 14a, and the inference result by the existing inference model 14a is output. The inference engine also calculates the reliability of this inference when making an inference, and outputs this reliability as well.

The category determination unit 14b determines the category of the image acquired by the image acquisition unit 13. The category determination unit 14b determines categories such as observation site, operation state, treatment, model used, etc., based on information recorded in category 1P1c of input image P1 and information such as equipment used.

The image similarity determination unit 14c determines whether the image acquired by the image acquisition unit 13 is a similar image. There are many images acquired by the image acquisition unit 13, and many of them are similar to each other. Therefore, the image similarity determination unit 14c determines whether or not the images are similar to each other. In addition, in this embodiment, as training data for generating a new inference model, image data obtained when the inference result of the existing inference model differs from the doctor's diagnosis result is acquired (contradiction determination unit 23, new inference model data 33c, etc.).

Whether an image with a high degree of similarity or an image with a low degree of similarity is required depends on the situation. Even if there is a sufficient number of similar training data, there are cases where similar images are requested as test data, and conversely, there are cases where dissimilar images are requested even if there are a sufficient number of similar images. be. Furthermore, even if the images are similar, the circumstances or objects in which the images were obtained may be rare. It is often helpful to have a program or the like.

In addition to this method, the image similarity determination unit 14c may acquire an image determined to be similar to an image that has been used as teacher data so far, and use this image. The image similarity determination unit 14c functions as an image similarity determination unit that selects training data candidates according to the degree of similarity between the image when the lesion is displayed and the training data group when the inference model is learned. In this case, the acquisition unit acquires the image determined as the teacher data candidate by the image similarity determination unit.

The display unit 12 has a display monitor such as a display, and displays menu screens (images for operation), internal examination images acquired by the image acquisition unit 13, and the like. A doctor or the like can operate the device 10 such as an endoscope while observing the acquired image (capture image), and can also perform examinations, diagnoses, and the like. The display unit 12 receives the image obtained by the image obtaining unit 13 and the inference result 12a by the inference unit 14, and displays the inference result. When displaying the inference result 12a, the inference reliability 12b may be displayed.

The control unit 11 is a processor including a CPU (Central Processing Unit) and various peripheral circuits. The processing device such as the CPU controls each section in the device 10 according to a program stored in the storage section of the device 10 . Note that the control unit 11 may perform some or all of the functions of the image acquisition unit 13 , the operation result reflection unit 15 , the inference unit 14 , and the display unit 12 . Note that the device 10 has, for example, a communication section and the like in addition to the above-described sections. This communication unit communicates with the hospital system 20 , but may communicate directly with the management server 30 .

The in-hospital system 20 is a server or the like provided in the hospital, can be connected to a plurality of devices 10 by wireless communication or wired communication, and can exchange various data. Note that the in-hospital system 20 may be provided in the cloud as long as it can be connected to the device 10 or the like.

The in-hospital system 20 receives the image input by the image acquisition unit 13 from the device 10, and the inference result, the reliability of the inference result, the category determination result, the image similarity determination result, etc. by the inference unit 14. The image acquired by the device 10 is recorded, and an image for generating a new inference model is selected according to the inference result and the reaction of the doctor or the like and transmitted to the management server 30 . In this case, an image for generating a new inference model can be selected according to the inference result, the image displayed according to the reaction of the doctor, etc., confirmed by the doctor, and the reaction of the doctor who confirmed the image. good. Furthermore, the image for generating the new inference model may be selected according to the reaction when the image is displayed, or the reaction when the observation is written after the image display is finished. Also, using these images, a request is made to generate a new inference model, and the generated new inference model is obtained. The hospital system 20 has a control unit 21 , a UI unit 22 , a contradiction determination unit 23 , a communication unit 24 and a recording unit 25 .

The UI unit 22 determines the operating state of the UI (user interface) of the device 10 . The UI unit 22 includes user interfaces such as a keyboard for manually inputting text information by a doctor and a microphone for inputting voice information. Further, the UI unit 22 may be an interface for inputting the findings of the doctor in charge, and may be linked with an electronic medical chart in which the doctor writes and creates the above-mentioned findings, for example. If it is not possible to link with the electronic medical chart, the medical chart information may be input manually, or the medical chart image may be read by a scanner or the like, and the result may be read as characters. In some cases, the medical record is recorded by voice, and in that case, the function to convert voice to text should be used together.

The contradiction determination unit 23 determines the inference result of the inference unit 14 and the reaction of the doctor who operated the device 10 by the UI unit 22 or who made a diagnosis by viewing the image (including the result of converting the above findings into text), in other words, Compare the reaction with the response of the expert who made the expert judgment, and determine whether there is a contradiction between the two. For example, when an existing inference model for inferring whether or not there is a tumor is set, the inference unit 14 infers that there is no tumor in the image acquired by the device 10. Suppose you observe and determine that there is a tumor. In this case, there is a contradiction between the inference result of the inference unit 14 and the doctor's diagnosis result. The contradiction determination unit 23 determines whether or not there is a contradiction between the inference result of the inference unit 14 and the diagnosis result by the doctor. In addition to determining whether or not there is a contradiction between the inference result and the doctor's diagnosis result, the contradiction determination unit 23 may determine the degree of importance in consideration of, for example, the reliability of the inference result. good. Determination by the contradiction determination unit 23 will be described later with reference to FIG.

The timing for the contradiction determination unit 23 to determine whether or not there is a contradiction may be when the doctor observes the acquired image when operating the device 10 . In addition to this timing, it may be performed when diagnosing while observing the image recorded in the recording unit 25 on the display unit of the in-hospital system after the examination. An electronic medical chart for recording diagnostic results may be filled in while observing images during or after an examination.

In addition, as a user interface in the UI unit 22, as described above, in addition to the electronic medical record, the operating state of the doctor's equipment may be input and used. For example, when it is output that there is no tumor as an inference result, the doctor may observe the site many times, or the site may be biopsied, dyed, or excised. If so, a tumor may have developed. In such a case, the contradiction determination unit 23 may determine that there is a contradiction in the inference result.

In addition, the contradiction determination unit 23 determines whether or not the operation guide display and the inference result are inconsistent, not only when the inference result and the action (reaction) of the doctor are not in the relationship expected for a lesion such as a tumor. You may make it judge. For example, if the existing inference model 14a is an inference model for displaying operation guidance for an endoscope, this inference model can be used to display the operation guidance. When this operation guide display and the operation of the doctor who operates the endoscope while looking at this operation guide do not match, the contradiction determination unit 23 determines that there is a mismatch, and the image and/or image when this mismatch occurs. Alternatively, operation information at that time may be acquired.

The functions of the UI unit 22 and the contradiction determination unit 23 may be realized by hardware and/or software. Of course, part of the functions of the UI unit 22 and the contradiction determination unit 23 may be implemented by hardware, and the rest may be implemented by software. Also, the functions of the UI unit 22 and the contradiction unit 23 may be combined with the control unit 21 . That is, a processor may be provided for realizing all or part of the functions of the control unit 21, the UI unit 22, and the contradiction determination unit 23. FIG.

The contradiction determination unit 23 is an acquisition unit (processor) that acquires image data when displaying when the relationship between the inference result in the inference unit and the action of the doctor who diagnoses the lesion is not in the expected relationship. (See S5 in FIG. 2, S13 to S17 in FIG. 3, S31 in FIG. 5, S53 in FIG. 6, etc.). The image data acquired by the acquisition unit when displaying is a moving image (time-series image group) including image frames at the time of display, and is data containing information that can specify the image frames. The image data acquired by the acquisition unit when displaying is a time-series image group including image frames at the time of display, and is data containing information that can specify the image frames. The contradiction determination unit 23 attaches information about the rarity of the image to the image acquired by the acquisition unit. The contradiction determination unit 23 functions as a training data candidate selection unit that selects the image data acquired by the acquisition unit as training data candidates (see S7 in FIG. 2, for example).

If images are scarce, image data can be selectively used according to the required performance and required specifications of the inference model when learning and testing the inference model. In addition, since the rarity can be determined from the attributes of the object from which the image was obtained and the characteristics of the object appearing in the image, the reason for the rarity can be recorded in the metadata of the image. You may do so. The acquired images may be assumed to be still images taken, or may be acquired as a series of inspection videos (time-series image group), and acquired in such a way that frames in them can be specified. The method of That is, the image data acquired by the acquisition unit when displaying is a time-series image group including image frames at the time of display, and may be data containing information that can specify the image frames. The system and control unit know which frame of the video is being displayed in live view or playback, and it is possible to determine which frame is being displayed, so important images can be recorded as frame-specific information. is easy.

As described above, the contradiction determination unit 23 and the like may determine whether or not there is an expected relationship between the display of the operation guide and the action (reaction) of the doctor or the like, not limited to the lesion such as a tumor. Furthermore, the in-hospital system 20 (which may be the device 10 or the management server 30) has an inference unit that makes inference by inputting sensor data provided in the medical device or doctor's operation data for the medical device into the inference model. If so, the contradiction determination unit 23 and the like, if the relationship between the result of inference and the action of the doctor using the medical device is not in the expected relationship, the sensor when the expected relationship is not met It has a function as a data processing section that can assign an identification code to data or operation data. This data processing unit may simply assign an identification code to sensor data or operation data when the expected relationship is not met, or simply select data to which an identification code has been assigned. , or may be further acquired and recorded.

The communication unit 24 has a communication circuit and can communicate with the communication unit 35 of the management server 30 and the like. The communication unit 24 can transmit information such as image data recorded in the recording unit 25 to the management server 30 . Further, the communication unit 24 creates teacher data by annotating image data and the like recorded in the recording unit 25, and receives generation of an inference model generated by the learning unit from the management server 30 or the like. You can also The communication unit 24 can also receive the existing inference model 32b stored in the management server 30 and transmit it to the inference unit 14 in the device 10 .

When the contradiction determination unit 23 determines that there is a discrepancy between the inference result and the doctor's diagnosis result, the communication unit 24 transmits image data and the like for generating a new inference model to the management server. (For example, the inference result is no tumor, while the doctor's diagnosis is a tumor), both are associated and transmitted. That is, since the acquisition timing of the image data differs from the timing at which the doctor's diagnosis results are obtained based on the description in the electronic medical record, if simultaneous transmission is not possible, the two are associated with each other. The control unit 21 may associate the two.

The recording unit 25 is an electrically rewritable non-volatile memory, and records the image data acquired by the image acquisition unit 13 and the recorded tag data associated with this image data. The recording unit 25 may record all the image data acquired by the image acquisition unit 13, or may record only images in which the inference result and the diagnosis result are contradicted by the contradiction determination unit 23. good. The recording unit 25 also stores a database showing expected relationships based on inference results and doctor actions, which will be described later with reference to FIG.

The recording unit 25 functions as a recording unit that attaches information about the rarity (importance) of the image to the image acquired by the acquisition unit and records the image (see, for example, S15 of FIG. 3, FIG. 4, etc.). Note that this recording unit is not limited to the hospital system 20, and may be provided in the management server 30. A recording unit is provided in the device 10 to determine the degree of rarity (importance) and record it in association with the image. may The recording unit 25 records a database indicating the expected relationship between the inference result obtained using the inference model and the action of the doctor who makes the diagnosis (see FIG. 4, for example).

The control unit 21 is a processor including a CPU (Central Processing Unit) and various peripheral circuits. The processing device such as the CPU controls each part within the hospital system 20 according to a program stored in the storage unit within the hospital system 20 . Note that the control unit 21 may perform some or all of the functions of the UI unit 22 , the contradiction determination unit 23 , the communication unit 24 and the recording unit 25 .

The management server 30 can be connected to a plurality of hospital systems 20 and the like through a communication network such as the Internet, records training data, generates an inference model based on this training data, or generates an inference model externally. can be requested to the learning device of This generated inference model can be transmitted to the hospital system 20 through a communication network such as the Internet. The management server 30 has a control unit 31 , an annotation/learning unit 32 , an existing inference model 32 b , a recording unit 33 , a recording control unit 34 and a communication unit 35 .

The recording unit 33 in the management server 30 has an electrically rewritable non-volatile memory and can record various data. The management server 30 records existing teacher data A33a, existing teacher data B33b, and new inference model data 33c in the recording area. The existing teacher data A33a is teacher data that has been used when generating the existing inference model 32b. On the other hand, the existing teacher data B32b is teacher data that was created as teacher data but was not adopted when generating the existing inference model 32b for some reason.

The recording unit 33 can record a plurality of pieces of teacher data as existing teacher data A32a and existing teacher data B32b. In FIG. 1B, the existing teacher data A 33a depicts three teacher data of a plurality of teacher data 33aa, 33ab, and 33ac, and the existing teacher data B 33b depicts one teacher data of the teacher data 33ba. , a large number of teacher data can be recorded. The teacher data 33aa consists of image data 33aaa, categories 133aab, and annotations 33aac.

The image data 33aaa is image data adopted as teacher data, and the category 133aab indicates the category of the image data. The annotation 33 aac is an annotation added by the annotation/learning unit 32 . As annotations, for example, the presence or absence of a tumor, the position of the tumor, the type of the tumor, and the like are added, and a doctor's action guide when the image data is acquired may be added. Action guides for doctors include, for example, treatment candidates displayed as guides such as biopsy, pigment spraying, staining, and excision, and may also be items described in medical charts. The image data in the teacher data 33aab, 33aac, and 33ba are recorded in association with the same information as the category 1 annotation teacher data 33aa.

The new inference model data 33c is image data or the like when the contradiction determination unit 23 in the hospital system 20 determines that there is a contradiction between the inference result of the existing inference model 14a and the doctor's diagnosis result. . The image data recorded as the new inference model data is the data transmitted to the management server 30 as the new inference model data among the image data recorded in the recording unit 25 . When this image data is recorded in the hospital system 20, it may not yet be determined to be inconsistent. At the same time, annotation information, category 1 information, etc. to be attached to the image data are sent to the annotation/learning unit 32, and teacher data with annotations is created and recorded as new inference model data 33c. The new inference model data 33c can be used not only as teaching data for generating a new inference model, but also for evaluating the new inference model and calculating reliability.

In addition to the method described above, the new inference model data 33c may be transmitted by the hospital system 20 to the management server 30 when the image data is temporarily recorded in the recording unit 25 in the hospital system 0, and the image data may be temporarily recorded, and when the contradiction determination unit 23 determines that there is a contradiction, the annotation/learning unit 32 may add an annotation and record it as new inference model data 33c. Also, similar images that are similar to the images for the new inference model may be collected. Further, as the data for the new inference model, not only the image data but also the specification of the additional teacher data may be recorded.

The annotation/learning unit 32 creates teacher data by annotating image data transmitted from the hospital system 20 or the like. Annotations may be added based on data transmitted together with image data from the hospital system 20 or the like, or may be automatically added based on other information. Alternatively, an expert such as a doctor may manually add annotations.

The annotation/learning unit 32 generates an inference model by machine learning such as deep learning using teacher data such as the new inference model data 33c. Therefore, the annotation/learning unit 32 has hardware such as an inference engine for inference model generation, or software for inference model generation. Further, if the inference model cannot be generated by learning within the management server 30, it may have a request unit for requesting an external learning device to generate the inference model.

The existing inference model 32b is an inference model generated using the existing teacher data A33a. This inference model 32 b is transmitted to the device 10 and set in the inference section 14 . As described above, the device 10 inputs the image data acquired by the image acquisition unit 13 to the inference unit 14, performs inference using the existing inference model 14a, and outputs the inference result to the display unit 12, the contradiction determination unit 23, and the like. do. Note that this inference may be performed in the in-hospital system 20 as well as in the device 10 . In this case, an inference section similar to the inference section 14 is provided within the hospital system 20 .

Here, deep learning will be explained. "Deep learning" is a multilayer structure of the process of "machine learning" using neural networks. A typical example is a "forward propagation neural network" that sends information from front to back and makes decisions. The simplest forward propagation neural network consists of an input layer composed of N1 neurons, an intermediate layer composed of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be discriminated. It suffices if there are three output layers composed of neurons. The neurons of the input layer and the intermediate layer, and the intermediate layer and the output layer are connected by connection weights, respectively, and the intermediate layer and the output layer are added with bias values, so that logic gates can be easily formed.

The neural network may have three layers for simple discrimination, but by increasing the number of intermediate layers, it is also possible to learn how to combine multiple feature values in the process of machine learning. In recent years, 9 to 152 layers have become practical from the viewpoint of the time required for learning, judgment accuracy, and energy consumption. In addition, a process called "convolution" that compresses the feature amount of an image may be performed, and a "convolution neural network" that operates with minimal processing and is strong in pattern recognition may be used. In addition, a "recurrent neural network" (fully-connected recurrent neural network), which can handle more complicated information and can handle information analysis whose meaning changes depending on the order and order, may be used in which information flows in both directions.

In order to realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays) may be used. However, not only this, but since most neural network processing is matrix multiplication, it is also possible to use processors called GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) that specialize in matrix calculations. good. In recent years, such artificial intelligence (AI) dedicated hardware "neural network processing unit (NPU)" is designed to be integrated and embedded with other circuits such as CPU, and has become a part of the processing circuit. In some cases.

In addition, there are other methods of machine learning, such as support vector machines and support vector regression. The learning here involves calculation of classifier weights, filter coefficients, and offsets, and there is also a method using logistic regression processing. If you want a machine to judge something, you have to teach the machine how to judge. In the present embodiment, a method of deriving image determination by machine learning is used. In addition, a rule-based method that applies rules acquired by humans through empirical rules and heuristics may be used.

The recording control unit 34 controls data recording in the recording unit 33 . As described above, when image data is transmitted from the recording unit 25 of the hospital system 20, or when data related to contradiction determination is transmitted from the contradiction determination unit 23, these data are recorded. control.

The communication unit 35 has a communication circuit and can communicate with the communication unit 24 of the hospital system 20 and the like. The communication unit 35 can receive information such as image data recorded in the recording unit 25 . The communication unit 35 can also communicate with servers other than the hospital system 24 to collect various information. Note that in FIGS. 1A and 1B, the device 10 and the management server 30 do not communicate directly, but they may communicate data directly. In this case, when the contradiction is clarified by the contradiction determination unit 23, the data may be recorded as new inference model data.

The control unit 31 is a processor that includes a CPU (Central Processing Unit), etc., and is composed of an ASIC (Application Specific Integrated Circuit) that includes various peripheral circuits. The processing device such as the CPU controls each section in the management server 30 according to a program stored in the storage section in the management server 30 . Note that the control unit 31 may perform some or all of the functions of the annotation/learning unit 32 , the recording control unit 34 , and the communication unit 35 .

Thus, in the data acquisition system in FIGS. 1A and 1B, the existing inference model 32b generated in the management server 30 or the like is set in the inference unit 14 in the device 10 as the existing inference model 14a. The input image P1 is inferred using the model 14a. As this inference, for example, the presence or absence of a lesion such as a tumor may be inferred, or a guide for operating the device 10 or the like may be inferred for a doctor or the like.

In addition, when there is a contradiction between the inference result of the inference unit 14 and the diagnosis result of the doctor using the device 10, in other words, the relationship between the inference result of the inference model and the action of the doctor who makes the diagnosis is expected. If there is no such relationship, the acquired image at that time is acquired as the new inference model data and recorded as the new inference model data 33c. A new inference model can be generated using the recorded new inference model data 33c. This new inference model can reduce the number of doctors' judgments that the inference results are strange when compared with the existing inference model 14a that has been used so far.

As described above, in the data acquisition system according to the present embodiment, in order to generate a new inference model, if the relationship between the inference result of the inference model and the action taken by the doctor making the diagnosis does not meet the expected relationship, , I am trying to get the image when it is displayed. In addition to the methods described above, there are other ways to acquire the data for the new inference model. A method of collecting this data judged to be rare as data for a new inference model, and (2) matching the doctor's reaction without actually producing an inference result, and using the data at that time as data for the new inference model There are ways to collect. The above method (1) will be described later with reference to FIG.

The new inference model data acquired in this embodiment can be used to create teacher data by annotating the presence or absence of lesions such as tumors and their positions, and using this teacher data to generate a new inference model. . In addition to the generation of this new inference model, it can also be used for purposes such as evaluation of the new inference model and determination of reliability.

Next, the operation of acquiring important images will be described using the flowchart shown in FIG. The important image is an image displayed when the relationship between the inference result of the existing inference model and the action performed by the doctor who makes the diagnosis does not meet the expected relationship. This flow is realized by the control unit 11 in the device 10 and/or the control unit 21 in the hospital system 20 operating in cooperation according to the programs stored in the memory in each control unit.

When the flow in Figure 2 starts, first, the patient is examined using AI (S1). Here, a doctor examines and diagnoses a patient or the like using a device 10 such as an endoscope. At this time, the existing inference model 14a (AI) of the inference unit 14 is used to infer the presence or absence of a lesion such as a tumor and its position. At this time, the guide display for operating the device 10 may be performed by AI.

Next, it is determined whether or not the doctor showed a reaction contrary to the inference result (S3). Here, the contradiction determination unit 23 determines whether or not there is a contradiction between the inference result of the existing inference model 14a of the inference unit 14 and the doctor's diagnosis result. That is, it is determined whether or not the prescription assumed from the inference result is performed. A doctor's prescription is determined, for example, based on an electronic medical chart or the like, and whether or not it is different from the inference result is determined based on the description of the electronic medical chart. In addition to judging the doctor's reaction based on the electronic medical record, for example, based on the output of the operation result reflection unit 15 or the like, biopsy, pigment spraying, staining, polyp Also, based on the image acquired by the image acquisition unit 13, the device 10 such as an endoscope is repeatedly operated and observed, and a normal part is determined. The determination may be made based on the doctor's operation state, such as whether or not an operation different from that for observation has been performed.　

Also, in the determination in step S3, the degree of importance of the image may be determined in consideration of the reliability of the inference result. Even if the reliability of the inference result is high, when the doctor makes a diagnosis different from the inference result, the importance of the image at that time is high, and it can be said that it is useful for generating a new inference model. For example, when creating teacher data for generating a new inference model, the priority of using an image with a high degree of importance may be increased. Further, when the number of images having a high degree of importance reaches a predetermined number or more, generation of a new inference model is started, and the degree of importance may be used when determining the timing of generation of a new inference model.

In step S1, when the guide display for operating the device 10 is performed by AI, in this step S3, it is determined whether or not the doctor has performed an operation contrary to the operation guide display by AI. good too. That is, it may be determined whether or not there is an expected relationship between the operation guide display and the action of the doctor who operates the endoscope.

As a result of the determination in step S3, if the doctor shows a reaction contrary to the inference result, the image at that time is acquired as the important image (S5). Here, an image is obtained in which the doctor shows a reaction contrary to the inference result. In addition, it may be found later that the doctor showed a reaction contrary to the inference result. For example, when an electronic medical chart is used as a doctor's response, the electronic medical chart may be created later than when the doctor observes the image. In that case, all the images acquired by the device 10 may be temporarily recorded and recorded as important images when it is found that the doctor has shown a reaction contrary to the inference result. Therefore, the image data is temporarily recorded in the recording unit 25 in the hospital system 20, or is temporarily recorded entirely in the management server 30, and when it is found that the doctor has shown a reaction contrary to the inference result, It may be recorded as an important image.

As mentioned above, the important image here does not have to be a still image record like a photograph, but the image under inspection should be recorded at all times, and the part corresponding to the important image frame is a frame. may be recorded as, for example, how many minutes and seconds from a specific reference time such as the start of a moving image, or how many frames. In other words, it is not necessary to separately record specific frames separately from those recorded as continuous frame images such as moving images and continuous shot frame images. In many cases, it is better to keep the original video because other image information can also be left as evidence.

If it is determined in step S3 that the doctor has performed an operation contrary to the operation guide display by AI, the doctor's operation information at this time is acquired in step S5. That is, when it is determined that the relationship between the operation guide display and the action of the doctor who operates the endoscope is not in the expected relationship, the operation-related data when the operation guide display is performed is acquired. In addition, when inference is performed by inputting sensor data provided in the medical device or doctor's operation data to the inference model in step S1, the inference is performed in step S3. and the action of the doctor using the medical device are in the expected relationship. An identification code may be assigned to data or operation data.

As a result of the determination in step S3, if the doctor did not show a reaction contrary to the inference result (such as when an assumed prescription was given), or if the image at that time was acquired as the important image in step S5, then annotation (S7). Here, since the image data of the important images for generating the new inference model has been acquired, it may be requested to annotate the image data in order to create the teacher data. After the processing in step S7 is performed, the flow for acquiring the important image in FIG. 2 ends.

Here, we have mainly explained the acquisition of important images for the inference model for lesion judgment, but in highly specialized equipment such as endoscopes, the inference model is often used as an operation guide. This operation guide is based on the movement of the subject in the acquired image data (this is mainly due to the operation of the endoscope) and the data on the operation of the operation unit (whether or not rotation or pressing is performed, and the amount of operation is checked at each timing). It assists the operator by presenting the operation required at that time according to the change over time of the data acquired in the past. Similar to the inference model for judging lesions, an inference model can be obtained by learning from image data and operation data the situations in which operation errors are likely to occur. , it will be possible to issue guides for such scenes in the future. In other words, inference is made using an inference model for displaying an endoscope operation guide. It is possible to improve the inference model by acquiring operation-related data (images obtained in time series, history of operation data, etc.) when performing.

Next, the operation of important image determination will be described using the flow shown in FIG. This flow shows the operation related to the determination of whether or not the image is an important image, which is performed in step S31 of FIG. 5 and step S53 of FIG. This flow is also realized by the control unit 11 in the device 10 and the control unit 21 in the hospital system 20 operating in cooperation according to the programs stored in the memories in the respective control units.

When the flow in FIG. 3 starts, first, the inference result and reliability determination are acquired (S11). Here, the contradiction determination unit 23 acquires the inference result and the reliability value in the inference unit 14 . In addition, the contradiction determination unit 23 acquires the doctor's reaction from the UI unit 22 based on the electronic medical record created by the doctor. As the response of the doctor, other than the electronic chart, as described above, biopsy, pigment spraying, staining, excision, and other treatments performed by the doctor may be obtained, and the operating state of the device 10 at the time of examination by the doctor may be acquired. may be used for determination.

Next, it is determined whether or not the doctor's reaction, inference results, reliability, etc. are in the expected relationship (S13). Here, based on the information acquired in step S11, the contradiction determination unit 23 determines whether there is any contradiction between the doctor's reaction and the inference result, in other words, whether the doctor reacted as expected from the inference result. judge. In this case, the degree of importance (rarity) may be determined in consideration of reliability. In other words, when the reliability of the inference result is high, if the doctor reacts contrary to this inference result, the importance (rarity) of the image is high because the existing inference model is not sufficient. Therefore, it is desirable to generate a new inference model considering this importance.

In addition, in operator training, etc., multiple experts (beginners and veteran trainers) may be present, and even if beginners who receive training refer to the output of the inference model, they may make erroneous judgments and operations. may do so. In this case, the trainer may give a warning depending on the mistakes of the beginner who was operating, but even in such a case, the inference model was not correctly communicated to the beginner, so the reaction of the veteran is valuable. is used to store important data. In addition, there are cases in which a veteran checks the results once determined by a novice and points out mistakes. This is also a situation in which the inference model was not correctly communicated to the beginner, so we will use the valuable veteran's reaction and leave it as important data. Also, it is possible to generate a new inference model by considering the importance of data at that time depending on whether the mistake is light or serious.

As a result of the determination in step S13, if it is determined that the expected relationship is not met, the image is acquired (or requested to be acquired) as an important image (rare image) (S15). As a result of the determination in step S13, since the expected relationship is not met, the image at this time is an important image (rare image). , ask to get. As described above, the important image here may be a specific frame in a moving image, and does not need to be recorded as a still image. The image under inspection may be recorded at all times so that it is possible to record which frame the part corresponding to the important image frame is, and to record information for designating this specific image frame. It is easier to handle if there is a video of the examination, and there are many cases where other image information can be saved as evidence.

In step S15, if the image is acquired (or the acquisition is requested), then the degree of importance (rarity) is determined and the image is recorded (S17). Here, the contradiction determination unit 23 determines the importance (rarity) based on the doctor's reaction, the inference result, the reliability, the teacher data of the inference model, the past inference image, etc., and determines the importance (rarity). is higher than a predetermined value, the image determined to have no expected relationship is temporarily recorded in the recording unit 25 or recorded in the recording unit 33 . As described above, when the result of inference using the existing inference model 14a is contrary to the diagnosis result of the doctor, that is, the relationship between the inference result of the inference model and the action of the doctor who makes the diagnosis does not match the expected relationship. If not, the teacher data is created, and the image is recorded as new inference model data in order to create a new inference model. The determination operations in steps 13 to S17 will be described later using FIG. 4, taking the case of tumor detection as an example.

In step S17, when the degree of importance is determined and the image is recorded, or when the result of determination in step S13 is that there is an expected relationship, the important image determination flow of FIG. 3 ends.

Next, using FIG. 4, the determination of whether or not the relationship between the inference result of the inference model and the action of the doctor making the diagnosis is in the expected relationship will be described using an example of tumor detection. In this example, the existing inference model 14a is generated by performing deep learning on an image obtained during an endoscopy using teacher data annotating the position of a tumor or the like. When an image obtained by the image obtaining unit 13 is input to the existing inference model 14a, whether or not a tumor is detected, and if detected, its position is inferred and output. The calculation result of the reliability of the inference at this time is also output.

The vertical axis on the left side of the chart in FIG. 4 has columns for the inference results of tumor detection and the reliability of the inference results. There is a column for what was obtained from the treatment or what was obtained from the treatment.

In the chart of FIG. 4, images are not acquired in the following cases.
(A-1) When it is inferred that no tumor is detected, the reliability of the inference result is high, and the doctor's action supported by the chart is no tumor (A-2) It is inferred that no tumor is detected , when the reliability of the inference result is low and the doctor's action supported by the medical record is no tumor (A-3) When it is inferred that a tumor has been detected and the reliability of the inference result is high (A-4) When it is inferred that a tumor has been detected, and the reliability of the inference result is low, and the doctor's action supported by the medical record indicates that there is a tumor (A-5) When it is inferred that a tumor has been detected, and the reliability of the inference result is high, and there is treatment by a doctor (A-6) When it is inferred that a tumor has been detected, and that inference When the reliability of the result is low and there is a doctor's treatment

The above situations (A-1) to (A-6) are all cases in which the inference results and the doctor's diagnosis match. In other words, this is the case where the inference for the acquired image is accurately performed and the existing inference model 14a does not need to be modified.

On the other hand, in the chart of FIG. 4, an image is acquired as an important image in the following cases.
(B-1) A case where it is inferred that no tumor is detected and the reliability of the inference result is high, but the doctor's action supported by the medical record indicates that there is a tumor. In this case, the importance is judged to be large.
(B-2) When it is inferred that a tumor is not detected and the reliability of the inference result is low, but when the doctor's action supported by the medical record is a tumor (B-3) It is inferred that a tumor has been detected , if the inference result is highly reliable, but if the doctor's action supported by the chart is tumor-free. In this case, the importance is judged to be large.
(B-4) When it is inferred that a tumor has been detected and the reliability of the inference result is low, but when the doctor's action supported by the medical record is no tumor (B-5) It is inferred that no tumor has been detected. , if the inference result is highly reliable, but if there is a doctor's treatment. In this case, the importance depends on the degree of treatment.
(B-6) When it is inferred that no tumor is detected, and the reliability of the inference result is low, but there is treatment by a doctor. In this case, the importance depends on the degree of treatment.

The above situations (B-1) to (B-6) are all cases in which the inference results and the doctor's diagnosis do not match. In other words, there is a possibility that the inference for the acquired image is not accurately performed, and it is better to correct the existing inference model 14a. These images are acquired as important images.

Also, in the above example, if the inference result and the doctor's diagnosis result do not match despite the high reliability of the inference result, the importance is high. When the reliability is high, although the inference result and the doctor's diagnosis result are highly likely to match, the results are contradictory, so the image acquired at that time is a rare image. When generating a new inference model, the degree of importance may be considered. For example, an image with a high degree of importance may have a high priority for adoption as teacher data when generating a new inference model. Also, if there are many images with a high degree of importance, it may be used when determining the timing, such as advancing the timing of generating a new inference model. Also, when evaluating a new inference model or when calculating reliability, an image with a high degree of importance may be preferentially used.

Next, the operation of the hospital system 20 will be described using the flowchart shown in FIG. This flow is realized by the controller 21 in the hospital system 20 controlling each part in the hospital system 20 according to the program stored in the memory in the controller.

When the flow in FIG. 5 starts, first, patient information is entered (S21). Here, when a patient receives an appointment at a medical facility or the like, the patient information at this time is recorded in the recording unit 25 through the UI unit 22 within the hospital system 20 . Also, if there is already recorded information, it may be associated with the patient information.

Next, device cooperation, information sharing, etc. are performed (S23). In this case, the information can be shared by cooperating with a device 10 such as an endoscope used in a medical facility or the like. As will be described later, in steps S43 and S57 (see FIG. 6), the hospital system 20 and the device 10 cooperate and share information.

Next, it is determined whether or not the doctor's findings have been input (S25). After performing an examination using the device 10 such as an endoscope, the doctor inputs the doctor's findings and the like through the UI unit 22 and the like. For example, a doctor's opinion is input by text input or voice input using a keyboard.

As a result of the determination in step S25, if the doctor's findings or the like are entered, they are reflected in the chart (S27). Here, the text input result is reflected in an electronic chart or the like. In the case of voice input, it can be converted into text data using voice recognition. It should be reflected in the chart.

Subsequently, IC (informed consent), treatment, prescription, etc. are performed (S29). Here, informed consent refers to obtaining consent from the subject after giving sufficient explanation. In other words, the patient/families should fully understand the medical condition and treatment, how the medical staff received the patient/family's intentions, various situations and explanations, what kind of medical care to choose, and how the patient・It is a process of sharing information and reaching consensus among concerned parties such as family members, medical professionals, social workers and care managers. Obtain informed consent from patients and their families when informing them of the test results and providing treatment, etc. based on the test results. On that basis, treatment is given and prescriptions are given according to the patient's symptoms. There are ICs for medical practice, ICs for handling information that can be personal information, and related information.

Next, if there is an important image in the input image, the image is acquired and recorded (S31). The contradiction determination unit 23 determines whether or not the image acquired by the device 10 is an important image, and acquires and records this image if it is determined to be important. The determination as to whether or not the image is the important image is made according to the flow in FIG. 3 (see, for example, S15 in FIG. 3). That is, whether or not the image is an important image is determined based on whether or not there is an expected relationship between the inference result of the inference model and the action of the doctor who makes the diagnosis. When judging the relationship between the inference result and the doctor's diagnosis result, the degree of importance is also judged (S17 in FIG. 3, see FIG. 4). In this step, if the image is an important image, the image is acquired and recorded, and the degree of importance is obtained and recorded.

As mentioned above, the important image here may be a specific frame in a moving image, and does not need to be recorded as a still image. Images during inspection are recorded at all times, and it is possible to determine which frames correspond to important image frames, such as the number of minutes and seconds from a specific reference time such as the start of a moving image, or the number of frames. This specific image frame may be specified by recording whether or not. In other words, it is not necessary to separately record specific image frames from those recorded as continuous frame images such as moving images and continuous shot frame images. Here too, it is easier to handle if there is a moving image of the examination, and there are many cases where other image information can be left as evidence.

If the important image is recorded in step S31, or if the doctor's findings are not input as a result of the determination in step S25, it is determined whether or not the medical examination has ended (S33). When the doctor completes the examination of the patient in step S21, the doctor inputs that effect to the hospital system 20 through the UI unit 22, so determination is made based on whether or not this input has been made. As a result of this determination, if the medical examination has not ended, the process returns to step S23 and the above-described operations are executed.

On the other hand, if the result of determination in step S33 is that the medical examination has ended, accounting, reservation prescription information, etc. are performed (S35). Since the medical examination is completed, the patient completes the payment of the medical examination fee, etc., makes an appointment for the next medical examination if necessary, and receives the prescription, if any. After executing these procedures, the flow of the hospital system ends.

Next, the operation of the device 10 such as an endoscope will be described using the flowchart shown in FIG. This flow is realized by the controller 11 in the device 10 controlling each part in the device 10 according to the program stored in the memory in the controller.

When the flow in FIG. 6 starts, it is first determined whether or not to cooperate with the hospital system (S41). As described above, the equipment 10 such as an endoscope and the hospital system 20 can cooperate with each other. In this step, it is determined whether or not there is an in-hospital system 20 with which the device 10 cooperates. If there are devices to be linked with the hospital system 20 in the medical facility, setting processing may be performed in advance so that they can be linked.

If the result of determination in step S41 is that cooperation with the hospital system is possible, cooperation and information are shared (S43). If communication between the hospital system 20 and the device 10 is enabled by wireless communication or wired communication, then in this step the two will cooperate and share information. As for cooperation, for example, image data and related information acquired by the device 10 are transmitted to the hospital system 20 , and the existing inference model received by the hospital system 20 from the management server 30 is transmitted to the device 10 . In addition, the inference performed using the existing inference model 14a does not necessarily have to be performed in the device 10. An inference unit is arranged in the hospital system 20, and inference is performed based on the image transmitted from the device 10 to the hospital system 20. may be performed, and the inference result may be sent back to the device 10 .

If cooperation and information sharing are performed in step S43, or if the result of determination in step S41 is that there is no cooperation, then patient information and the like are input (S45). Here, patient information such as the patient's name and examination items is input through the UI unit (not shown) of the device 10 . As the information, all information regarding the patient may be entered, or only necessary information may be entered. Further, the hospital system 20 may transfer the information input in step S21 (see FIG. 5) to the device 10. FIG.

Next, prepare for inspection (S47). Here, preparations are made for examination using the device 10 such as an endoscope. For example, in the case of endoscopic examination, an endoscope is prepared according to the examination site, and the patient is asked to drink an antifoaming agent and an anesthetic and enter the examination room. When the preparation for the inspection is finished, it is next determined whether or not the inspection is in progress (S49). Here, if the doctor or the like is using the device 10 such as an endoscope, the control unit 11 determines that the examination is being performed.

As a result of the determination in step S49, when the examination starts, first, diagnostic guidance, diagnostic assistance determination, etc. are performed, and evidence recording, etc. is performed (S51). Here, a diagnosis guide and a diagnosis assistance determination are performed when a doctor makes a diagnosis using the device 10 such as an endoscope. For example, as a diagnostic guide, a method of operating the device 10 may be displayed, and as diagnostic assistance determination, whether or not a lesion such as a tumor is present in an acquired image may be displayed. In addition, data that serves as evidence, such as still images and videos during examinations, is recorded. If the device 10 has a recording unit, the data may be recorded in this recording unit or may be recorded in the recording unit 25 within the hospital system 20 .

Next, if there are important images, etc., request image acquisition (S53). Whether or not an image is an important image is determined according to the flow shown in FIG. That is, whether or not the image is an important image is determined based on whether or not there is an expected relationship between the inference result of the inference model and the action of the doctor who makes the diagnosis. If the device 10 can determine whether the image is important, it acquires the image. If the determination cannot be made within the device 10, the in-hospital system 20 may be requested to make the determination in step S57. As described above, the image acquisition here may be to allow designation of a specific frame in a moving image, and does not have to be recorded as a still image. It is easier to handle if there is a video of the examination, and there are many cases where other image information can be saved as evidence. In many cases, it is troublesome to record important videos, including those that serve as evidence, and then leave separate still images and manage them in association with each other.

Next, it is determined whether or not to cooperate with the hospital system (S55). Similar to step S41, it is determined whether or not there is an in-hospital system 20 with which the device 10 cooperates. As a result of this determination, if cooperation with the hospital system is possible, cooperation and information are shared (S57). Here, as in step S43, cooperation is established between the hospital system 20 and the device 10 to share information.

If cooperation and information sharing are performed in step S57, or if the result of determination in step S55 is that there is no cooperation with the hospital system, it is next determined whether or not to end (S59). When the examination using the device 10 such as an endoscope is completed, the doctor inputs that fact through the UI unit in the device 10, so determination is made based on whether or not this input has been made. If the result of this determination is that the inspection has not ended, the process returns to step S49 and the above-described operations are executed.

On the other hand, if the result of determination in step S59 is that it has ended, and if there is an important image or the like, the IC or the like is also recorded (S61). Here, together with the important image in step S53, the fact that informed consent has been given in step S29 (see FIG. 5) is recorded. In order to obtain informed consent (IC), print out an explanatory document to the effect that the subject's data will be used in AI development research, and present it to the subject. Then, the doctor or the like inputs the subject's agreement or non-agreement, and the subject clicks an icon of an agreement button or a disagreement button. In addition to this method, a consent form may be signed by the subject, signed by a doctor or the like, and the signed consent form may be scanned. Alternatively, the IC may be obtained by e-mail, entry on the Web, or the like. In any case, it is sufficient to ensure that sufficient explanation is provided, that the individual agrees, and that these facts are recorded. IC may be performed before the examination, and may be performed after the examination as long as the sedative has worn off and the patient is in a normal state. When the processing in step S61 ends, the operation of the device in FIG. 6 ends.

In this way, in the operation flow of the in-hospital system in FIG. 5 and the operation flow of the device (endoscope) in FIG. If the image is determined to be an important image, the image is acquired and recorded (for example, see S31 and S53). In the flow shown in FIGS. 5 and 6, in steps S31 and S53, it is determined whether or not the image is an important image, and if it is an important image, it is recorded. Either of them may carry out, or both may carry out in cooperation with each other.

In the present embodiment, there are several patterns in which an image is determined to be an important image (rare image). For example, there are the following cases.
(Pattern 1) Cases in which doctors, etc. did not perform treatment even though AI discovered a tumor. In this case, whether it was an erroneous diagnosis or whether it was decided not to treat the polyp because it is a safe polyp that does not turn into cancer is linked to the information in the electronic medical record.
(Pattern 2) When treatment was performed on a site where AI could not detect a tumor. In this case, it is possible to determine whether or not the image is an important image (rare image) to be learned based on the image being treated and the information in the electronic medical record.

Next, using FIG. 7, a method of using an inference model to select image data for teacher data will be described. In the above-described embodiment, when a doctor performs an examination/diagnosis using the device 10 such as an endoscope, the doctor takes an action that the inference result of the existing inference model is contrary to the doctor's diagnosis result. In the event that the On the other hand, in the example shown in FIG. 7, whether or not an image is an important image is determined by inputting an image or the like when a doctor diagnoses it into an inference model for extracting teacher data candidates. In this example, an inference model is generated for inferring a case where the inference result of the existing inference model and the doctor's diagnosis result contradict each other (see FIG. 7(a)). When this inference model is generated, a test image can be input to this inference model during normal examination/diagnosis, and training data candidates (annotation candidates) can be extracted (see FIG. 7B).

FIG. 7(a) shows deep learning for generating an inference model for inferring training data candidates. A series of inspection image groups P11 to P15 are images acquired when a doctor or the like performs inspection Ex1 using a device 10 such as an endoscope. timing). Inspection images P11 and P12 are a group of images when approaching the discrimination part, image P13 is an image in the vicinity of a site (differentiation part) that is suspected to be a tumor, and images P14 and P15 are images of the discrimination part. This is a group of images when moving away from . These inspection image groups P11 to P15, P21 to P25, and P31 to P35 are recorded in the recording unit. Note that the images P22 and P33 are images near the discrimination portion.

After acquiring the image groups P11 to P15, P21 to P25, and P31 to P35 during the inspection, these images are then annotated to create training data. In this case, at least images P13, P22, and P33 have lesions (discrimination portions) such as tumors, and teacher data T13, T22, and T33 annotated so that their positions can be identified are created. When the teacher data is created, the teacher data is input to the input layer IN of the neural network NNW for generating the inference model, and the position of the lesion (discrimination part) is output from the output layer OUT as the discrimination result Output1. , determine the weighting of the intermediate layers. Learning is performed to make an inference that an image with such a tendency should be training data. That is, when the learning is completed, an inference model is created that outputs the image as a training data candidate when the inference result of the existing inference model contradicts the doctor's diagnosis result when an inspection image is input.

As described above, the important images in the learning required here, such as the teacher data T13, T22, and T33, are specific frames in the moving image recorded as still images. No need. Images are recorded at all times during the inspection, and it is possible to determine which frames correspond to the important image frames, such as how many minutes and seconds from a specific reference time such as the start of a moving image, or how many frames. This specific image frame may be specified by recording whether or not. In other words, it is not necessary to separately record the specific image frames of the teacher data one by one from those recorded as continuous frame images such as moving images and continuous shot frame images. Here too, it is easier to handle if there is a moving image of the examination, and there are many cases where other image information can be left as evidence. Based on this idea, it is not necessary to extract and record the important image from the moving image in FIG.

FIG. 7(b) shows an inference operation of outputting an annotation candidate image when an acquired inspection image is input using the inference model generated in FIG. 7(a). In this example, a doctor performs an examination Exa and acquires examination images Px1 to Px5. When the obtained inspection images Px1 to Px5 are input to the input layer In of the inference engine having the neural network NNW, the annotation candidate Output2 is output from the output layer Out. By performing this inference, even if the conventional AI (inference using an existing inference model) does not output an image as an annotation candidate at the critical moment, By using the inference model, it is possible to obtain an image to be used as training data. The acquired image here is also a specific frame in a moving image, and does not need to be recorded as a still image. For example, it is possible to specify the number of minutes and seconds from a specific reference time such as the start of a movie, or the number of frames, etc. If necessary, it may be possible to record separately from the movie. Alternatively, a signal specifying a specific image frame of a moving image may be recorded.

In FIG. 7A, when acquiring the inspection image group P31 to P35, the operation information Op1 to Op3 may be acquired together and recorded in association with the inspection image group. Operation information is omitted in the examinations Ex1 and Ex2 of FIG. Operation information is associated and recorded. Operation information is also associated when creating teacher data, and by performing deep learning using this teacher data, an inference model that takes into account the doctor's operation information is generated. At the time of inference in FIG. 7(b), in addition to the inspection images Px1 to Px5, operation information Opx1 to Opx3 are input to the input layer In of the neural network NNW to infer annotation candidates in consideration of the doctor's operation. be able to.

In this way, in the example shown in FIG. 7, once the inference model for extracting teacher data candidates is generated, if a test image is subsequently input to this inference model, it is possible to determine whether the inference result of the existing inference model contradicts the doctor's diagnosis result. If it is inferred that the inference result and the diagnosis result do not match, the image can be output as a teacher data candidate. Therefore, it is possible to determine whether or not an image is an image to be a teacher data candidate without displaying the image acquired by the device 10 such as an endoscope. In other words, it is possible to determine whether or not the relationship was as expected based on the action taken by the doctor in a state where the inference result of the inference unit is not displayed.

As described above, in one embodiment of the present invention, inference is performed using an inference model for inferring and displaying a lesion from an image (see, for example, S1 in FIG. 2), and the inference result and When the action relationship of the doctor diagnosing the lesion is not in the expected relationship (for example, see S3 Yes, S5, etc. in FIG. 2), the image data at the time of display is acquired (for example, FIG. 2, S5, etc.). Therefore, rare (important) data to be input to the inference unit can be obtained. This acquired data can be efficiently acquired for generating a new inference model, and for calculating the evaluation and reliability of the new inference model. In addition, the acquired data can be used when generating a new inference model.

Further, in one embodiment of the present invention, inference is performed using an inference model for displaying an operation guide for an endoscope (see, for example, S1 in FIG. 2, FIG. 7B, etc.), and operation guide display and , the operation-related data when the operation guide is displayed is acquired when the relationship between the actions of the doctor who operates the endoscope is not the expected relationship (for example, S3 Yes in FIG. 2, S5 in FIG. 7 etc.). The acquired data can be efficiently acquired for generating a new inference model for operation guidance, and for calculating the evaluation and reliability of the new inference model for operation guidance. In addition, the acquired data can be used when generating a new inference model.

Further, in one embodiment of the present invention, sensor data provided in the medical device or operation data of the medical device by the doctor is input to the inference model for inference (for example, S1 in FIG. 2, FIG. 7B) etc.), and when the relationship between the result of inference and the action of the doctor using the medical device does not meet the expected relationship, an identification code is given to the sensor data or the operation data when the expected relationship does not exist. (See, for example, S3 Yes, S5 in FIG. 2, FIG. 7, etc.). The data to which this identification code is assigned can be used for efficiently collecting data for generating a new inference model and calculating the evaluation and reliability of the new inference model. Also, the data with the identification code can be used when generating a new inference model. In the case of a medical device used by a specialist such as a doctor, even if an inference model for the medical device is generated, there may be a case where the inference model cannot perform appropriate inference when actually used. Even in such a case, according to this embodiment, it is possible to find out data contrary to the opinion of experts such as doctors. In addition, it is possible to modify and generate inference models using collected data quickly and efficiently.

In addition, in one embodiment of the present invention, the explanation was focused on the endoscopic image, but the present invention can also be applied to an information processing device that generates an inference model using images of various inspection devices other than the endoscopic image. can. In other words, the technique of selecting training data candidates from time-series image frames is expected to be used in various fields. In the present embodiment, an example has been described in which an inference result for an image acquired in a body cavity unique to an endoscope and an evaluator's table of the image do not match, and are selected as teaching data candidates or the like. However, such a so-called normalized procedure is not limited to endoscopic imaging, as it can be in any field.

Further, in one embodiment of the present invention, the device 10, the hospital system 20, and the management server 30 are described as separate entities, but these three may be configured integrally. may be configured integrally. Also, in one embodiment of the present invention, the controllers 11, 21, and 31 have been described as devices configured from CPUs, memories, and the like. However, in addition to being configured as software by a CPU and a program, part or all of each part may be configured as a hardware circuit, and is described in Verilog, VHDL (Verilog Hardware Description Language), etc. A hardware configuration such as a gate circuit generated based on a program language may be used, or a hardware configuration using software such as a DSP (Digital Signal Processor) may be used. Of course, these may be combined as appropriate.

In addition, the control units 11, 21, and 31 are not limited to CPUs, and may be elements that function as controllers. good. For example, each unit may be a processor configured as an electronic circuit, or may be each circuit unit in a processor configured with an integrated circuit such as an FPGA (Field Programmable Gate Array). Alternatively, a processor composed of one or more CPUs may read and execute a computer program recorded on a recording medium, thereby executing the function of each unit.

Also, in one embodiment of the present invention, the device 10 has been described as having the control unit 11, the display unit 12, the image input unit 13, the inference unit 14, and the operation result reflection unit 15. However, they do not need to be provided in an integrated device, and the above-described units may be distributed as long as they are connected by a communication network such as the Internet. Similarly, the hospital system 20 has been described as having the control unit 11 , the UI unit 22 , the contradiction determination unit 23 , the communication unit 24 and the recording unit 25 . However, they do not need to be provided in an integrated device, and the above-described units may be distributed as long as they are connected by a communication network such as the Internet. Furthermore, the management server 30 has been described as having a control unit 31 , an annotation/learning unit 32 , a recording control unit 34 and a communication unit 35 . However, they do not need to be provided in an integrated device, and the above-described units may be distributed as long as they are connected by a communication network such as the Internet.

In addition, in recent years, artificial intelligence that can collectively determine various judgment criteria is often used, and improvements such as collectively performing each branch of the flow chart shown here are also within the scope of the present invention. It goes without saying that the If the user can input good or bad for such control, it is possible to learn the user's preference and customize the embodiment shown in the present application in a direction suitable for the user.

In addition, among the techniques described in this specification, the control described mainly in the flowcharts can often be set by a program, and may be stored in a recording medium or recording unit. The method of recording in the recording medium and the recording unit may be recorded at the time of product shipment, using a distributed recording medium, or downloading via the Internet.

In addition, in one embodiment of the present invention, the operation in this embodiment was explained using a flowchart, but the order of the processing procedure may be changed, or any step may be omitted. Steps may be added, and specific processing contents within each step may be changed.

In addition, even if the operation flow in the claims, the specification, and the drawings is explained using words expressing the order such as "first" and "next" for convenience, in places not specifically explained, It does not mean that it is essential to carry out in this order.

The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the spirit of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components of all components shown in the embodiments may be deleted. Furthermore, components across different embodiments may be combined as appropriate.

DESCRIPTION OF SYMBOLS 10... Equipment, 11... Control part, 12... Display part, 12a... Inference result, 12b... Reliability, 13... Image acquisition part, 14... Inference part, 14a Existing inference model 14b Category determination unit 14c Image similarity determination unit 15 Operation result reflection unit 20 In-hospital system 21 Control unit 22 UI unit 23 contradiction determination unit 24 communication unit 25 recording unit 30 management server 31 control unit 32 annotation/learning unit 32b . 35... Communication unit

Claims (12)

1. an inference unit having an inference model for inferring and displaying lesions from images;
   an acquisition unit that acquires image data when the display is performed when the relationship between the inference result of the inference unit and the action of a doctor who diagnoses the lesion is not in the expected relationship;
   A data acquisition system comprising:
2. 2. The image data acquired by the acquisition unit when the display is performed is a moving image including image frames at the time of the display, and is data including information capable of specifying the image frames. The data acquisition system described in .
3. 　The data acquisition system according to claim 1, further comprising a recording unit that adds information about the rarity of the image to the image acquired by the acquisition unit and records the image.
4. 3. The data acquisition system according to claim 1, further comprising a training data candidate selection unit that selects the image data acquired by the acquisition unit as training data candidates.
5. The image data acquired by the acquisition unit when the display is performed is a time-series image group including the image frames at the time of the display, and is data containing information capable of specifying the image frames. The data acquisition system of Claim 1.
6. 　The data acquisition system according to claim 1, characterized by having a database that records an expected relationship between the inference result obtained using the inference model and the action of the doctor who makes the diagnosis.
7. an image similarity determination unit that determines the training data candidate according to the degree of similarity between the image when the lesion is displayed and the training data group when the inference model is learned;
   2. The data acquisition system according to claim 1, wherein the acquisition unit acquires the image determined as the teacher data candidate by the image similarity determination unit.
8. 　The data acquisition system according to claim 1, characterized in that, in a state in which the inference results of the inference unit are not displayed, it is determined whether or not the relationship is the expected relationship based on the action taken by the doctor.
9. Inference using an inference model for inferring and displaying lesions from images,
   Acquiring the image data when the display is performed when the relationship between the result of the inference and the action of the doctor who diagnoses the lesion is not in the expected relationship.
   A data acquisition method characterized by:
10. inference using an inference model for displaying an endoscope operation guide,
    Acquiring operation-related data when the operation guide display is performed when the relationship between the operation guide display and the action of the doctor who operates the endoscope is not in an expected relationship;
    A data acquisition method characterized by:
11. Inference by inputting sensor data provided in the medical device or operation data of the medical device by the doctor into the inference model,
    When the relationship between the result of the inference and the action of the doctor using the medical device does not meet the expected relationship, an identification code can be assigned to the sensor data or the operation data when the expected relationship does not exist. to make
    A data processing method characterized by:
12. an inference unit that inputs sensor data provided in a medical device or operation data of a doctor on the medical device into an inference model and makes an inference;
    When the relationship between the result of the inference and the action of the doctor using the medical device does not meet the expected relationship, an identification code can be assigned to the sensor data or the operation data when the expected relationship does not exist. a data processing unit for
    A data processing system comprising:

Images