JP7343470B2 - Control device, control method, and control program - Google Patents

Description

The present invention relates to a control device, a control method, and a control program for controlling a medical treatment device for treating a patient.

Conventionally, medical treatment devices for treating patients are known. For example, Patent Document 1 discloses a medical treatment device including a chair, medical instruments, a lighting device, and other configurations for treating a patient.

Patent No. 6388849

According to the medical treatment device disclosed in Patent Document 1, a surgeon can treat a patient by using each configuration of the medical treatment device according to the contents of the medical treatment. Considering that in recent years, technology has been created that uses AI (Artificial Intelligence) to analyze, learn, and utilize data, using AI technology in medical equipment will make medical treatment more convenient for surgeons. equipment can be provided.

An object of the present invention is to provide a control device, a control method, and a control program that can improve convenience for a surgeon.

According to an example of the present disclosure, a control device that controls a medical treatment device for treating a patient is provided. The control device includes an input section into which image-related data captured by a camera of a medical treatment space including medical treatment equipment is input, and a detection device that detects the position of a person included in the medical treatment space based on the image-related data input from the input section. and at least one position data for specifying the position of the person detected by the detection unit, into a first estimation model including a first neural network, and the medical device is output as an output of the first estimation model. It includes a generation section that generates control data for controlling, and an output section that outputs the control data generated by the generation section to a controlled object.

According to an example of the present disclosure, a control method for controlling a medical treatment device for treating a patient using a computer is provided. The control method includes the steps of inputting image-related data captured by a camera of a medical treatment space including medical equipment, and detecting the position of a person included in the medical treatment space based on the image-related data, as processes executed by a computer. and, by inputting at least one position data for specifying the position of the person detected in the detecting step into the first neural network, control data for controlling the medical treatment device is generated as an output of the first estimation model. and outputting the generated control data to a controlled object.

According to an example of the present disclosure, a control program for controlling a medical treatment device for treating a patient is provided. The control program includes a step of detecting the position of a person included in the medical treatment space based on the image-related data, and a first step of transmitting at least one position data for specifying the position of the person detected in the detecting step. a step of generating control data for controlling the medical device as an output of the first estimation model by inputting the input into a first estimation model including a neural network; and a step of outputting the generated control data to a controlled object. Execute.

According to the present disclosure, convenience for the operator can be improved.

FIG. 1 is a schematic diagram showing the overall configuration of a control system according to the present embodiment. FIG. 3 is a diagram for explaining an example of an image taken by a tray camera. FIG. 3 is a diagram for explaining an example of an image taken by a patient camera. FIG. 3 is a diagram for explaining an example of an image taken by a general camera. FIG. 1 is a block diagram showing the internal configuration of a control system according to the present embodiment. FIG. 3 is a diagram for explaining an example of position data acquired by the control device according to the present embodiment. FIG. 3 is a diagram for explaining an example of detection points of position data. FIG. 3 is a diagram for explaining an example of position coordinates corresponding to position data. FIG. 2 is a block diagram showing the functional configuration of the control device according to the present embodiment at a learning stage. FIG. 2 is a block diagram showing the functional configuration of the control device according to the present embodiment at an operation stage. It is a figure for explaining an example of the photographed image of the tray camera in root canal treatment. FIG. 3 is a diagram for explaining an example of tray image data acquired from photographic information of a tray camera. FIG. 3 is a diagram for explaining an example of patient image data acquired from photographic information of a patient camera. FIG. 3 is a diagram for explaining an example of position data acquired from photographic information of a general camera. FIG. 3 is a diagram for explaining an example of medical care-related data acquired from a medical device. FIG. 3 is a diagram for explaining an example of synchronous data. FIG. 3 is a diagram for explaining an example of synchronous data. FIG. 3 is a diagram for explaining an example of control by a control device. FIG. 3 is a diagram for explaining an example of control by a control device. FIG. 3 is a diagram for explaining an example of control by a control device. 3 is a flowchart for explaining an example of a control data generation process executed by a control device.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are designated by the same reference numerals, and the description thereof will not be repeated.

<Overall configuration of control system>
FIG. 1 is a schematic diagram showing the overall configuration of a control system 1000 according to this embodiment. A control system 1000 according to this embodiment controls a medical treatment device 1 for treating a patient.

“Medical treatment” includes at least one of medical examination and treatment. "Examination" includes the operator's investigation of the patient's medical condition and etiology. "Treatment" includes a practitioner curing a patient's illness and a practitioner maintaining the patient's beauty and health (eg, aesthetic treatment). The subject of medical treatment (examination, treatment) includes at least one of dentistry and medicine. "Dentistry" is a medical department that deals with diseases related to teeth or tooth-related tissues (periodontal tissues, etc.), and includes, for example, general dentistry, orthodontics, oral surgery, dental radiology, pediatric dentistry, and the like. "Medical" includes, for example, internal medicine, pediatrics, surgery, ophthalmology, otorhinolaryngology, obstetrics and gynecology, dermatology, neurosurgery, cardiology, and orthopedics. The medical treatment device 1 according to the present embodiment is used by an operator to perform dental treatment on a patient. “Surgeons” include doctors who treat patients (e.g., dentists), assistants who assist doctors (e.g., dental assistants), teachers at dental or medical schools, students at dental or medical schools, and dental technicians. including.

As shown in FIG. 1, the control system 1000 includes a medical device 1 and a control device 100 that controls the medical device 1. The medical treatment device 1 is, for example, a chair unit for an operator to perform dental treatment on a patient. The medical device 1 includes a chair 11, a basin unit 12, a tray table 13, an instrument holder 14 holding medical instruments 15, a foot controller 16, a display 17, an operation panel 18, a lighting device 19, and instruments. It includes a control device 21, a display control device 22, and a speaker 35.

The chair 11 is a chair on which a patient sits during medical treatment, and includes a headrest 11a that supports the patient's head, a backrest 11b that supports the patient's back, a seat sheet 11c that supports the patient's tail, and a footrest that supports the patient's feet. 11d.

The basin unit 12 is a water supply/drainage device that includes a bowl 12a in which a drainage port is formed, a cup stand 12b on which a cup is placed, and a water tap 12c for supplying water to the cup.

The tray table 13 is used as a storage stand during medical treatment. The tray table 13 is connected to an arm (not shown) extending from the chair 11 or the floor on which the chair 11 is installed. Note that the tray table 13 may be suspended by an arm (not shown) extending from the chair 11 or the floor on which the chair 11 is installed. For example, the tray table 13 may be suspended by an arm 6 that branches at the top of a pole 5 that extends from the chair 11 or the floor on which the chair 11 is installed. The operator can manually rotate, horizontally move, and vertically move the tray table 13 with respect to the chair 11. During medical treatment, the surgeon may place the tray 30 on the upper surface of the tray table 13. One or more medical instruments for treating a patient are placed on the tray 30.

"Medical instruments" are medical instruments used by a surgeon during medical treatment, such as tweezers, mirrors, excavators, deep needles, and microscopes. Although the medical instruments 15 held by the instrument holder 14 of the medical device 1 are also included in the "medical instruments," medical instruments that are not held by the instrument holder 14 are mainly placed on the tray 30.

The medical instrument 15 is an instrument for dental treatment, such as an air turbine handpiece, a micromotor handpiece, an ultrasonic scaler, and a vacuum syringe, and is driven under the control of the instrument control device 21 . The medical instruments 15 are not limited to these, and may be an intraoral camera, a photopolymerization irradiator, a root canal length measuring device, a three-dimensional scanner, a root canal expander, or a mirror, a syringe, and a root canal expander. It may also be a non-driven device, such as a filling device.

The foot controller 16 has a plurality of switches (pedals) that accept foot operations by the operator. The operator can assign a predetermined function to each of the plurality of switches. For example, the surgeon can assign the function of changing the posture of the chair 11 to a switch on the foot controller 16, and when the foot controller 16 receives a foot operation of the switch by the surgeon, the headrest will be adjusted based on the foot operation. 11a etc. are driven. Furthermore, the operator can also assign the function of driving the medical instrument 15 to a switch of the foot controller 16, and when the foot controller 16 receives a foot operation of the switch by the operator, the instrument is controlled based on the foot operation. The medical instrument 15 is driven by the control of the device 21. Note that the surgeon can also assign other functions to the switches of the foot controller 16, such as a function to control lighting and turning off of the lighting device 19.

The display 17 is attached to the tray table 13 and displays various images under the control of the display control device 22.

The operation panel 18 includes switches for operating the chair 11, the medical instruments 15, etc., or for setting the operations. For example, when the operation panel 18 receives an input operation for operating the chair 11, the chair 11 operates based on the input operation. For example, when the operation panel 18 receives an input operation for setting the rotational speed of the medical instrument 15, etc., the rotational speed and the like of the medical instrument 15 can be set based on the input operation.

The lighting device 19 is provided at the tip of an arm 6 that branches at the top of the pole 5 extending from the chair 11 or the floor on which the chair 11 is installed. The lighting device 19 can be switched between a lighting state and a non-lighting state, and supports medical treatment by a surgeon by illumination. Note that not only the lighting device 19 but also the display 17 may be attached to the tip of the pole 5 or the arm 6.

The speaker 35 outputs various sounds, such as a sound to alert the surgeon and the patient, and an assist sound to assist in medical treatment.

An operator can treat a patient by using each configuration of the medical device 1 according to the contents of the medical treatment. For example, when placing a patient in a chair, the operator operates the foot controller 16 to operate the chair 11 so that the patient is in a comfortable sitting position. For example, when treating a patient, the surgeon switches the illumination device 19 from an off state to an illuminated state to make it easier to visually recognize the inside of the patient's oral cavity. For example, when treating a patient, a surgeon drives the medical instrument 15 by operating the foot controller 16.

In this way, the surgeon can treat the patient by operating each component of the medical treatment device 1 while treating the patient. If the medical device 1 can be controlled by reading ahead using the medical information, it is possible to provide the medical device 1 which is highly convenient for the operator.

Here, during medical treatment, the dentist, dental assistant, and patient each perform actions according to a routine that is generally determined depending on the content of the medical treatment, and take approximately predetermined postures depending on the content of the medical treatment. The order and posture of the actions performed by the dentist, the dental assistant, and the patient can be said to represent a medical treatment procedure, and the medical treatment apparatus 1 is also controlled to correspond to the procedure. Therefore, if the procedure of medical treatment can be understood based on the order of actions and postures performed by each of the dentist, dental assistant, and patient, it becomes possible to predict the control of the medical treatment apparatus 1 in advance.

Further, the operator selects an appropriate medical instrument from among the plurality of medical instruments according to the medical treatment content, takes it out from the tray 30, and performs medical treatment using the taken out medical instrument. The type of medical instrument selected and the order in which the medical instruments are used can be said to represent a medical procedure, and the medical apparatus 1 is also controlled to correspond to the procedure. Therefore, if the procedure of medical treatment can be understood based on the type of medical instruments selected by the operator and the order in which the medical instruments are used, it becomes possible to predict the control of the medical apparatus 1 in advance.

Therefore, the control system 1000 (control device 100) according to the present embodiment performs control for controlling the medical treatment device 1 by analyzing and learning the behaviors of dentists, dental assistants, and patients using AI. Provide technology for estimating data. Furthermore, the control system 1000 (control device 100) according to the present embodiment uses AI to analyze and learn the types of medical instruments selected by the surgeon and the order in which the medical instruments are used. The present invention provides a technique for estimating control data of No. 1. Hereinafter, control of the medical treatment device 1 by the control system 1000 (control device 100) according to the present embodiment will be specifically explained.

A plurality of cameras are attached to the medical device 1 according to this embodiment. Specifically, the control system 1000 includes a tray camera 51 attached to the display 17, a patient camera 52 attached to the illumination device 19, and an overall camera 53 attached to the top of the pole 5.

The tray camera 51 is arranged to take a video or still image of an area including at least the tray 30. Image data including photographic information obtained by the tray camera 51 (hereinafter also referred to as “tray image data”) is acquired by the control device 100. Various settings such as the shutter speed in the tray camera 51 are adjusted in advance to such an extent that the control device 100 can recognize the presence or absence, shape, type, etc. of medical instruments placed on the tray 30. Note that the tray camera 51 may be installed at any location as long as it can photograph an area including at least the tray 30.

The patient camera 52 is arranged so as to take a video or still image of an area including at least the inside of the patient's oral cavity. Normally, during medical treatment, the inside of the patient's oral cavity is illuminated by the illumination device 19, so the patient camera 52 attached to the illumination device 19 can naturally photograph an area including at least the inside of the patient's oral cavity. Image data including imaging information obtained by the patient camera 52 (hereinafter also referred to as “patient image data”) is acquired by the control device 100. Various settings such as the shutter speed in the patient camera 52 are adjusted in advance to the extent that the control device 100 can recognize the inside of the oral cavity of the patient during treatment. Note that the patient camera 52 may be installed at any location as long as it can photograph an area including at least the inside of the patient's oral cavity. The patient camera 52 is a three-dimensional camera similar to the general camera 53 so that the control device 100 can detect the positional relationship in the depth direction (for example, the direction in which the patient is viewed from the patient camera 52) between the area around the oral cavity and the medical instruments. The position coordinates may be detectable.

The general camera 53 is arranged so as to photograph a region including at least the medical treatment space including the medical treatment device 1 as a moving image or a still image. Specifically, the overall camera 53 is arranged so as to take a moving image or a still image of a wide area including at least the dentist, dental assistant, and patient in the medical treatment space during medical treatment using the medical treatment device 1. There is. Since the general camera 53 is attached to the upper part of the pole 5, it photographs the actions of the dentist, dental assistant, and patient during treatment, as well as the operation of the medical treatment apparatus 1 from above.

The overall camera 53 is configured with a 3D camera or the like so as to be able to detect the three-dimensional position coordinates of the operator and the patient. For example, the overall camera 53 is a ToF (Time of Flight) camera that can detect three-dimensional position coordinates by photographing with a camera and measuring the distance to an object using reflection of light (for example, infrared rays), or , a stereo camera that can detect three-dimensional position coordinates by taking pictures with two cameras.

The general camera 53 includes a camera that takes images from the front of the chair 11 of the medical device 1 (or a patient sitting on the chair 11) as shown in FIG. The camera may also include a camera that takes pictures from the back of the patient. In this way, the overall camera 53 can photograph the medical treatment space not only from the front of the medical treatment device 1 but also from the back, so it can photograph the medical treatment space in more detail and over a wider range without blind spots. be able to. Further, the general camera 53 may be capable of tracking a subject to be imaged (for example, a person such as a surgeon and a patient) present in the medical treatment space.

The "medical treatment space" is not limited to a space that includes all of the objects included in the treatment space, but also includes at least some of the objects included in the treatment space, such as the dentist, dental assistant, patient, and medical treatment equipment 1. It may be a space containing objects. Image data (hereinafter also referred to as “overall image data”) including photographic information obtained by the overall camera 53 is acquired by the control device 100. Various settings such as the shutter speed in the general camera 53 are adjusted in advance to such an extent that the control device 100 can recognize the dentist's actions, the dental assistant's actions, the patient's actions, the actions of the medical device 1, and the like. Note that the general camera 53 may be installed at any location as long as it can take at least an image of the medical treatment space including the medical treatment device 1.

Each of the above-mentioned tray image data, patient image data, and whole image data is one embodiment of "image-related data," and these data will also be referred to as "image-related data" below. Note that the phrase "image-related data" may mean all of tray image data, patient image data, and overall image data; however, it may refer to at least any of tray image data, patient image data, and overall image data. It can also mean one of the following.

Further, as will be described in detail later, the control device 100 detects the presence or absence of an object such as a medical instrument shown in the captured image and its type based on the tray image data and patient image data. The control device 100 detects the positions of people, such as a dentist, a dental assistant, and a patient, shown in the captured image based on the overall image data. The "image related data" may include the detection results by the control device 100 described above. That is, "image-related data" may include tray image data, patient image data, and whole image data themselves, or may include detection results obtained by control device 100 based on these data. .

The control device 100 is equipped with a computer (computation device 102 described later) that estimates control data for the medical device 1 based on various acquired data. Specifically, the control device 100 acquires image data from at least one of the tray camera 51, patient camera 52, and overall camera 53 attached to the medical treatment device 1. Furthermore, the control device 100 acquires data related to medical treatment (hereinafter also referred to as “medical treatment-related data”) acquired by the medical treatment device 1. The control device 100 estimates control data for the medical device 1 based on at least one of the acquired image data and medical care-related data.

"Medical care related data" includes at least any of the chair 11, basin unit 12, lighting device 19, instrument control device 21, display control device 22, medical instrument 15, foot controller 16, display 17, and operation panel 18 provided in the medical device 1. The control data includes at least one past and present control data for one of the above. For example, "medical care-related data" includes the chair 11, basin unit 12, lighting device 19, instrument control device 21, display control device 22, medical instrument 15, foot controller 16, display 17, and operation panel 18 provided in the medical device 1. Contains log data showing the operation and control history of each. Note that the "medical care-related data" is not limited to historical data of operations and controls in various devices such as log data, but may include real-time data of operations and controls in various devices. Furthermore, the "medical treatment-related data" may include data regarding the current status of various devices. For example, the "medical treatment-related data" may include data for specifying the current state (posture, position, etc.) of the chair 11.

<Medical content>
Examples of treatment in medical practice include caries treatment, root canal treatment, plaque removal, implants, orthodontics, and whitening. Any cosmetic treatment that maintains the beauty and health of a patient's teeth is included in the scope of medical treatment.

Actions (treatments) that a surgeon performs to cure a patient's dental disease and actions (treatments) that a surgeon performs to maintain the beauty and health of the patient's teeth are also called treatments. Consists of a combination of treatments. Examples of procedures include examination, cutting, filing, root canal cleaning, drying, temporary sealing, impressions, scaling, prosthetic modifications, periodontal pocket measurements, suction, and tooth extraction; The content of treatment includes the content of treatment performed to cure a dental disease, and the content of treatment performed by an operator to maintain the beauty and health of the patient's teeth.

Let us take root canal treatment as an example of medical treatment. Root canal treatment is a treatment that is necessary when the pulp (nerves, blood vessels, etc.) in the root of the tooth becomes inflamed or infected due to the progression of caries. The treatment involves enlarging the area by cutting it, cleaning and disinfecting it, and placing a filling inside the root of the tooth to prevent reinfection. Root canal treatment consists of examination, pulp extraction, root canal length measurement/enlargement, cleaning/disinfection, root canal filling, and filling/covering, and the surgeon performs each of these procedures on the patient in order. Root canal treatment can be performed by performing this procedure.

The examination is a procedure that involves the surgeon interviewing the patient and inspecting the patient's oral cavity to identify the patient's dental condition and etiology and establishing a treatment plan. Pulp extraction is a procedure that involves removing the damaged pulp during root canal treatment. Root canal length measurement and enlargement is a procedure that involves measuring the position of the hollow root apex after removing the pulp, and cutting and enlarging the root canal up to the root apex. Cleaning and disinfection is a procedure that involves deep cleaning and disinfection of the hollow root canal. Root canal filling is a procedure that involves filling the root canal with a special drug to prevent bacteria from entering the root canal after cleaning and disinfection. Filling/covering is a procedure that involves filling the root canal with a rubber-like filling, creating a metal or fiber base on top of it, and then placing a crown on the base. Note that the content and number of treatments included in the root canal treatment described above are only an example, and the root canal treatment may include other treatments or some of the above-mentioned treatments may be omitted. .

<Image taken by tray camera>
FIG. 2 is a diagram for explaining an example of an image taken by the tray camera 51. As shown in FIG. 2, one or more medical instruments placed on the tray 30 are photographed by the tray camera 51.

For example, in the example shown in FIG. 2, a rubber dam moisture-proof set 301, a rubber dam sheet 302, a root canal length measuring device 303, a bur set 304, a file (reamer) 305, a mouth corner counter electrode 306, a fur clip 307, a broach 308, and a cleaning needle 309 , a cleaning syringe 310, a temporary sealant filler 311, a cleanser 312, a turbine 313, tweezers 314, a vacuum 315, a mirror 316, an excavator 317, a deep needle 318, and a root canal material injector 319. The photographed image shows how the medical instruments are placed on the tray 30.

During medical treatment, the operator selects and uses a desired medical instrument from among the plurality of medical instruments placed on the tray 30. Not displayed. For example, when the mirror 316 is used during medical treatment, since the mirror 316 does not exist from above the tray 30, the mirror 316 is not reflected in the image taken by the tray camera 51.

The control device 100 analyzes the tray image data including the photographed image acquired by the tray camera 51 as shown in FIG. etc. can be detected.

<Image taken by patient camera>
FIG. 3 is a diagram for explaining an example of an image taken by the patient camera 52. As shown in FIG. 3, the patient camera 52 photographs an area including at least the inside of the patient's oral cavity.

For example, in the photographed image shown in FIG. 3, the photographed image shows the position of medical instruments (in this example, tweezers 314 and mirror 316) on the teeth near the lower lip of the right cheek in the patient's oral cavity. .

During medical treatment, dental treatment using medical instruments will be performed inside the patient's oral cavity, but as shown in FIG. The medical equipment that will be used will be displayed.

The control device 100 can detect the position of the medical instrument in the patient's oral cavity by analyzing patient image data, including images captured by the patient camera 52 as shown in FIG. 3, using image recognition or the like. .

<Images taken by the overall camera>
FIG. 4 is a diagram for explaining an example of an image taken by the general camera 53. As shown in FIG. 4, the entire camera 53 photographs an area including at least the medical treatment space.

For example, in the photographed image shown in FIG. and medical instruments on the tray 30) are shown in the photographed image.

During medical treatment, the dentist performs dental treatment using medical instruments, but as shown in FIG. The patient's behavior, the patient's behavior, the state of the medical device 1, etc. are displayed.

The control device 100 analyzes the overall image data including the photographed image acquired by the overall camera 53 as shown in FIG. Y coordinate) can be detected. Further, as will be described later, the control device 100 can detect the positions (for example, Z coordinates) of the operator and patient during medical treatment by irradiating light such as infrared light from the general camera 53.

<Internal configuration of control system>
FIG. 5 is a block diagram showing the internal configuration of control system 1000 according to this embodiment. As shown in FIG. 5, the control system 1000 includes a plurality of cameras (a tray camera 51, a patient camera 52, and an overall camera 53) and a medical treatment device 1.

The medical treatment device 1 includes a chair 11, an instrument control device 21, a display control device 22, a sound control device 32, a basin unit 12, and a control device 100.

The chair 11 includes a headrest 11a, a backrest 11b, a seat 11c, and a footrest 11d, each of which is driven under the control of the chair controller 111. Specifically, the chair control unit 111 transmits control data based on the operator's operation received by the foot controller 16 or the operation panel 18, or control data from the control device 100 via CAN (Controller Area Network) communication. When received, based on the control data, the seat 11c is raised or lowered, and the headrest 11a, backrest 11b, and footrest 11d are moved vertically or horizontally with respect to the seat 11c. . When the headrest 11a, the backrest 11b, and the footrest 11d are positioned perpendicularly to the seat 11c, the patient sitting on the chair 11 assumes a sitting posture. When the headrest 11a, the backrest 11b, and the footrest 11d are positioned horizontally with respect to the seat 11c, the patient sitting on the chair 11 assumes a supine posture. In this way, the chair control unit 111 changes the posture of the chair 11 by driving the headrest 11a, the backrest 11b, the seat seat 11c, and the footrest 11d.

The instrument control device 21 includes an instrument control section 211. When the instrument control unit 211 receives control data based on the operator's operation accepted by the foot controller 16 or the operation panel 18, or control data from the control device 100 via CAN communication, the instrument control unit 211 performs the following based on the control data: Controls the drive or settings of the medical instrument 15. For example, when the operator presses a switch for driving the air turbine handpiece in the foot controller 16, the instrument control device 21 rotates the cutting tool held in the head of the air turbine handpiece. For example, when the control device 100 outputs control data for controlling the rotational direction, rotational speed, etc. of the air handpiece, the instrument control device 21 outputs control data for controlling the rotational direction, rotational speed, etc. of the air turbine handpiece based on the control data. Set.

Display control device 22 includes a display control section 221 and a panel control section 222. When the display control unit 221 receives control data from the control device 100 via CAN communication, the display control unit 221 controls the display 17 based on the control data. When the panel control unit 222 receives control data based on the control of the control device 100 via CAN communication, the panel control unit 222 controls the operation panel 18 based on the control data.

The sound control device 32 includes a sound control section 321. When the sound control unit 321 receives control data from the control device 100 via CAN communication, it controls the speaker 35 based on the control data.

Basin unit 12 includes a basin control section 121 and a lighting control section 122. Upon receiving control data from the control device 100 via CAN communication, the basin control unit 121 controls water supply and drainage in the basin unit 12 based on the control data. When the lighting control unit 122 receives control data from the control device 100 via CAN communication, it controls lighting and extinguishing of the lighting device 19 based on the control data.

Each of the chair control section 111, appliance control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 described above is a CPU mounted on a board (not shown). (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). Note that each of the chair control section 111, instrument control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 is installed in the medical device 1 in advance. Alternatively, it may be modularized so that it can be attached to the medical device 1 as an option.

Each of the control device 100, chair control section 111, appliance control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 communicates with each other by CAN communication. . In CAN communication, the control units communicate with each other by converting medical treatment-related data including log data from each control unit into communication packets. Note that in communication between the control units, as long as the medical care-related data is transmitted and received, the medical care-related data does not necessarily need to be converted into communication packets.

Furthermore, the basin unit 12 includes a storage section 123 that stores medical care-related data including log data within the medical device 1.

The storage unit 123 communicates with each of the chair control unit 111, appliance control unit 211, display control unit 221, panel control unit 222, sound control unit 321, basin control unit 121, and lighting control unit 122 by CAN communication. By doing so, medical care-related data is collected and accumulated from each control unit. The storage unit 123 may be configured with a memory such as a ROM or RAM mounted on a substrate (not shown), or may be configured with a nonvolatile recording medium such as a memory card.

The control device 100 includes a communication device 101, a calculation device 102, and a recording device 103.

The communication device 101 transmits and receives data by communicating with each of the tray camera 51, patient camera 52, and general camera 53. The communication device 101 acquires image-related data from each camera by communicating with each camera through wired or wireless LAN (Local Area Network) communication. Note that the communication device 101 may acquire image-related data from each camera in other formats. For example, the communication device 101 may acquire image-related data that is temporarily recorded on a nonvolatile recording medium such as a memory card taken out from each camera. The communication device 101 acquires medical care-related data accumulated in the storage section 123 of the basin unit 12 by communicating with the basin unit 12 through wired or wireless LAN communication. Note that the basin unit 12 is not limited to collecting and accumulating medical care-related data, and other components such as the chair 11, instrument control device 21, display control device 22, and sound control device 32 may accumulate medical care-related data. Alternatively, the control device 100 may acquire medical care-related data from these other configurations.

The computing device 102 executes various programs (for example, a control program 141 described later) to perform various processes such as control data generation processing for controlling the medical device 1 and learning processing for estimating control data. It is the main body of calculation that executes processing. Computing device 102 is one embodiment of a "computer." The arithmetic device 102 includes, for example, a CPU, an FPGA (Field-Programmable Gate Array), and a GPU (Graphics Processing Unit).

Note that the arithmetic device 102 may be composed of at least one of a CPU, an FPGA, and a GPU, or may be composed of a CPU and an FPGA, an FPGA and a GPU, a CPU and a GPU, or a CPU, an FPGA, and a GPU. may be done. Further, the arithmetic device 102 may be configured with a processing circuitry.

The recording device 103 is an embodiment of a "recording section". The recording device 103 provides a recording area for temporarily storing program codes, work memory, etc. when the arithmetic device 102 executes an arbitrary program. The recording device 103 may be configured with a volatile memory device such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory), or a non-volatile memory device such as a hard disk or SSD (Solid State Drive). or The recording device 103 may be configured with a database engine.

The recording device 103 is not limited to that included in the control device 100. For example, an in-hospital server communicably connected to the medical device 1 may include the recording device 103, or an external server installed outside the medical treatment space where the medical device 1 is installed may include the recording device 103. Good too. Furthermore, the recording device 103 may exist in a cloud computing mode in which each of the plurality of control devices 100 included in each of the plurality of medical treatment devices 1 can communicate with each other. In this way, image-related data and medical treatment-related data acquired from a plurality of medical treatment devices 1 can be uniformly stored and managed by the recording device 103.

The recording device 103 stores an estimated model 161, a control program 141, and an OS (Operating System) 142.

The estimation model 161 is used to estimate control data of the medical device 1 based on at least one of image-related data and medical care-related data acquired from the medical device 1 . The estimation model 161 is optimized (adjusted) by performing machine learning based on at least one of the image-related data and the medical care-related data, and can improve the estimation accuracy of the control data of the medical device 1.

Note that the process of generating control data for the medical device 1 by the arithmetic device 102 of the control device 100 is also referred to as a "control data generation process," and the process of learning the estimation model 161 by the arithmetic device 102 of the control device 100 is referred to as a "learning process." Also called. Furthermore, the estimated model 161 that has been optimized through the learning process is also particularly referred to as a "trained model." That is, in this embodiment, while the pre-learning estimation model 161 and the trained estimation model 161 are collectively referred to as the "estimation model", the trained estimation model 161 is also particularly referred to as the "trained model". to be called.

The control program 141 includes a program for the arithmetic device 102 to execute a control data generation process and a learning process, and a program for outputting the generated control data to each control target.

<Position data>
With reference to FIGS. 6 to 8, position data of the operator and the patient acquired by the control device 100 based on the imaging information obtained by the general camera 53 will be described. FIG. 6 is a diagram for explaining an example of position data acquired by control device 100 according to the present embodiment. FIG. 7 is a diagram for explaining an example of detection points of position data. FIG. 8 is a diagram for explaining an example of position coordinates corresponding to position data.

In the control device 100, the arithmetic device 102 generates and generates position data for specifying the positions of persons included in the photographed image, such as the surgeon and patient undergoing medical treatment, based on the photographic information acquired by the general camera 53. The recorded position data is recorded on the recording device 103.

Specifically, as shown in FIG. 6, the recording device 103 collectively records position data for each person, such as person A, person B, and person C. For example, person A corresponds to a dentist, person B corresponds to a dental assistant, and person C corresponds to a patient. The recording device 103 records the position data generated by the arithmetic device 102 in association with time information (time stamp). The recording device 103 records the position data newly generated by the calculation device 102 when the position data newly generated by the calculation device 102 is different from the recorded position data. That is, in this embodiment, the recording device 103 does not record new position data unless the position of the person changes, but records new position data on the condition that the position of the person changes. Note that the recording device 103 may record the position data generated by the arithmetic device 102 in association with a time stamp at predetermined time intervals (for example, 1 second) regardless of whether the position data changes or not. .

The position data includes three-dimensional (X, Y, Z) position coordinates for each detection point determined for the person to be detected. As shown in FIGS. 6 and 7, for example, the detection points include the right wrist, right elbow, right shoulder, head, left shoulder, left elbow, left wrist, right ear, and left ear of the person (dentist 3 in this example). , including right eye, left eye, nose, mouth, hip, right knee, left knee, right ankle, and left ankle. Note that the control device 100 may acquire position data for points other than the detection points shown in FIG. 6, or may acquire position data only for some of the detection points shown in FIG. .

Note that when the control device 100 cannot specify some of the predetermined detection points, the control device 100 may cause the recording device 103 to record the position data corresponding to the previously specified detection points as the current position data. For example, if a person to be photographed enters a blind spot of the general camera 53, the control device 100 may not be able to identify some detection points. The position data corresponding to some of the detection points that are missing this time may be complemented using the position data corresponding to the detection points specified before entering the blind spot of 53.

Position data in the case where the overall camera 53 is composed of a ToF type camera will be explained. For example, the positional coordinates corresponding to the positional data set the origin of the X and Y coordinates to the upper left corner of the photographed image obtained by the general camera 53 as shown in FIG. Furthermore, as shown in FIG. 8, the position coordinates corresponding to the position data have the origin of the Z coordinate as the position where the general camera 53 is placed.

When the arithmetic device 102 acquires a captured image as shown in FIG. 7 from the general camera 53, it identifies three people, person A, person B, and person C, by image recognition. The computing device 102 identifies detection points such as the right wrist and right elbow for each person by image recognition, and identifies the position coordinates of each identified detection point. Each person present in the medical treatment space, such as a dentist, a dental assistant, and a patient, has a roughly determined location (home position). Therefore, the computing device 102 can identify whether each person is a dentist, a dental assistant, or a patient based on the position data of each of the plurality of people shown in the captured image of the general camera 53. good. Further, each person present in the medical treatment space, such as a dentist, a dental assistant, and a patient, generally wears a certain color or shape. For example, a dentist usually wears a white coat. For this reason, the computing device 102 may identify whether each person is a dentist, a dental assistant, or a patient based on the clothes of each of the plurality of people shown in the captured image of the general camera 53. .

For example, focusing on person A (dentist 3), the computing device 102 calculates P1 (right wrist), P2 (right elbow), P3 (right shoulder), P4 (head), P5 (left shoulder), P6 (left elbow). ), and P7 (left wrist) are identified, and the positional information of each of the detection points P1 to P7 is identified. For example, as shown in FIG. 8, the calculation device 102 specifies X1 as the X coordinate and Y1 as the Y coordinate, based on the captured image, regarding the position coordinates of P1. Further, the calculation device 102 identifies Z1 as the Z coordinate based on the reflected light of the light emitted from the origin of the Z coordinate toward P1 by the general camera 53. Specifically, the calculation device 102 calculates the distance L between the origin of the Z coordinate and P1 based on the time from when light is irradiated by the general camera 53 until the reflected light of that light returns. . Then, the calculation device 102 draws a perpendicular line from P1 to the Z axis, and uses trigonometric functions based on the length of the perpendicular line and the distance L to specify Z1 as the Z coordinate.

The arithmetic device 102 causes the recording device 103 to record the position coordinates of P1 identified as described above as position data of P1. At this time, if the newly acquired position data of P1 is different from the position data of P1 already recorded in the recording device 103, the calculation device 102 causes the recording device 103 to record the newly acquired position data of P1 together with a time stamp. . On the other hand, if the newly acquired position data of P1 is the same as the position data of P1 already recorded in the recording device 103, the arithmetic device 102 does not cause the recording device 103 to record the newly acquired position data of P1.

In this way, the computing device 102 does not cause the recording device 103 to record new position data each time the person's position is detected, but records new position data only when the person's position data changes. Since the data is recorded by the recording device 103, an increase in the amount of data recorded by the recording device 103 can be suppressed as much as possible. Note that, as described above, the arithmetic device 102 may cause the recording device 103 to record the position data at predetermined time intervals (for example, one second) regardless of whether the position data changes or not. In this way, the time at which position data is recorded will not be random, so there is no need to obtain a time stamp each time position data is recorded.

Furthermore, by having the recording device 103 record the position data together with a time stamp, the arithmetic device 102 can use the position data used when performing new machine learning again when relearning. Furthermore, when multiple control devices 100 share the position data recorded in the recording device 103, a time stamp is associated with the position data, so that the multiple control devices 100 perform similar machine learning. be able to.

In this way, the control device 100 acquires position data of each detection point for specifying the positions of the surgeon and patient during treatment, based on the imaging information acquired from the general camera 53. The control device 100 associates the acquired position data with a time stamp, and causes the recording device 103 to store and record the data as whole image data. Further, the control device 100 can identify the motions of a person's hands, arms, head, waist, legs, etc. by virtually connecting each detection point with a straight line or the like.

Note that the method for specifying the position coordinates of the detection point described above is an example, and the control device 100 may specify the position coordinates of the detection point using other methods. Further, the control device 100 may specify the position coordinates of not only a person involved in a series of medical treatments but also a key item involved in a series of medical treatments. In this way, the control device 100 can perform machine learning on the estimation model 161 using the position coordinates of the identified key item.

In the present embodiment, the calculation device 102 calculates the Z coordinate based on the photographing information acquired from the general camera 53, and then records the Z coordinate together with the X coordinate and the Y coordinate in the recording device 103 or the recording device 203. Met. However, instead of the Z coordinate, the calculation device 102 may cause the recording device 103 or 203 to record a depth frame that frames the distances to all objects shown in the captured image together with the image frame. In this way, even after a series of medical treatments have been completed, the Z coordinate can be calculated by analyzing the depth frame and image frame recorded on the recording device 103 or the recording device 203.

<Functional configuration of control device>
[Functional configuration of control device at learning stage]
FIG. 9 is a block diagram showing the functional configuration of control device 100 in the learning stage according to the present embodiment. As shown in FIG. 9, the control device 100 includes an input section 150, a position detection section 157, an object detection section 152, a conversion section 154, a synchronization section 155, and a segmentation section 156 as main functional sections. , a generation section 160, and an output section 180. Note that the input unit 150 and the output unit 180 are functional units of the communication device 101, and the position detection unit 157, object detection unit 152, conversion unit 154, synchronization unit 155, segmentation unit 156, and generation unit 160 are functional units, respectively. , which are functional units of the arithmetic device 102.

The input unit 150 includes image-related data (tray image data, patient image data, and overall image data) captured by each of the tray camera 51, patient camera 52, and overall camera 53, and medical treatment-related data acquired by the medical treatment device 1. are input in chronological order. Note that depth frames, which are frames of the distance to the object shown in the captured image, may be input to the input unit 150 in chronological order.

The position detection section 157 includes an XY detection section 151 and a Z detection section 153. The XY detection unit 151 performs image recognition based on the overall image data of the overall camera 53 input from the input unit 150 to detect positional data (X coordinates, Y coordinates) of persons such as the operator and the patient. As explained in FIG. 8, the Z detection unit 153 detects the person based on the X and Y coordinates of the person identified by the Detect position data (Z coordinate) of For example, based on the overall image data of the overall camera 53, the position detection unit 157 detects the position of the person at predetermined timings (tC1, tC2, tC3, ... shown in FIG. 14, which will be described later), as shown in FIG. 14, which will be described later. Detect position data (X coordinate, Y coordinate, Z coordinate).

Here, the configuration of the XY detection section 151 will be explained in more detail. As shown by the broken line C in the figure, the XY detection unit 151 includes an estimated model 1511. Furthermore, the estimation model 1511 includes a neural network 1512 and parameters 1513 used by the neural network 1512. The parameters 1513 include weighting coefficients used in calculations by the neural network 1512 and determination values used in estimation determination. Note that the estimation model 1511 corresponds to an embodiment of a "second estimation model," and the neural network 1512 corresponds to an embodiment of a "second neural network."

The neural network 1512 is an image recognition method using deep learning, such as a convolution neural network (CNN), a recurrent neural network (RNN), or a LSTM network (Long Short Term Memory Network). A well-known neural network used in processing is applied.

The XY detection unit 151 uses positional data of persons such as the operator and the patient (X coordinates, Y coordinates).

In the learning stage, the estimation model 1511 uses the person's position data (X coordinate, Y coordinate of each detection point) associated with the whole image data and the person's position data (X coordinate of each detection point) using the whole image data. It is optimized (adjusted) by learning based on the estimation results of the coordinates and Y coordinates.

Specifically, when the estimation model 1511 receives whole image data as training data, the neural network 1512 estimates the detection points of the person shown in the photographed image based on the whole image data. Note that the detection points estimated by the estimation model 1511 are determined in advance as shown in FIG. When the estimation model 1511 estimates a detection point for each person, it estimates position data (X coordinate, Y coordinate) for each estimated detection point. Note that the origin of the position data is determined in advance as shown in FIGS. 7 and 8.

The estimation model 1511 determines whether or not its own estimation result matches the person's position data, which is the correct data associated with the input whole image data, and does not update the parameter 1513 if both match. On the other hand, if the two do not match, the parameter 1513 is updated so that the two match, thereby optimizing the parameter 1513. Note that the above-described learning of the estimation model 1511 may be performed not only in the learning stage but also in the operational stage.

Note that the XY detection unit 151 is not limited to one that detects the position data of a person through AI image recognition using a neural network. For example, the XY detection unit 151 may detect the position data of the person by comparing the photographed image with a template prepared in advance, such as known pattern matching. Furthermore, the function of the XY detection unit 151 may not be possessed by the control device 100, but may be possessed by the general camera 53, or by an edge computer disposed near the general camera 53. good. In this case, detection results by an edge computer or the like are input to the input unit 150 as image-related data.

The object detection unit 152 can detect the presence, shape, type, etc. of medical instruments placed on the tray 30 by performing image recognition based on the tray image data of the tray camera 51 input from the input unit 150. can. For example, the object detection unit 152 detects a predetermined timing (tA1 shown in FIG. 12, which will be described later) as shown in FIG. , tA2, tA3, ...), the presence or absence, shape, type, etc. of the medical instruments placed on the tray 30 are estimated. For example, in the tray image data shown in FIG. 12, data "0" is stored in the recording area corresponding to the medical instruments present on the tray 30, and data "0" is stored in the recording area corresponding to the medical instruments not present on the tray 30. 1" data is stored.

Further, the object detection unit 152 can detect the position of the medical instrument in the patient's oral cavity by performing image recognition based on patient image data from the patient camera 52 input from the input unit 150. For example, the object detection unit 152 detects the data at predetermined timings (tB1, tB2, tB3, . . . shown in FIG. 13, which will be described later), as shown in FIG. In addition, the position of medical instruments within the patient's oral cavity is estimated. For example, in the patient image data shown in FIG. 13, in the positional relationship between the medical instrument and the patient, if both are not located within a predetermined range, data of "0" is stored in the recording area; If it is located, data "1" is stored in the recording area.

Although not shown, the object detection section 152, like the XY detection section 151, has an estimation model including a neural network and parameters. Then, the object detection unit 152 estimates the presence or absence, shape, type, etc. of a medical instrument, and estimates the position of the medical instrument in the patient's oral cavity, through AI image recognition using a neural network.

For example, the object detection unit 152 has an estimation model including a known neural network used in image recognition processing using deep learning, such as a CNN, RNN, and LSTM network. In the learning stage, the estimation model is based on the existence, shape, and type of medical instruments associated with the tray image data, and the estimation results of the presence, shape, and type of medical instruments using the tray image data. Optimized (adjusted) by learning. Furthermore, in the learning stage, the estimation model uses the position of the medical instrument in the patient's oral cavity associated with the patient image data, and the estimation result of the position of the medical instrument in the patient's oral cavity using the patient image data. It is optimized (adjusted) by learning based on the following. Note that the above-described learning of the estimation model is not limited to the learning stage, but may also be performed during the operation stage.

Note that the object detection unit 152 is limited to detecting the presence, shape, type, etc. of medical instruments through AI image recognition using a neural network, and detecting the position of medical instruments in the patient's oral cavity. do not have. For example, the object detection unit 152 can detect the presence, shape, and type of medical instruments by comparing a photographed image with a template prepared in advance, as in the case of well-known pattern matching. The position of a medical instrument may also be detected. Further, the function of the object detection unit 152 is not one that the control device 100 has, but may be one that the patient camera 52 or the tray camera 51 has, or an edge detector disposed near the patient camera 52 or the tray camera 51. It may be something that a computer has. In this case, detection results by an edge computer or the like are input to the input unit 150 as image-related data.

The conversion unit 154 converts the medical care-related data input from the basin unit 12 to the input unit 150 into a predetermined format. Specifically, the conversion unit 154 converts the medical care-related data input into the input unit 150 into a format suitable for synchronizing it with the image-related data also input into the input unit 150. For example, the conversion unit 154 converts the format of the medical care-related data so that the time axis of the medical care-related data matches the time axis of the image-related data. Alternatively, the conversion unit 154 converts the format of the medical care-related data so that the time axis of the medical care-related data is aligned with a common time axis to match the time axis of the image-related data.

The synchronization unit 155 chronologically synchronizes the image-related data including the detection results by the position detection unit 157 and the object detection unit 152 and the medical care-related data converted by the conversion unit 154. For example, as shown in FIGS. 16 and 17, which will be described later, the synchronization unit 155 synchronizes image-related data (tray image data, patient image data, whole image data) and medical treatment-related data in accordance with the time axis of the synchronization time. Synchronization is performed at predetermined timings (T1, T2, T3, . . . shown in FIGS. 16 and 17, which will be described later). Data obtained by synchronizing image-related data and medical care-related data is also referred to as "synchronized data." Note that as the synchronization time used when generating the synchronization data, an elapsed time from the start of medical treatment or an absolute time that can be specified from a timestamp (for example, real time time) may be used.

The segmentation unit 156 segments the synchronous data by separating the image-related data and the medical care-related data input from the input unit 150 at a predetermined timing. Specifically, the segmentation unit 156 records that a predetermined event has occurred at a predetermined timing with respect to the image-related data and medical care-related data (i.e., synchronized data) synchronized by the synchronization unit 155. The synchronous data is segmented as follows. "The occurrence of a predetermined event" includes, for example, starting or stopping the driving of medical instruments 15 such as handpieces, starting or stopping lighting of the lighting device 19, starting or ending water supply/drainage in the basin unit 12, or Events related to medical treatment are included, such as when a medical instrument on the tray 30 is taken out or when a medical instrument is placed on the tray 30.

Furthermore, the segmentation unit 156 records the continuation of a predetermined event at a predetermined timing with respect to the image-related data and medical care-related data (synchronized data) synchronized by the synchronization unit 155. Segment the synchronous data. “Continuation of a predetermined event” includes, for example, the continuation of the driving state of the medical instrument 15 such as a handpiece, the continuation of the lighting state of the lighting device 19, the continuation of water supply/drainage in the basin unit 12, and the continuation of the operation of the medical instrument 15 such as a handpiece. This includes the continuation of events related to medical treatment, such as the continuation of the state in which medical instruments exist or the continuation of the state in which they do not exist.

For example, the segmentation unit 156 segments the synchronous data for each period by dividing the image-related data and the medical care-related data at predetermined timings. As for the segmentation method, flag data may be set at each predetermined timing, or synchronized data may be moved or copied to a predetermined recording area for each segmented period.

The "predetermined timing" at which the synchronized data is segmented by the segmentation unit 156 may be a timing that occurs periodically at predetermined time intervals (for example, every 5 seconds).

Note that the "predetermined timing" at which the synchronous data is divided by the segmentation unit 156 is the "0" or "1" data stored in the image-related data (tray image data, patient image data) and the data related to the medical treatment. It may be determined based on stored data of "0" or "1".

For example, the "predetermined timing" at which the synchronous data is divided by the segmentation unit 156 is data that drives the chair 11 of the medical device 1 and data that drives the medical instrument 15 of the medical device 1, which are included in the medical care-related data. Based on at least one of data for driving the lighting device 19 included in the medical treatment device 1 and data for driving the basin unit 12 included in the medical treatment device 1, the occurrence or continuation of driving of these medical treatment devices is determined. It may be timing. For example, as shown in FIG. 17, at timing T3 when the pick-up of the medical instrument 15 switches from "0" to "1", timing T6 when the pick-up of the medical instrument 15 switches from "1" to "0", etc., the synchronous data is may be separated.

Note that even if the segmentation unit 156 cannot segment the synchronous data based on medical care-related data, it can also segment the synchronous data based on image-related data such as tray image data. For example, as shown in FIG. 16, at timing T8 when the determination of the presence or absence of the cleaning needle 309 and the cleaning syringe 310 is switched from "0" to "1", the determination of the presence or absence of the cleaning needle 309 and the cleaning syringe 310 is "1". The synchronized data may be separated at timing T10 when the value changes from "0" to "0". That is, the segmentation unit 156 divides the image-related data and the medical care-related data into a predetermined pattern based on data indicating the occurrence of an event related to patient medical care that is included in at least one of the image-related data and the medical care-related data. You can also segment based on timing.

In this way, when segmenting the synchronous data, the segmentation unit 156 can segment the control data by combining the image-related data and the medical care-related data, even if the data is insufficient for each of them alone. The synchronized data may be separated at appropriate timings for estimating.

Note that if the image capturing timing in the image-related data input from the input unit 150 and the log data acquisition timing in the medical care-related data are synchronized in advance, the converting unit 154 and the synchronizing unit 155 may not be used. . In this case, the segmentation unit 156 directly uses the image-related data and the medical treatment-related data input from the input unit 150 instead of the synchronous data, and divides the image-related data and the medical treatment-related data at a predetermined timing. You can segment it by separating it.

The generation unit 160 estimates control data of the medical device 1 based on the image-related data and medical care-related data segmented by the segmentation unit 156, and generates the estimated control data.

The generation unit 160 has an estimation model 161. Furthermore, the estimation model 161 includes a neural network 162 and parameters 163 used by the neural network 162. The parameters 163 include weighting coefficients used in calculations by the neural network 162 and determination values used in estimation determination. Note that the estimation model 161 corresponds to an embodiment of a "first estimation model", and the neural network 162 corresponds to an embodiment of a "first neural network".

In the learning stage, the estimation model 161 is trained based on the image-related data and medical care-related data segmented by the segmentation unit 156, and the estimation results of control data using the image-related data and the medical care-related data. It is optimized (adjusted) by

For example, when whole image data (position data of a person) is input as training data, the estimation model 161 uses a neural network 162 to grasp the order and posture of the person's actions based on the whole image data. 1 and estimates the control data of the medical device 1.

The estimation model 161 determines whether or not its own estimation result matches the control data of the medical device 1, which is the correct data associated with the input overall image data, and if they match, it changes the parameter 163. The parameter 163 is optimized by not updating the parameter 163, but by updating the parameter 163 so that the two match if they do not match.

In this way, in the learning stage, the control device 100 selects among the image-related data such as tray image data, patient image data, and whole image data, and the medical care-related data acquired by the medical device 1 such as log data. The estimation model 161 including the neural network 162 is subjected to machine learning so that the control data can be estimated based on at least one of the data.

Note that the control device 100 may perform machine learning on the estimation model 161 including the neural network 162 not only in the learning stage but also in the operational stage. For example, in the operation stage, when the medical device 1 operates based on the control data generated by the generation unit 160, a user such as a surgeon inputs correct data to cause the medical device 1 to perform a different operation. , the estimation model 161 may be optimized (readjusted) by learning based on the estimation result and the correct data. In this way, since the control device 100 performs machine learning even during the operation stage, the more a user such as a surgeon uses it, the more the control data estimation accuracy improves.

Note that the estimation model 161 of the generation unit 160 is not limited to a supervised learning algorithm, and may perform machine learning using a known algorithm such as an unsupervised learning algorithm or a reinforcement learning algorithm. For example, when using an unsupervised learning algorithm, the estimation model 161 is based on image-related data, the types of medical instruments used during treatment, the timing of their use, the positional relationship between the medical instruments and the patient's oral cavity, and the The operator's and patient's actions and the log data of each part of the medical treatment device 1 are classified (clustered), and based on the classification results, the medical procedure (for example, the content of the next treatment to be performed during the medical treatment) is estimated. Good too.

The output unit 180 outputs the control data generated by the generation unit 160 to the control target 190 in the medical device 1 . The controlled object 190 includes at least one of the chair 11 , the medical instrument 15 , the lighting device 19 , the basin unit 12 , the foot controller 16 , the display 17 , the operation panel 18 , and the speaker 35 .

[Functional configuration of control device during operation stage]
FIG. 10 is a block diagram showing the functional configuration of control device 100 according to the present embodiment at an operation stage. As shown in FIG. 10, in the control device 100 in the operation stage, each of the XY detection section 151, the object detection section 152, and the generation section 160 has been trained.

In the operation stage, image-related data is input from each camera to the input section 150 in real time while the surgeon is treating a patient, and further, medical treatment-related data is input from the basin unit 12 to the input section 150.

The XY detection unit 151 performs image recognition based on the overall image data of the overall camera 53 input from the input unit 150 to detect positional data (X coordinates, Y coordinates) of persons such as the operator and the patient. The Z detection unit 153 calculates the position data (Z coordinate) of the person based on the X and Y coordinates of the person identified by the XY detection unit 151 and the distance L to the person calculated from the reflected light from infrared rays or the like. To detect.

The object detection unit 152 performs image recognition based on the tray image data of the tray camera 51 input from the input unit 150 to detect the presence or absence, shape, type, etc. of medical instruments placed on the tray 30. Further, the object detection unit 152 detects the position of the medical instrument in the patient's oral cavity by performing image recognition based on patient image data from the patient camera 52 input from the input unit 150. The object detection unit 152 may detect the position of a key item related to medical treatment, such as a medical instrument, by performing image recognition based on the overall image data of the overall camera 53 input from the input unit 150.

The conversion unit 154 converts the medical care-related data input from the basin unit 12 to the input unit 150 into a predetermined format.

The synchronization unit 155 generates synchronized data by chronologically synchronizing the image-related data including the detection results by the position detection unit 157 and the object detection unit 152 and the medical care-related data converted by the conversion unit 154.

The segmentation unit 156 segments the synchronous data by separating the image-related data and the medical care-related data input from the input unit 150 at a predetermined timing.

The generation unit 160 estimates the control data of the medical device 1 based on the image-related data and medical treatment-related data segmented by the segmentation unit 156 and the learned estimation model 161 including the neural network 162. Generate control data.

The output unit 180 outputs the control data generated by the generation unit 160 to the control target 190 in the medical device 1 .

In this way, the control device 100 uses image-related data such as tray image data, patient image data, and whole image data, medical treatment-related data such as log data acquired by the medical treatment device 1, and estimation including the neural network 162. Control data for controlling the medical treatment device 1 is generated based on the model 161. As a result, the medical device 1 is controlled by the control device 100 prior to the actions of an operator such as a dentist or a dental assistant, so convenience for the operator can be improved.

Note that the control device 100 is not limited to generating control data for controlling the medical device 1 based on both the image-related data and the medical treatment-related data and the estimation model 161. The control device 100 may generate control data for controlling the medical treatment device 1 based on the image-related data and the estimation model 161 without using the medical treatment-related data.

For example, the control device 100 detects position data of people such as a surgeon and a patient from the overall image data acquired by the overall camera 53, and uses the estimation model 161 to determine the actions each person performs based on the position data of each person. By understanding the order and posture of the patient, the control details of the medical device 1 may be read in advance, and the control data of the medical device 1 may be estimated.

For example, the control device 100 detects the position of the medical instrument in the patient's oral cavity from patient image data acquired by the patient camera 52, and uses the estimation model 161 to detect the position of the medical instrument in the patient's oral cavity. By understanding the order and posture of the operations performed by the patient, the control details of the medical device 1 may be read in advance and the control data of the medical device 1 may be estimated.

For example, the control device 100 detects the presence, shape, and type of medical instruments placed on the tray 30 from the tray image data acquired by the tray camera 51, and uses the estimation model 161 to detect the presence, shape, and type of medical instruments placed on the tray 30. By understanding the order and posture of the operator's actions based on the presence, shape, and type of instruments, the control details of the medical device 1 may be read in advance and the control data of the medical device 1 may be estimated.

<Example of estimation of control data>
An example of estimation of control data by the control device 100 will be described with reference to FIGS. 11 to 17. Note that FIGS. 11 to 17 show examples of root canal treatment.

During medical treatment, one or more medical instruments placed on the tray 30 are photographed by the tray camera 51 at predetermined timings (tA1, tA2, tA3, . . . ) depending on the shutter timing.

For example, FIG. 11 is a diagram for explaining an example of an image taken by the tray camera 51 during root canal treatment. As shown in FIG. 11, at timing tA2 when an examination is being performed as a treatment included in root canal treatment, the operator uses tweezers 314, mirror 316, and deep needle 318, so the image captured by tray camera 51 is The tweezers 314, mirror 316, and deep needle 318 are not shown in the image. At timing tA3 when pulp extraction is being performed as a treatment included in root canal treatment, the file 305 and vacuum 315 are used by the operator, so the file 305 and vacuum 315 are not displayed in the captured image of the tray camera 51. At timing tA5 when root canal length measurement and enlargement are being performed as a procedure included in root canal treatment, the root canal length measurement device 303 and file 305 are used by the operator, so the root canal is not included in the image taken by the tray camera 51. Pipe length measuring device 303 and file 305 are not displayed. At timing tA7 when disinfection and cleaning are performed as a procedure included in root canal treatment, the operator uses the cleaning needle 309, the cleaning syringe 310, and the vacuum 315, so the image taken by the tray camera 51 does not include the following: Cleaning needle 309, cleaning syringe 310, and vacuum 315 are not shown. At timing tA9 when root canal filling is being performed as a procedure included in root canal treatment, the mirror 316 and the root canal material injector 319 are used by the operator, so the image taken by the tray camera 51 does not include the mirror 316 and the root canal. Tubing syringe 319 is not shown. At timing tA11 when stuffing and capping are performed as a treatment included in root canal treatment, the temporary sealant filler 311, tweezers 314, and mirror 316 are used by the operator, so the image taken by the tray camera 51 does not include the following: Temporary sealant filler 311, tweezers 314, and mirror 316 are not displayed.

In this way, during medical treatment, the presence, absence, shape, type, etc. of the medical instruments placed on the tray 30 are displayed using images taken by the tray camera 51.

When data of an image captured by the tray camera 51 (image-related data as tray image data) as described above is input from the input unit 150, the control device 100 causes the object detection unit 152 to detect the feature amount included in the image. Extract.

For example, FIG. 12 is a diagram for explaining an example of tray image data acquired from photographic information of the tray camera 51. As shown in FIG. 12, the control device 100 estimates the presence/absence, shape, type, etc. of medical instruments placed on the tray 30 at predetermined timings (tA1, tA2, tA3,...), and Data "0" is stored in the recording area corresponding to the medical instruments that are present on the tray 30, and data "1" is stored in the recording area corresponding to the medical instruments that are not present on the tray 30. Thereby, as shown in FIG. 12, the control device 100 can obtain tray image data (image-related data) that can distinguish the presence or absence of medical instruments at each predetermined timing.

During medical treatment, the patient's oral cavity is photographed by the patient camera 52 at predetermined timings (tB1, tB2, tB3, . . . ) depending on the shutter timing.

Then, when data of an image captured by the patient camera 52 (image-related data as patient image data) is input from the input unit 150, the control device 100 causes the object detection unit 152 to extract feature amounts included in the image. .

For example, FIG. 13 is a diagram for explaining an example of patient image data acquired from imaging information of the patient camera 52. As shown in FIG. 13, the control device 100 estimates the position of the medical instrument in the patient's oral cavity at each predetermined timing (tB1, tB2, tB3,...), and estimates the position of the medical instrument and the patient's lips, or Regarding the positional relationship between the instrument and the patient's cheek, if both are not located within a predetermined range, "0" is stored in the recording area, and if both are located within the predetermined range, "1" is stored in the recording area. Store data. Thereby, as shown in FIG. 13, the control device 100 can obtain patient image data (image-related data) that can specify the position of the medical instrument in the patient's oral cavity at each predetermined timing.

During medical treatment, the overall camera 53 photographs the medical treatment space, the actions of the surgeon and the patient, the operation of the medical treatment apparatus 1, etc. at predetermined timings depending on the shutter timing.

Then, when data of an image taken by the overall camera 53 (image-related data as overall image data) is input from the input unit 150, the control device 100 causes the position detection unit 157 to extract feature amounts included in the image. By doing so, position data of people such as the surgeon and the patient is generated. If the position data newly generated by the position detection unit 157 is different from the position data already recorded in the recording device 103, the control device 100 causes the recording device 103 to record the position data newly generated by the position detection unit 157. .

For example, FIG. 14 is a diagram for explaining an example of position data acquired from photographic information of the general camera 53. As shown in FIG. 14, the control device 100 associates the generated position data of the person with time information (time stamp) and records it as whole image data (image-related data). In the example shown in FIG. 14, position data of person A (dentist) is recorded in the recording device 103 at predetermined timings (tC1, tC2, tC3, . . . ). Thereby, the control device 100 can obtain whole image data (image-related data) including positional data that can specify the operator's actions and patient's actions at each predetermined timing. In particular, if the control device 100 knows the positional relationship between the person and the medical device 1 in the medical treatment space based on the positional data recorded in the recording device 103, the control device 100 can avoid unnecessary contact between the person and the medical device 1, for example. Even in such a case, driving of the medical device 1 can be stopped based on the control data.

During medical treatment, the chair 11, basin unit 12, lighting device 19, instrument control device 21, and display included in the medical treatment device 1 are activated at predetermined timings (tD1, tD2, tD3, ...) that depend on the timing of acquiring log data. Log data indicating the operation and control of at least one of the control devices 22 is output from the medical treatment device 1.

For example, FIG. 15 is a diagram for explaining an example of medical care-related data acquired from the medical device 1. As shown in FIG. 15, the medical care-related data includes the presence or absence of a pickup, the rotational speed of the air turbine, the rotational speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the operation of the vacuum syringe, and the operation of the Saraiba ejector regarding the medical instruments 15. , and log data representing the operating state of water injection, etc. Contains log data representing the seat height, back plate angle, and operating state of the chair 11 for the chair 11, and for the foot controller 16, includes log data representing the operating state of the chair 11, etc. , a depression pattern, a switch state, etc. are included, the lighting device 19 includes log data showing the lighting state and the light emission intensity, etc., and the basin unit 12 includes log data showing whether or not a cup is placed, etc. Contains data.

In the example of FIG. 15, data indicating the degree of the rotation speed of the air turbine, the rotation speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the amount of depression of the foot controller 16, and the emission intensity of the illumination device 19, such as large or small or high or low, is shown. is included in medical care-related data (log data). The medical care-related data includes the rotational speed of the air turbine, the rotational speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the amount of depression of the foot controller 16, and the value (for example, absolute value) of the light emission intensity of the lighting device 19. It's okay to stay. The depression pattern of the foot controller 16 may not be included in the medical care-related data. For example, the control device 100 may identify the depression pattern of the foot controller 16 by analyzing changes in the amount of depression of the foot controller 16 over time.

Note that the log data is not limited to that output from the medical device 1 at predetermined timings (tD1, tD2, tD3, ...) that depend on the acquisition timing of the log data, but is also based on the occurrence of an event (for example, the movement of the chair 11). It may also be outputted from the medical device 1 every time the medical device 15 changes its operation, changes the operation of the medical instrument 15, etc.).

When medical care-related data is input from the input unit 150, the control device 100 converts the medical care-related data into a format suitable for synchronization with the image-related data using the conversion unit 154.

Next, the control device 100 causes the synchronization unit 155 to synchronize the image-related data shown in FIGS. 12 to 14 and the medical care-related data shown in FIG. 15 at predetermined timings (T1, T2, T3,...).

For example, FIGS. 16 and 17 are diagrams for explaining examples of synchronous data. As shown in FIGS. 16 and 17, the synchronized data includes image-related data and medical care-related data that are synchronized at predetermined timings (T1, T2, T3, ...) according to the synchronization time axis. Contains data.

For example, data at tA3 and tA4 in the tray image data shown in FIG. 12 is associated with data at timings (T3 to T5) corresponding to tA3 and tA4 in the synchronization data shown in FIG. 16. The data of tB3 and tB4 in the patient image data shown in FIG. 13 are associated with the data of the timing (T3 to T5) corresponding to tB3 and tB4 in the synchronization data shown in FIG. The data at tC3 in the entire image data (position data) shown in FIG. 14 is associated with the data at the timing corresponding to tC3 in the synchronization data shown in FIG. 16.

Next, the control device 100 causes the segmentation unit 156 to segment the image-related data and the medical care-related data included in the synchronized data shown in FIGS. 16 and 17 by separating them at a predetermined timing.

For example, the control device 100 segments the synchronization data at timings that are periodically visited at predetermined time intervals (for example, every 5 seconds).

Alternatively, the control device 100 can control the data of "0" or "1" stored in the image-related data (tray image data, patient image data) and the data of "0" or "1" stored in the medical care-related data. Segment synchronized data based on

Here, even if there is no change in each of the tray image data, patient image data, whole image data, and medical treatment-related data alone, there may be a change in the data when other data is referred to. That is, the control device 100 can divide the synchronous data at an appropriate timing by referring to and combining a plurality of pieces of data from each of the tray image data, patient image data, whole image data, and medical treatment-related data.

For example, as shown in FIG. 16, in the tray image data, each of the data T3 to T7 is the same for the file 305, but the data of each of T3 to T5 is the same for the vacuum. Therefore, by simply referring to the tray image data, the control device 100 may decide whether to segment the synchronization data at T3 to T5 or T3 to T7. However, as shown in FIG. 17, in the medical treatment-related data, each of the data T3 to T5 is the same for the medical instrument and the foot controller 16. Therefore, the control device 100 segments the synchronized data at T3 to T5, which are expected to have a high probability, based on the tray image data and the medical care-related data.

In this way, when segmenting the synchronized data, the control device 100 can segment the synchronous data even if there is insufficient data for each of the tray image data, patient image data, whole image data, and medical treatment-related data. By referring to the data, you can separate the synchronized data at the appropriate timing.

As described above, during medical treatment, tray image data, patient image data, whole image data, and medical treatment-related data are each input to the control device 100 in chronological order. The control device 100 generates synchronization data based on these input data and segments the synchronization data. The control device 100 can estimate control data by learning the estimation model 161 included in the generation unit 160 based on the synchronization data.

For example, in the case of root canal treatment, the control device 100 inputs synchronized data corresponding to T1 and T2 in "examination", inputs synchronized data corresponding to T3 to T5 in "pulp extraction", and inputs synchronized data corresponding to T3 to T5 in "root canal length measurement". In ``, synchronous data corresponding to T6 and T7 is input, synchronous data corresponding to T8 and T9 is input in ``cleaning/disinfection'', synchronous data corresponding to T10 and T11 is input in ``root canal filling'', In "filling/covering", synchronization data corresponding to T12 and T13 is input.

For example, in the case of root canal treatment, the control device 100 learns the above-mentioned synchronization data, so that when synchronization data corresponding to T1 and T2 is input, the control device 100 thereafter performs the synchronization data corresponding to T3 to T5. It is estimated that such synchronization data will be input, and the control data for "extraction" will be generated. When the synchronous data corresponding to T3 to T5 is input, the control device 100 estimates that synchronous data corresponding to T6 and T7 will be input thereafter, and performs the process for "root canal length measurement". Generates control data. When the synchronization data corresponding to T6 and T7 is input, the control device 100 estimates that the synchronization data corresponding to T8 and T9 will be input after that, and the control device 100 assumes that the synchronization data corresponding to T8 and T9 will be input. Generate control data. When the synchronous data corresponding to T8 and T9 is input, the control device 100 estimates that the synchronous data corresponding to T10 and T11 will be input thereafter, and performs the process for "root canal filling". Generate control data. When the synchronization data corresponding to T10 and T11 is input, the control device 100 estimates that synchronization data corresponding to T12 and T13 will be input after that, and performs a process for "stuffing/covering". Generate control data.

As described above, the control device 100 uses so-called AI technology such as a neural network to estimate control data based on at least one of tray image data, patient image data, whole image data, and medical treatment-related data. The control data can be estimated by finding the characteristics for

<Example of control by control device>
An example of control by the control device 100 will be described with reference to FIGS. 18 to 20. 18 to 20 are diagrams for explaining an example of control by the control device 100.

The control data generated by the control device 100 includes data indicating the start of driving the controlled object 190, data indicating the end of driving the controlled object 190, data indicating prohibition of driving the controlled object 190, data indicating continuation of driving the controlled object, At least one of data indicating the drive amount for the controlled object 190, data indicating the upper limit value regarding the driving amount, data indicating the lower limit value regarding the driving amount, data indicating the driving time for the controlled object 190, and data regarding the status of the controlled object 190. Contains one.

For example, as shown in FIG. 18(A), when the control device 100 detects that the patient 2 approaches the chair 11 based on the position data calculated from the captured image acquired by the general camera 53, the control device 100 moves the chair 11 toward the patient. Control data for moving the chair to the second introduction position is generated, and the generated control data is output to the chair control section 111. As shown in FIG. 18(B), the chair control unit 111 moves the footrest 11d of the chair 11 to the introduction position based on the control data from the control device 100.

As shown in FIG. 18(C), the control device 100 allows the patient 2 to sit on the chair 11 and the dentist 3 to approach the chair 11 based on the position data calculated from the photographed image acquired by the general camera 53. When detected, control data for moving the chair 11 to the medical treatment position is generated, and the generated control data is output to the chair control unit 111. As shown in FIG. 18(D), the chair control unit 111 moves the backrest 11b and footrest 11d of the chair 11 to the medical treatment position based on the control data from the control device 100.

The control device 100 causes the dentist to stop looking into the patient's oral cavity and move the medical instruments outside the oral cavity based on the position data calculated from the photographed image acquired by the general camera 53. When detecting a movement such as turning toward the table 12b, control data for moving the chair 11 to the gargling position is generated, and the generated control data is output to the chair control unit 111. The chair control unit 111 moves the chair 11 to the gargling position based on control data from the control device 100.

The control device 100 detects that the patient has completed gargling, placed the cup on the cup stand 12b, and leaned against the backrest 11b of the chair 11, based on the position data calculated from the photographed image acquired by the general camera 53, At that time, if it is detected that the dentist is waiting there, control data for moving the chair 11 to the treatment position again is generated, and the generated control data is output to the chair control section 111. The chair control unit 111 moves the chair 11 to the medical treatment position based on control data from the control device 100.

On the other hand, the control device 100 detects that the patient has completed gargling, placed the cup on the cup stand 12b, and leaned against the backrest 11b of the chair 11, based on the position data calculated from the photographed image acquired by the general camera 53. At the same time, if it is detected that the dentist has left the place, control data for moving the chair 11 to the exit position is generated, and the generated control data is output to the chair control section 111. The chair control unit 111 moves the chair 11 to the exit position based on control data from the control device 100.

When the control device 100 detects that a dentist performs medical treatment using a microscope based on the position data calculated from the photographed image acquired by the general camera 53, the control device 100 changes the speed of the lifting and lowering movement of the chair 11 to a lower speed than usual. It generates control data for setting the , and outputs the generated control data to the chair control section 111 . The chair control unit 111 sets the speed of the chair 11 to a low speed based on control data from the control device 100.

When the control device 100 detects that the dentist adjusts the shape of a dental filling (inlay) when installing a dental filling (inlay) based on the position data calculated from the photographed image acquired by the general camera 53, the control device 100 starts water injection into the medical instruments. Control data for setting the function to OFF is generated, and the generated control data is output to the appliance control unit 211. The instrument control unit 211 sets the water injection function of the medical instrument to OFF based on the control data from the control device 100.

When the control device 100 detects that the dentist is filling the composite resin based on the position data calculated from the photographed image acquired by the general camera 53, the control device 100 turns on the lighting device 19 to suppress the hardening of the composite resin. generates control data for dimming the light, and outputs the generated control data to the illumination control unit 122. The lighting control unit 122 dims the lighting device 19 based on control data from the control device 100.

When the control device 100 detects that the line of sight of the dentist holding the medical instrument (for example, a handpiece) has moved away from the patient's oral cavity for a certain period of time based on position data calculated from the photographed image acquired by the general camera 53, , generates control data for suppressing (stopping, reducing the rotational speed) the driving of the medical instrument, and outputs the generated control data to the instrument control unit 211 . The instrument control unit 211 suppresses the driving of the medical instrument (stops, reduces the rotational speed) based on control data from the control device 100.

When the control device 100 detects that the position of a medical instrument (for example, a handpiece) has deviated from the medical treatment space of a predetermined area for a certain period of time based on the position data calculated from the photographed image acquired by the general camera 53, , generates control data for suppressing (stopping, reducing the rotational speed) the driving of the medical instrument, and outputs the generated control data to the instrument control unit 211 . The instrument control unit 211 suppresses the driving of the medical instrument (stops, reduces the rotational speed) based on control data from the control device 100.

The control device 100 generates control data for outputting a warning sound from the speaker 35 when it detects that the patient has raised his hand during treatment based on the position data calculated from the photographed image acquired by the general camera 53. Then, the generated control data is output to the sound control section 321. The sound control unit 321 outputs a warning sound from the speaker 35 based on control data from the control device 100.

The control device 100 generates control data for displaying a warning image on the display 17 when it detects that the patient has raised his hand during medical treatment based on position data calculated from the photographed image acquired by the general camera 53. Then, the generated control data is output to the display control section 221. The display control unit 221 causes the display 17 to display a warning image based on control data from the control device 100.

When the control device 100 detects that the patient has raised his hand during medical treatment based on the position data calculated from the photographed image acquired by the general camera 53, the control device 100 suppresses the driving of the medical instruments (stops, reduces the rotation speed). The generated control data is output to the appliance control unit 211. The instrument control unit 211 suppresses the driving of the medical instrument (stops, reduces the rotational speed) based on control data from the control device 100.

When the control device 100 detects an abnormality different from normal (for example, the patient is in an abnormal posture) during the operation of the chair 11 based on the position data calculated from the photographed image acquired by the general camera 53, the control device 100 controls the chair 11. 11 is generated, and the generated control data is output to the chair control unit 111. The chair control unit 111 suppresses the operation of the chair 11 (stops, reduces the operation speed) based on control data from the control device 100.

When the control device 100 detects that the dentist has performed an action (gesture) to turn on or off the lighting device 19 based on the position data calculated from the captured image acquired by the general camera 53, the control device 100 turns the lighting device 19 on or off. It generates control data for lighting up or turning off the lights, and outputs the generated control data to the lighting control section 122. The lighting control unit 122 lights up or turns off the lighting device 19 based on control data from the control device 100.

As shown in FIG. 19, when the control device 100 detects that a certain treatment has been completed based on the position data calculated from the captured image acquired by the general camera 53, the control device 100 operates the medical device 1 for the next treatment. It is also possible to generate control data for informing the surgeon of the operation of the chair 11, the operation of the medical instrument 15, etc., and output the generated control data to the medical device 1. As a notification mode, the control device 100 sends control data to the medical device 1 for blinking an icon image displayed on the operation panel 18 or blinking an operation button of a medical device 15 such as a vacuum or air turbine handpiece. You can also output to The control device 100 is equipped with a display function that notifies the user of the next treatment through sounds such as a buzzer or voice, and informs the user of the next treatment by displaying illustrations or text on the display 17. Control data may be generated to provide operational support through other notifications, such as displaying information on glasses to notify the next treatment details.

As shown in FIG. 20, the control device 100 detects, based on position data calculated from the photographed image acquired by the general camera 53, that the surgeon is performing an abnormal movement that is not normally possible during a certain procedure. When detected, control data for outputting an alert may be generated, and the generated control data may be output to the medical treatment device 1. For example, when the control device 100 detects that the dentist 3 is looking backwards while holding a medical instrument during a certain procedure, the control device 100 sends control data to the medical device 1 to output an alert from the operation panel 18 and the speaker 35. You can also output it. Furthermore, the control device 100 increases the output level (brightness and volume level) of the alert or increases the output frequency of the alert, the higher the degree of abnormality (risk) of the operator's behavior. control data may also be generated.

Note that, as shown in FIGS. 19 and 20, the control system 1000 may include a mobile terminal 401 or wearable glasses 402 separately from the medical device 1 or as a part of the medical device 1. Based on the generated control data, the control device 100 displays a predetermined image (for example, operation of the medical device 1 for the next treatment) on a mobile terminal 401 (for example, a tablet terminal) or wearable glasses 402 that can be worn by a surgeon or the like. , an alert image indicating an abnormality, etc.) or outputting a predetermined sound (for example, a sound indicating operation of the medical device 1 for the next treatment, an alert sound indicating an abnormality, etc.) good.

The control device 100 generates control data for controlling the status of the controlled object 190 based on the position data calculated from the photographed image acquired by the general camera 53, and outputs the generated control data to the medical treatment device 1. It's okay. For example, the control device 100 checks the status of the medical device 1, such as the state of the sensors included in the medical device 1, the state of the actuators, and the version of the program, and controls the control data that is originally generated according to the confirmed status. The control data may be changed to the control data estimated according to the person's motion, or the control data estimated according to the person's motion may be added.

Specifically, the control device 100 may cause the display 17 to display an icon indicating the medical treatment status or the like according to the status of the controlled object 190. When controlling drive sources such as motors, compressors, and heaters, the control device 100 may perform feedback control of these drive sources according to sensor states (output values) so that desired control values are achieved. At this time, control data for feedback control may be newly generated or changed depending on the person's motion.

The control device 100 generates control data for controlling various settings such as the safety mode based on the position data calculated from the photographed image acquired by the general camera 53, and transmits the generated control data to the medical treatment device 1. You can also output it. For example, there are various settings depending on the mode, such as a limit on the brightness of the lighting device 19, a limit on the output of the handpiece, a limit on the vacuum suction, a limit on the tilting speed of the backrest 11b, and a limit on the lifting speed of the seat 11c. When control of the controlled object 190 is determined, the control device 100 may generate control data for changing control values set according to these modes.

In this way, the control device 100 may generate control data for directly controlling the controlled object 190, or control data for controlling the mode for controlling the controlled object 190, etc. 190) may also be generated.

In this way, the control device 100 controls the medical device 1 prior to the person's behavior by analyzing the person's behavior based on the position data calculated from the photographed image acquired by the general camera 53. Convenience for the operator can be improved.

Note that in the example described above, the control device 100 generates control data based on position data calculated from the photographed image acquired by the general camera 53, but the control device 100 generates the control data based on the position data calculated from the photographed image acquired by the general camera 53. The control data may be generated by referring to other data such as.

<Control data generation process of control device>
The control data generation process executed by the control device 100 in the operation stage will be described with reference to FIG. 21. FIG. 21 is a flowchart for explaining an example of a control data generation process executed by the control device 100. Each step (hereinafter referred to as "S") shown in FIG. 21 is realized by the arithmetic unit 102 of the control device 100 executing the OS 142 and the control program 141.

First, the control device 100 determines whether a start condition for the control data generation process is satisfied. For example, as shown in FIG. 21, it is determined whether image-related data has been input (S1). When the image-related data is input (YES in S1), the control device 100 performs image recognition (S2). For example, when the entire image data is input, the control device 100 uses the position detection unit 157 to detect the position of the person based on the entire image data and generates position data. When patient image data or tray image data is input, the control device 100 uses the object detection unit 152 to detect the presence, shape, type, etc. of an object such as a medical instrument. In this way, the control device 100 executes the control data generation process when image-related data is input and image recognition is performed based on the image-related data. For example, when the control device 100 detects a patient in the chair 11, it starts executing the control data generation process.

If image-related data has not been input (NO in S1), or after the process in S2, the control device 100 determines whether medical treatment-related data has been input (S3). When the medical care-related data is input (YES in S3), the control device 100 converts the medical care-related data into a predetermined format using the conversion unit 154 (S4).

If no medical care-related data has been input (NO in S3), or after the processing in S4, the control device 100 determines whether a predetermined amount of image-related data and medical care-related data has been accumulated (S5). For example, the control device 100 determines whether the amount of data required to estimate and generate control data has been accumulated. If the control device 100 has not accumulated a predetermined amount of image-related data and medical treatment-related data (NO in S5), the control device 100 returns to the process in S1.

On the other hand, when the control device 100 has accumulated a predetermined amount of image-related data and medical care-related data (YES in S5), the synchronization unit 155 generates synchronized data by synchronizing the image-related data and the medical care-related data ( S6).

Next, the control device 100 segments the synchronous data by having the segmentation unit 156 separate the image-related data and the medical care-related data at a predetermined timing based on the synchronous data (S7).

Next, the control device 100 uses the generation unit 160 to estimate control data based on the segmented synchronization data, and generates the estimated control data (S8).

Next, the control device 100 outputs the generated control data to the control target 190 of the medical device 1 (S9), and ends this process.

In this way, the control device 100 uses the control method for the medical device 1 defined by the control program 141 to control at least one of image-related data and medical care-related data using so-called AI technology such as a neural network. Based on this, control data can be generated. As a result, the control device 100 can control the medical treatment device 1 based on the movement of a person existing in the medical treatment space, the presence or absence of objects such as medical instruments, and the operation of the medical treatment device 1, which is convenient for the surgeon. can improve sex.

Note that the control device 100 may start executing the control data generation process when image-related data is input and a start condition other than performing image recognition based on the image-related data is satisfied. The start condition may be satisfied, for example, when the power to the control device 100 is turned on, or when the mode corresponding to the control data generation process is switched after the power to the control device 100 is turned on. You can. Further, the start condition may be satisfied when a certain amount of image-related data and medical treatment-related data is acquired. Furthermore, the start condition may be satisfied when the operator performs an input operation (for example, operating a start switch on the control device 100) that notifies the start of medical treatment, or the operator notifies the end of medical treatment. It may be established when an input operation (for example, operation of an end switch of control device 100) is performed. The start condition may be one that is satisfied when some kind of start event occurs in the control device 100.

<Main disclosure>
As described above, this embodiment includes the following disclosures.

[Configuration 1]
The control device (100) has an input section (150) into which image-related data captured by a camera (51, 52, 53) of a medical treatment space including the medical treatment device (1) is inputted, and an input section (150) that receives input from the input section (150). a generation unit (160) that generates control data for controlling the medical treatment device (1) based on image-related data and a first estimation model (161) including a first neural network (162); and an output section (180) that outputs the control data generated by (160) to the controlled object (190).

Thereby, the control device 100 can control the medical treatment device 1 based on the image-related data using so-called AI technology such as the neural network 162, so that convenience for the operator can be improved.

[Configuration 2]
The control device (100) further includes a detection unit (151) that detects the position of a person included in the medical treatment space based on image-related data (whole image data) input from the input unit (150). The generation unit (160) generates control data based on at least one position data for specifying the position of the person detected by the detection unit (151) and the first estimation model (161).

With this, the control device 100 can detect the position of the person included in the medical treatment space, predict the next movement of the person, and control the medical treatment device 1 according to the next movement of the person. , it is possible to improve convenience for the operator.

[Configuration 3]
The detection unit (151) detects the position of the person based on image-related data input from the input unit (150) and a second estimation model (1511) including a second neural network (1512).

Thereby, the control device 100 can detect the position of the person based on the image-related data using the so-called AI technology such as the neural network 1512, and therefore can control the medical treatment device 1 with more precision.

[Configuration 4]
The position data includes at least data regarding the position of any one of a person's joints, eyes, nose, mouth, ears, and head.

Thereby, the control device 100 controls the medical treatment device 1 with more precision by detecting the positions of detection points such as joints, eyes, nose, mouth, ears, and head where the motion of the person is likely to be reflected. be able to.

[Configuration 5]
The control device (100) further includes a recording unit (103) that records position data in association with time information (time stamp).

Thereby, the control device 100 can use the position data used when performing new machine learning again when relearning. Furthermore, when multiple control devices 100 share the position data recorded in the recording device 103, a time stamp is associated with the position data, so that the multiple control devices 100 perform similar machine learning. be able to.

[Configuration 6]
The recording unit (103) records the position data newly obtained by the detection unit (151) when the position data newly obtained by the detection unit (151) is different from the recorded position data.

As a result, the control device 100 does not have the recording device 103 record new position data each time the person's position is detected, but records new position data only when the person's position data changes. Since the data is recorded by the device 103, an increase in the amount of data recorded by the recording device 103 can be suppressed as much as possible.

[Configuration 7]
The input unit (150) further receives input of medical care-related data acquired by the medical device (1). The generation unit (160) generates control data based on the image-related data and medical treatment-related data input from the input unit (150) and the first estimation model (161).

As a result, the control device 100 generates control data for controlling the medical device 1 based on the medical care-related data acquired by the medical device 1 in addition to the image-related data, so the control device 100 can control the medical device 1 with more precision. can be controlled.

[Configuration 8]
The medical care-related data includes the chair (11), medical equipment (15), lighting device (19), water supply/drainage device (12), foot controller (16), display (17), and operation included in the medical device (1). It includes past and/or current control data for at least one of the panels (18).

As a result, the control device 100 can transmit information such as data for driving the chair 11, data for driving the medical equipment 15, data for driving the lighting device 19, and data for driving the water supply/drainage device (basin unit) 12 during medical treatment. By using the generated data, it is possible to compensate for the lack of image-related data for estimating control data, and it is possible to control the medical device 1 with more precision.

[Configuration 9]
The controlled objects (190) include a chair (11), a medical instrument (15), a lighting device (19), a water supply/drainage device (12), a foot controller (16), and a display (17) included in the medical device (1). , an operation panel (18), and a speaker (35).

Thereby, the control device 100 can control each control target 190, such as the chair 11, included in the medical treatment device 1, so that convenience for the operator can be improved.

[Configuration 10]
The control data includes data indicating the start of driving the controlled object (190), data indicating the end of driving the controlled object (190), data indicating prohibition of driving the controlled object (190), and data indicating continuation of driving the controlled object (190). data, data indicating the drive amount for the controlled object (190), data indicating the upper limit value regarding the driving amount, data indicating the lower limit value regarding the driving amount, data indicating the driving time for the controlled object (190), and the controlled object (190) includes at least one of the data regarding the status of.

Thereby, the control device 100 can directly or indirectly control the controlled object 190 using the generated control data.

[Configuration 11]
The first neural network (162) is a recurrent neural network.

Thereby, the control device 100 can generate control data with higher accuracy by using a recurrent neural network, which is an AI technology suitable for image-related data and medical care-related data input in time series.

[Configuration 12]
A control method for controlling a medical treatment device (1) for treating a patient using a computer (102) includes inputting image-related data captured by a camera (51, 52, 53) of a medical treatment space including the medical treatment device (1). control data for controlling the medical device (1) based on the input image-related data and the first estimation model (161) including the first neural network (162); It includes a step of generating (S9) and a step (S10) of outputting the generated control data to the controlled object (190).

Thereby, the control device 100 can control the medical treatment device 1 based on the image-related data using so-called AI technology such as the neural network 162, so that convenience for the operator can be improved.

[Configuration 13]
The control program (141) includes steps (S2, S4) in which image-related data captured by a camera (51, 52, 53) of a medical treatment space including the medical treatment device (1) is input into the computer (102); a step (S9) of generating control data for controlling the medical device (1) based on the image-related data to be generated and the first estimation model (161) including the first neural network (162); A step (S10) of outputting the control data to the controlled object (190) is executed.

Thereby, the control device 100 can control the medical treatment device 1 based on the image-related data using so-called AI technology such as the neural network 162, so that convenience for the operator can be improved.

<Modified example>
The present invention is not limited to the above embodiments, and can be further modified and applied in various ways. Modifications applicable to the present invention will be described below.

[About the camera]
In this embodiment, one tray camera 51, one patient camera 52, and one general camera 53 are each attached to the medical device 1, but a plurality of these cameras may be attached to the medical device 1. .

The tray camera 51, the patient camera 52, and the general camera 53 are not limited to fixed types, and may be movable types that follow the movement of the tray table 13 on which the tray 30 is placed or the movement of the operator.

The tray camera 51, the patient camera 52, and the overall camera 53 may be mounted on the medical device 1, or may be cameras whose optical axes are directed from the periphery of the medical device 1 toward the medical device 1. Furthermore, in a medical treatment space that can be understood by images captured by cameras, a plurality of cameras may be installed so that there are no blind spots in imaging. Furthermore, the control device 100 may generate the control data using an image taken by an intraoral camera mounted on a medical instrument, a microscope, or a camera mounted on glasses used by a surgeon.

By analyzing the image captured by the tray camera 51 through image recognition, the control device 100 not only identifies the presence, shape, and type of medical instruments placed on the tray 30, but also identifies the presence, shape, and type of medical instruments placed on the tray 30. The medical instrument held in the operator's hand, or the medical instrument placed in the operator's hands, such as a dentist or dental assistant, may be identified.

[About the control device]
The control device 100 may be included in the basin unit 12 and included in the medical treatment device 1. Alternatively, the control device 100 may be a computer such as an in-hospital server that is separate from the medical device 1 and is communicably connected to the medical device 1. Furthermore, the control device 100 may exist in the form of cloud computing, such as an external server installed outside the medical treatment space where the medical treatment device 1 is installed. In this case, the control device 100 is connected to a plurality of medical treatment devices 1 installed in the medical treatment space, and is also connected to a plurality of medical treatment devices 1 installed in other dental clinics. For each, control data may be generated based on image-related data and medical treatment-related data. In this way, the frequency of machine learning by the control device 100 will further increase, and the control device 100 can generate control data with higher accuracy.

The generation unit 160 of the control device 100 is not limited to estimating control data of the medical device 1 using the estimation model 161. For example, the generation unit 160 may estimate the next operation (control content) of the medical device 1 using the estimation model 161, and generate control data corresponding to the estimation result. Alternatively, the generation unit 160 may estimate the next movement of the person using the estimation model 161 and generate control data corresponding to the estimation result. Alternatively, the generation unit 160 may estimate the medical treatment procedure (for example, the content of the next treatment to be performed during the medical treatment) using the estimation model 161, and generate control data corresponding to the estimation result.

Based on the estimation results of the estimation model 161, the control device 100 is not limited to directly controlling the medical device 1, but can also display a warning image on a display separate from the medical device 1, or output a warning sound to a speaker. Alternatively, the operator may operate the medical device 1 (for example, perform a stop operation) by doing so. That is, the control device 100 may not directly control the medical device 1, but may indirectly control the medical device 1 by guiding the surgeon.

The control device 100 according to the present embodiment controls the medical treatment device 1 used by an operator to perform dental treatment on a patient. It may also be something that controls the device.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the above description, and it is intended that all changes within the scope and meanings equivalent to the claims are included. Note that the configurations exemplified in this embodiment and the configurations exemplified in the modified examples can be combined as appropriate.

1 Medical equipment, 2 Patient, 3 Dentist, 4 Dental assistant, 5 Pole, 6 Arm, 11 Chair, 11a Headrest, 11b Backrest, 11c Seat, 11d Foot rest, 12 Basin unit, 12a Bowl, 12b Cup stand , 12c water tap, 13 tray table, 14 instrument holder, 15 medical instruments, 16 foot controller, 17 display, 18 operation panel, 19 lighting device, 21 instrument control device, 22 display control device, 30 tray, 32 sound control device, 35 speaker, 51 tray camera, 52 patient camera, 53 overall camera, 100 control device, 101 communication device, 102 arithmetic device, 103 recording device, 111 chair control section, 121 basin control section, 122 illumination control section, 123 storage section, 141 control program, 150 input unit, 151 XY detection unit, 152 object detection unit, 153 Z detection unit, 154 conversion unit, 155 synchronization unit, 156 segmentation unit, 157 position detection unit, 160 generation unit, 161, 1511 estimation model , 162, 1512 neural network, 163, 1513 parameter, 180 output section, 190 controlled object, 211 instrument control section, 221 display control section, 222 panel control section, 301 rubber dam moisture proofing set, 302 rubber dam sheet, 303 root canal length measuring device , 304 bar set, 305 file, 306 mouth corner counter electrode, 307 foul clip, 308 brooch, 309 cleaning needle, 310 cleaning syringe, 311 temporary sealant filler, 312 cleanser, 313 turbine, 314 tweezers, 315 vacuum, 316 mirror , 317 excavator, 318 needle, 319 root canal material injector, 321 sound control unit, 401 mobile terminal, 402 wearable glasses, 1000 control system.

Claims (12)

A control device that controls a medical treatment device for treating a patient,
an input unit into which image-related data captured by a camera of a medical treatment space including the medical treatment device is input;
a detection unit that detects the position of a person included in the medical treatment space based on the image-related data input from the input unit;
By inputting at least one position data for specifying the position of the person detected by the detection unit into a first estimation model including a first neural network, the medical treatment apparatus is used as an output of the first estimation model. a generation unit that generates control data for controlling;
A control device comprising: an output section that outputs the control data generated by the generation section to a controlled object. The control device according to claim 1 , wherein the detection unit detects the position of the person based on the image-related data input from the input unit and a second estimation model including a second neural network. The control device according to claim 1 or 2 , wherein the position data includes at least data regarding the position of any one of joints, eyes, nose, mouth, ears, and head of the person. The control device according to any one of claims 1 to 3 , further comprising a recording unit that records the position data in association with time information. 5. The recording unit records the position data newly obtained by the detection unit when the position data newly obtained by the detection unit is different from the recorded position data. Control device as described. The input unit further receives medical care-related data acquired by the medical device,
The control device according to any one of claims 1 to 5 , wherein the generation unit generates the control data based on the position data, the medical care-related data, and the first estimation model. The medical treatment-related data includes past and present control data on at least one of the chair, medical instruments, lighting device, water supply/drainage device, foot controller, display, and operation panel included in the medical treatment device. The control device according to claim 6 , comprising: Claims 1 to 7 , wherein the controlled object includes at least one of a chair, a medical instrument, a lighting device, a water supply/drainage device, a foot controller, a display, an operation panel, and a speaker included in the medical treatment device. The control device according to any one of the above. The control data includes data indicating the start of driving the controlled object, data indicating the end of driving the controlled object, data indicating prohibition of driving the controlled object, data indicating continuation of driving the controlled object, and data indicating driving the controlled object. data indicating the amount, data indicating an upper limit value regarding the drive amount, data indicating a lower limit value regarding the drive amount, data indicating a driving time for the controlled object, and data regarding the status of the controlled object, including at least one of the following. The control device according to any one of claims 1 to 8 . The control device according to any one of claims 1 to 9 , wherein the first neural network is a recurrent neural network. A control method for controlling a medical treatment device for treating a patient using a computer, the control method comprising:
As the process executed by the computer,
inputting image-related data captured by a camera of a medical treatment space including the medical treatment device;
Detecting the position of a person included in the medical treatment space based on the image-related data;
By inputting at least one position data for specifying the position of the person detected in the detecting step to a first estimation model including a first neural network, the medical treatment apparatus can be used as an output of the first estimation model. a step of generating control data for controlling the
A control method comprising the step of outputting the generated control data to a controlled object. A control program that controls a medical treatment device for treating a patient,
to the computer,
inputting image-related data captured by a camera of a medical treatment space including the medical treatment device;
Detecting the position of a person included in the medical treatment space based on the image-related data;
By inputting at least one position data for specifying the position of the person detected in the detecting step to a first estimation model including a first neural network, the medical treatment apparatus can be used as an output of the first estimation model. a step of generating control data for controlling the
and outputting the generated control data to a controlled object.

Images