JP7428634B2 - Control device, control method, and control program - Google Patents

Description

The present invention relates to a control device, a control method, and a control program for controlling the recording of medical data related to a series of medical treatments.

Conventionally, medical treatment devices for treating patients are known. For example, Patent Document 1 discloses a medical treatment device including a chair, medical instruments, a lighting device, and other configurations for treating a patient.

Patent No. 6388849

According to the medical treatment device disclosed in Patent Document 1, a surgeon can treat a patient by using each configuration of the medical treatment device according to the contents of the medical treatment. Considering that in recent years, technology has been created that uses AI (Artificial Intelligence) to analyze, learn, and utilize data, using AI technology in medical equipment will make medical treatment more convenient for surgeons. equipment can be provided.

In order to use AI technology in a medical treatment device, it is necessary to collect and accumulate medical data related to a series of medical treatments that are necessary for machine learning. However, if medical data is recorded without any modification, the recorded medical data may be complicated and difficult to utilize during machine learning.

The present invention aims to record clinical data that is easy to utilize.

According to an example of the present disclosure, a control device that controls recording of dental treatment data regarding a series of dental treatments is provided. The control device includes an input section into which dental treatment data is input, and a control section that controls a recording section that records the dental treatment data. The control unit causes the recording unit to start recording the dental treatment data input to the input unit when a series of dental treatment start conditions is satisfied. The start condition is a condition in which the start of a series of dental treatments is presumed; The height of the bed has exceeded the specified height, the angle of the backrest of the chair has exceeded the specified angle, the driving of medical instruments has been turned on, the power of the lighting device has been turned on, This includes at least one of detecting a person sitting on a chair, detecting a medical instrument in the medical treatment space, and detecting a person in the medical treatment space.

According to an example of the present disclosure, a control method for controlling recording of dental treatment data regarding a series of dental treatments by a computer is provided. The control method includes the steps of inputting dental treatment data, determining whether a series of dental treatment start conditions are satisfied, and recording the input dental treatment data when the start conditions are satisfied. and a step of causing the recording unit to end recording of the input medical treatment data when a series of dental treatment termination conditions are satisfied. The start condition is a condition in which the start of a series of dental treatments is presumed; The height of the bed has exceeded the specified height, the angle of the backrest of the chair has exceeded the specified angle, the driving of medical instruments has been turned on, the power of the lighting device has been turned on, This includes at least one of detecting a person sitting on a chair, detecting a medical instrument in the medical treatment space, and detecting a person in the medical treatment space.

According to an example of the present disclosure, a control program that controls recording of dental treatment data regarding a series of dental treatments is provided. The control program includes the steps of inputting dental treatment data into the computer, determining whether or not a series of dental treatment start conditions are satisfied, and determining whether or not the input dental treatment data is established when the start conditions are satisfied. To cause a recording unit to start recording data, and to cause a recording unit to terminate recording of input medical data when a series of dental treatment termination conditions are satisfied. The start condition is a condition in which the start of a series of dental treatments is presumed; The height of the bed has exceeded the specified height, the angle of the backrest of the chair has exceeded the specified angle, the driving of medical instruments has been turned on, the power of the lighting device has been turned on, This includes at least one of detecting a person sitting on a chair, detecting a medical instrument in the medical treatment space, and detecting a person in the medical treatment space.

According to the present disclosure, it is possible to record medical data that is easy to utilize.

FIG. 1 is a schematic diagram showing the overall configuration of a control system according to the present embodiment. FIG. 3 is a diagram for explaining an example of an image taken by a tray camera. FIG. 3 is a diagram for explaining an example of an image taken by a patient camera. FIG. 3 is a diagram for explaining an example of an image taken by a general camera. FIG. 1 is a block diagram showing the internal configuration of a control system according to the present embodiment. FIG. 3 is a diagram for explaining an example of position data acquired by the control device according to the present embodiment. FIG. 3 is a diagram for explaining an example of detection points of position data. FIG. 3 is a diagram for explaining an example of position coordinates corresponding to position data. FIG. 2 is a block diagram showing the functional configuration of the control device according to the present embodiment at the stage of recording medical data. FIG. 3 is a diagram for explaining the data structure of learning data. FIG. 2 is a block diagram showing the functional configuration of the control device according to the present embodiment at a learning stage. FIG. 2 is a block diagram showing the functional configuration of the control device according to the present embodiment at an operation stage. It is a figure for explaining an example of the photographed image of the tray camera in root canal treatment. FIG. 3 is a diagram for explaining an example of tray image data acquired from photographic information of a tray camera. FIG. 3 is a diagram for explaining an example of patient image data acquired from photographic information of a patient camera. FIG. 3 is a diagram for explaining an example of position data acquired from photographic information of a general camera. FIG. 3 is a diagram for explaining an example of device-related data acquired from a medical treatment device. FIG. 3 is a diagram for explaining an example of synchronous data. FIG. 3 is a diagram for explaining an example of synchronous data. It is a timing chart for explaining the recording timing of medical data. 3 is a flowchart for explaining an example of a recording control process executed by a control device. It is a flowchart for explaining an example of learning processing which a control device performs.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are designated by the same reference numerals, and the description thereof will not be repeated.

<Overall configuration of control system>
FIG. 1 is a schematic diagram showing the overall configuration of a control system 1000 according to this embodiment. A control system 1000 according to this embodiment controls a medical treatment device 1 for treating a patient.

“Medical treatment” includes at least one of medical examination and treatment. "Examination" includes the operator's investigation of the patient's medical condition and etiology. "Treatment" includes a practitioner curing a patient's illness and a practitioner maintaining the patient's beauty and health (eg, aesthetic treatment). The subject of medical treatment (examination, treatment) includes at least one of dentistry and medicine. "Dentistry" is a medical department that deals with diseases related to teeth or tooth-related tissues (periodontal tissues, etc.), and includes, for example, general dentistry, orthodontics, oral surgery, dental radiology, pediatric dentistry, and the like. "Medical" includes, for example, internal medicine, pediatrics, surgery, ophthalmology, otorhinolaryngology, obstetrics and gynecology, dermatology, neurosurgery, cardiology, and orthopedics. The medical treatment device 1 according to the present embodiment is used by an operator to perform dental treatment on a patient. “Surgeons” include doctors who treat patients (e.g., dentists), assistants who assist doctors (e.g., dental assistants), teachers at dental or medical schools, students at dental or medical schools, and dental technicians. including.

A series of actions in which a surgeon performs examination or treatment on a patient who comes for treatment is also referred to as a "series of medical treatment." In other words, a "series of medical treatments" includes all actions (examination, treatment) performed by the surgeon from the start of the medical treatment until the end of the medical treatment. A “series of medical treatments” may be a medical treatment unit predetermined by the designer of the medical device 1 or the like. A "series of medical treatments" may be a medical treatment unit predetermined by the operator. A "series of medical treatments" may be a single medical treatment unit performed for one patient.

As shown in FIG. 1, the control system 1000 includes a medical device 1, a control device 100, and a server device 200.

The medical treatment device 1 is, for example, a chair unit for an operator to perform dental treatment on a patient. The medical device 1 includes a chair 11, a basin unit 12, a tray table 13, an instrument holder 14 holding medical instruments 15, a foot controller 16, a display 17, an operation panel 18, a lighting device 19, and instruments. It includes a control device 21, a display control device 22, and a speaker 35.

The chair 11 is a chair on which a patient sits during medical treatment, and includes a headrest 11a that supports the patient's head, a backrest 11b that supports the patient's back, a seat sheet 11c that supports the patient's tail, and a footrest that supports the patient's feet. 11d. Note that the chair 11 does not necessarily need to include the backrest 11b and the seat seat 11c. For example, the chair 11 may include a bed in which a portion corresponding to the backrest 11b and a portion corresponding to the seat sheet 11c are integrated.

The basin unit 12 is a water supply/drainage device that includes a bowl 12a in which a drainage port is formed, a cup stand 12b on which a cup is placed, and a water tap 12c for supplying water to the cup.

The tray table 13 is used as a storage stand during medical treatment. The tray table 13 is connected to an arm (not shown) extending from the chair 11 or the floor on which the chair 11 is installed. Note that the tray table 13 may be suspended by an arm (not shown) extending from the chair 11 or the floor on which the chair 11 is installed. For example, the tray table 13 may be suspended by an arm 6 that branches at the top of a pole 5 that extends from the chair 11 or the floor on which the chair 11 is installed. The operator can manually rotate, horizontally move, and vertically move the tray table 13 with respect to the chair 11. During medical treatment, the surgeon may place the tray 30 on the upper surface of the tray table 13. One or more medical instruments for treating a patient are placed on the tray 30.

"Medical instruments" are medical instruments used by a surgeon during medical treatment, such as tweezers, mirrors, excavators, deep needles, and microscopes. Although the medical instruments 15 held by the instrument holder 14 of the medical device 1 are also included in the "medical instruments," medical instruments that are not held by the instrument holder 14 are mainly placed on the tray 30.

The medical instrument 15 is an instrument for dental treatment, such as an air turbine handpiece, a micromotor handpiece, an ultrasonic scaler, and a vacuum syringe, and is driven under the control of the instrument control device 21 . The medical instruments 15 are not limited to these, and may be an intraoral camera, a photopolymerization irradiator, a root canal length measuring device, a three-dimensional scanner, a root canal expander, or a mirror, a syringe, and a root canal expander. It may also be a non-driven device, such as a filling device.

The foot controller 16 has a plurality of switches (pedals) that accept foot operations by the operator. The operator can assign a predetermined function to each of the plurality of switches. For example, the surgeon can assign the function of changing the posture of the chair 11 to a switch on the foot controller 16, and when the foot controller 16 accepts the operator's foot operation of the switch, the headrest will be adjusted based on the foot operation. 11a etc. are driven. Furthermore, the operator can also assign the function of driving the medical instrument 15 to a switch of the foot controller 16, and when the foot controller 16 receives a foot operation of the switch by the operator, the instrument is controlled based on the foot operation. The medical instrument 15 is driven by the control of the device 21. Note that the surgeon can also assign other functions to the switches of the foot controller 16, such as a function to control lighting and turning off of the lighting device 19.

The display 17 is attached to the tray table 13 and displays various images under the control of the display control device 22.

The operation panel 18 includes switches for operating the chair 11, the medical instruments 15, etc., or for setting the operations. For example, when the operation panel 18 receives an input operation for operating the chair 11, the chair 11 operates based on the input operation. For example, when the operation panel 18 receives an input operation for setting the rotational speed of the medical instrument 15, etc., the rotational speed and the like of the medical instrument 15 can be set based on the input operation.

The lighting device 19 is provided at the tip of an arm 6 that branches at the top of the pole 5 extending from the chair 11 or the floor on which the chair 11 is installed. The lighting device 19 can be switched between a lighting state and a non-lighting state, and supports medical treatment by a surgeon by illumination. Note that not only the lighting device 19 but also the display 17 may be attached to the tip of the pole 5 or the arm 6.

The speaker 35 outputs various sounds, such as a sound to alert the surgeon and the patient, and an assist sound to assist in medical treatment.

An operator can treat a patient by using each configuration of the medical device 1 according to the contents of the medical treatment. For example, when placing a patient in a chair, the operator operates the foot controller 16 to operate the chair 11 so that the patient is in a comfortable sitting position. For example, when treating a patient, the surgeon switches the illumination device 19 from an off state to an illuminated state to make it easier to visually recognize the inside of the patient's oral cavity. For example, when treating a patient, a surgeon drives the medical instrument 15 by operating the foot controller 16.

Considering that in recent years, technology has been created that uses AI (Artificial Intelligence) to analyze, learn, and utilize data, using AI technology in the medical treatment device 1 can be highly convenient for the surgeon. A medical device 1 can be provided.

For example, a surgeon can treat a patient by operating each component of the medical treatment device 1 during a series of medical treatments for the patient. If it is possible to control the medical device 1 by reading ahead using technology, it is possible to provide the medical device 1 that is highly convenient for the operator.

During a series of medical treatments, the dentist, dental assistant, and patient each perform actions according to a routine that is generally determined according to the content of the medical treatment, and take approximately predetermined postures depending on the content of the medical treatment. The order and posture of the actions performed by the dentist, the dental assistant, and the patient can be said to represent a medical treatment procedure, and the medical treatment apparatus 1 is also controlled to correspond to the procedure. Therefore, if the procedure of medical treatment can be understood based on the order of actions and postures performed by each of the dentist, dental assistant, and patient, it becomes possible to predict the control of the medical treatment apparatus 1 in advance.

Further, the operator selects an appropriate medical instrument from among the plurality of medical instruments according to the medical treatment content, takes it out from the tray 30, and performs medical treatment using the taken out medical instrument. The type of medical instrument selected and the order in which the medical instruments are used can be said to represent a medical procedure, and the medical apparatus 1 is also controlled to correspond to the procedure. Therefore, if the procedure of medical treatment can be understood based on the type of medical instruments selected by the operator and the order in which the medical instruments are used, it becomes possible to predict the control of the medical apparatus 1 in advance.

The control system 1000 (control device 100) according to the present embodiment analyzes and learns the behaviors of dentists, dental assistants, and patients using AI to generate control data for controlling the medical device 1. is configured to estimate. Furthermore, the control system 1000 (control device 100) according to the present embodiment uses AI to analyze and learn the types of medical instruments selected by the surgeon and the order in which the medical instruments are used. 1 is configured to estimate control data of 1.

Here, in order to use AI technology in the medical device 1, it is necessary to collect and accumulate clinical data related to a series of medical treatments required for machine learning. However, if medical data is recorded without any modification, the recorded medical data may be complicated and difficult to utilize during machine learning. For example, if you do not start recording medical data at the start of a series of medical treatments, or if you do not finish recording medical data at the end of a series of medical treatments, There is a risk that there is not enough medical data, or that the medical data contains useless data that is unrelated to the series of medical treatments. Even if machine learning is performed using such medical data, it is difficult to improve the accuracy of machine learning.

Therefore, the control system 1000 (control device 100) according to the present embodiment provides a technique for recording medical data that can be easily utilized as learning data for machine learning. Recording of medical data and machine learning by the control system 1000 (control device 100) according to the present embodiment will be specifically described below.

A plurality of cameras are attached to the medical device 1 according to this embodiment. Specifically, the control system 1000 includes a tray camera 51 attached to the display 17, a patient camera 52 attached to the illumination device 19, and an overall camera 53 attached to the top of the pole 5.

The tray camera 51 is arranged to take a video or still image of an area including at least the tray 30. Image data including photographic information obtained by the tray camera 51 (hereinafter also referred to as “tray image data”) is acquired by the control device 100. Various settings such as the shutter speed in the tray camera 51 are adjusted in advance to such an extent that the control device 100 can recognize the presence or absence, shape, type, etc. of medical instruments placed on the tray 30. Note that the tray camera 51 may be installed at any location as long as it can photograph an area including at least the tray 30.

The patient camera 52 is arranged so as to take a video or still image of an area including at least the inside of the patient's oral cavity. Normally, during medical treatment, the inside of the patient's oral cavity is illuminated by the illumination device 19, so the patient camera 52 attached to the illumination device 19 can naturally photograph an area including at least the inside of the patient's oral cavity. Image data including imaging information obtained by the patient camera 52 (hereinafter also referred to as “patient image data”) is acquired by the control device 100. Various settings such as the shutter speed in the patient camera 52 are adjusted in advance to the extent that the control device 100 can recognize the inside of the oral cavity of the patient during treatment. Note that the patient camera 52 may be installed at any location as long as it can photograph an area including at least the inside of the patient's oral cavity. The patient camera 52 is a three-dimensional camera similar to the general camera 53 so that the control device 100 can detect the positional relationship in the depth direction (for example, the direction in which the patient is viewed from the patient camera 52) between the area around the oral cavity and the medical instruments. The position coordinates may be detectable.

The general camera 53 is arranged so as to photograph a region including at least the medical treatment space including the medical treatment device 1 as a moving image or a still image. Specifically, the overall camera 53 is arranged so as to take a moving image or a still image of a wide area including at least the dentist, dental assistant, and patient in the medical treatment space during medical treatment using the medical treatment device 1. There is. Since the general camera 53 is attached to the upper part of the pole 5, it photographs the actions of the dentist, dental assistant, and patient during treatment, as well as the operation of the medical treatment apparatus 1 from above.

The overall camera 53 is configured with a 3D camera or the like so as to be able to detect the three-dimensional position coordinates of the operator and the patient. For example, the overall camera 53 is a ToF (Time of Flight) camera that can detect three-dimensional position coordinates by photographing with a camera and measuring the distance to an object using reflection of light (for example, infrared rays), or , a stereo camera that can detect three-dimensional position coordinates by taking pictures with two cameras.

The general camera 53 includes a camera that takes images from the front of the chair 11 of the medical device 1 (or a patient sitting on the chair 11) as shown in FIG. The camera may also include a camera that takes pictures from the back of the patient. In this way, the overall camera 53 can photograph the medical treatment space not only from the front of the medical treatment device 1 but also from the back, so it can photograph the medical treatment space in more detail and over a wider range without blind spots. be able to. Further, the general camera 53 may be capable of tracking a subject to be imaged (for example, a person such as a surgeon and a patient) present in the medical treatment space.

The "medical treatment space" is not limited to a space that includes all of the objects included in the treatment space, but also includes at least some of the objects included in the treatment space, such as the dentist, dental assistant, patient, and medical treatment equipment 1. It may be a space containing objects. Image data (hereinafter also referred to as “overall image data”) including photographic information obtained by the overall camera 53 is acquired by the control device 100. Various settings such as the shutter speed in the general camera 53 are adjusted in advance to such an extent that the control device 100 can recognize the dentist's actions, the dental assistant's actions, the patient's actions, the actions of the medical device 1, and the like. Note that the general camera 53 may be installed at any location as long as it can take at least an image of the medical treatment space including the medical treatment device 1.

Each of the above-mentioned tray image data, patient image data, and whole image data is one embodiment of "image-related data," and these data will also be referred to as "image-related data" below. Note that the phrase "image-related data" may mean all of tray image data, patient image data, and overall image data; however, it may refer to at least any of tray image data, patient image data, and overall image data. It can also mean one of the following.

Further, as will be described in detail later, the control device 100 detects the presence or absence of an object such as a medical instrument shown in the captured image and its type based on the tray image data and patient image data. The control device 100 detects the positions of people, such as a dentist, a dental assistant, and a patient, shown in the captured image based on the overall image data. The "image related data" may include the detection results by the control device 100 described above. That is, "image-related data" may include tray image data, patient image data, and whole image data themselves, or may include detection results obtained by control device 100 based on these data. .

The control device 100 is equipped with a computer (computation device 102 described later) that estimates control data for the medical device 1 based on various acquired data. Specifically, the control device 100 acquires image-related data from at least one of the tray camera 51, patient camera 52, and overall camera 53 attached to the medical treatment device 1. Furthermore, the control device 100 acquires data related to the medical treatment device 1 (hereinafter also referred to as “device related data”) acquired by the medical treatment device 1. The control device 100 estimates control data for the medical treatment device 1 based on at least one of the acquired image-related data and device-related data.

“Device-related data” includes at least one of the chair 11, basin unit 12, lighting device 19, instrument control device 21, display control device 22, medical instrument 15, foot controller 16, display 17, and operation panel 18 that the medical device 1 is equipped with. The control data includes at least one past and present control data for one of the above. For example, "device related data" includes the chair 11, basin unit 12, lighting device 19, instrument control device 21, display control device 22, medical instrument 15, foot controller 16, display 17, and operation panel 18 included in the medical device 1. Contains log data showing the operation and control history of each. Note that the "device-related data" is not limited to historical data of operations and controls in various devices such as log data, but may include real-time data of operations and controls in various devices. Furthermore, "device related data" may include data regarding the current status of various devices. For example, the "apparatus related data" may include data for specifying the current state (posture, position, etc.) of the chair 11.

Since the above-mentioned "image-related data" and "apparatus-related data" are data related to a series of medical treatments, these data are also referred to as "medical care data." "Medical care data" includes at least one of "image-related data" and "device-related data."

<Medical content>
Examples of treatment in medical practice include caries treatment, root canal treatment, plaque removal, implants, orthodontics, and whitening. Any cosmetic treatment that maintains the beauty and health of a patient's teeth is included in the scope of medical treatment.

Actions (treatments) that a surgeon performs to cure a patient's dental disease and actions (treatments) that a surgeon performs to maintain the beauty and health of the patient's teeth are also called treatments. Consists of a combination of treatments. Examples of procedures include examination, cutting, filing, root canal cleaning, drying, temporary sealing, impressions, scaling, prosthetic modifications, periodontal pocket measurements, suction, and tooth extraction; The content of treatment includes the content of treatment performed to cure a dental disease, and the content of treatment performed by an operator to maintain the beauty and health of the patient's teeth.

Let us take root canal treatment as an example of medical treatment. Root canal treatment is a treatment that is necessary when the pulp (nerves, blood vessels, etc.) in the root of the tooth becomes inflamed or infected due to the progression of caries. The treatment involves enlarging the area by cutting it, cleaning and disinfecting it, and placing a filling inside the root of the tooth to prevent reinfection. Root canal treatment consists of examination, pulp extraction, root canal length measurement/enlargement, cleaning/disinfection, root canal filling, and filling/covering, and the surgeon performs each of these procedures on the patient in order. Root canal treatment can be performed by performing this procedure.

The examination is a procedure that involves the surgeon interviewing the patient and inspecting the patient's oral cavity to identify the patient's dental condition and etiology and establishing a treatment plan. Pulp extraction is a procedure that involves removing the damaged pulp during root canal treatment. Root canal length measurement and enlargement is a procedure that involves measuring the position of the hollow root apex after removing the pulp, and cutting and enlarging the root canal up to the root apex. Cleaning and disinfection is a procedure that involves deep cleaning and disinfection of the hollow root canal. Root canal filling is a procedure that involves filling the root canal with a special drug to prevent bacteria from entering the root canal after cleaning and disinfection. Filling/covering is a procedure that involves filling the root canal with a rubber-like filling, creating a metal or fiber base on top of it, and then placing a crown on the base. Note that the content and number of treatments included in the root canal treatment described above are only an example, and the root canal treatment may include other treatments or some of the above-mentioned treatments may be omitted. .

<Image taken by tray camera>
FIG. 2 is a diagram for explaining an example of an image taken by the tray camera 51. As shown in FIG. 2, one or more medical instruments placed on the tray 30 are photographed by the tray camera 51.

For example, in the example shown in FIG. 2, a rubber dam moisture-proof set 301, a rubber dam sheet 302, a root canal length measuring device 303, a bur set 304, a file (reamer) 305, a mouth corner counter electrode 306, a fur clip 307, a broach 308, and a cleaning needle 309 , a cleaning syringe 310, a temporary sealant filler 311, a cleanser 312, a turbine 313, tweezers 314, a vacuum 315, a mirror 316, an excavator 317, a deep needle 318, and a root canal material injector 319. The photographed image shows how the medical instruments are placed on the tray 30.

During medical treatment, the operator selects and uses a desired medical instrument from among the plurality of medical instruments placed on the tray 30. Not displayed. For example, when the mirror 316 is used during medical treatment, since the mirror 316 does not exist from above the tray 30, the mirror 316 is not reflected in the image taken by the tray camera 51.

The control device 100 analyzes the tray image data including the photographed image acquired by the tray camera 51 as shown in FIG. etc. can be detected.

<Image taken by patient camera>
FIG. 3 is a diagram for explaining an example of an image taken by the patient camera 52. As shown in FIG. 3, the patient camera 52 photographs an area including at least the inside of the patient's oral cavity.

For example, in the photographed image shown in FIG. 3, the photographed image shows the position of medical instruments (in this example, tweezers 314 and mirror 316) on the teeth near the lower lip of the right cheek in the patient's oral cavity. .

During medical treatment, dental treatment using medical instruments will be performed inside the patient's oral cavity, but as shown in FIG. The medical equipment that will be used will be displayed.

The control device 100 can detect the position of the medical instrument in the patient's oral cavity by analyzing patient image data, including images captured by the patient camera 52 as shown in FIG. 3, using image recognition or the like. .

<Images taken by the overall camera>
FIG. 4 is a diagram for explaining an example of an image taken by the general camera 53. As shown in FIG. 4, the entire camera 53 photographs an area including at least the medical treatment space.

For example, in the photographed image shown in FIG. and medical instruments on the tray 30) are shown in the photographed image.

During medical treatment, the dentist performs dental treatment using medical instruments, but as shown in FIG. The patient's behavior, the patient's behavior, the state of the medical device 1, etc. are displayed.

The control device 100 analyzes the overall image data including the photographed image acquired by the overall camera 53 as shown in FIG. Y coordinate) can be detected. Further, as will be described later, the control device 100 can detect the positions (for example, Z coordinates) of the operator and patient during medical treatment by irradiating light such as infrared light from the general camera 53.

<Internal configuration of control system>
FIG. 5 is a block diagram showing the internal configuration of control system 1000 according to this embodiment. As shown in FIG. 5, the control system 1000 includes a plurality of cameras (a tray camera 51, a patient camera 52, and an overall camera 53) and a medical treatment device 1.

The medical treatment device 1 includes a chair 11, an instrument control device 21, a display control device 22, a sound control device 32, a basin unit 12, and a control device 100.

The chair 11 includes a headrest 11a, a backrest 11b, a seat 11c, and a footrest 11d, each of which is driven under the control of the chair controller 111. Specifically, the chair control unit 111 transmits control data based on the operator's operation received by the foot controller 16 or the operation panel 18, or control data from the control device 100 via CAN (Controller Area Network) communication. When received, based on the control data, the seat 11c is raised or lowered, and the headrest 11a, backrest 11b, and footrest 11d are moved vertically or horizontally with respect to the seat 11c. . When the headrest 11a, the backrest 11b, and the footrest 11d are positioned perpendicularly to the seat 11c, the patient sitting on the chair 11 assumes a sitting posture. When the headrest 11a, the backrest 11b, and the footrest 11d are positioned horizontally with respect to the seat 11c, the patient sitting on the chair 11 assumes a supine posture. In this way, the chair control unit 111 changes the posture of the chair 11 by driving the headrest 11a, the backrest 11b, the seat seat 11c, and the footrest 11d.

The instrument control device 21 includes an instrument control section 211. When the instrument control unit 211 receives control data based on the operator's operation accepted by the foot controller 16 or the operation panel 18, or control data from the control device 100 via CAN communication, the instrument control unit 211 performs the following based on the control data: Controls the drive or settings of the medical instrument 15. For example, when the operator presses a switch for driving the air turbine handpiece in the foot controller 16, the instrument control device 21 rotates the cutting tool held in the head of the air turbine handpiece. For example, when the control device 100 outputs control data for controlling the rotational direction, rotational speed, etc. of the air handpiece, the instrument control device 21 outputs control data for controlling the rotational direction, rotational speed, etc. of the air turbine handpiece based on the control data. Set.

Display control device 22 includes a display control section 221 and a panel control section 222. When the display control unit 221 receives control data from the control device 100 via CAN communication, the display control unit 221 controls the display 17 based on the control data. When the panel control unit 222 receives control data based on the control of the control device 100 via CAN communication, the panel control unit 222 controls the operation panel 18 based on the control data.

The sound control device 32 includes a sound control section 321. When the sound control unit 321 receives control data from the control device 100 via CAN communication, it controls the speaker 35 based on the control data.

Basin unit 12 includes a basin control section 121 and a lighting control section 122. Upon receiving control data from the control device 100 via CAN communication, the basin control unit 121 controls water supply and drainage in the basin unit 12 based on the control data. When the lighting control unit 122 receives control data from the control device 100 via CAN communication, it controls lighting and extinguishing of the lighting device 19 based on the control data.

Each of the chair control section 111, appliance control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 described above is a CPU mounted on a board (not shown). (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). Note that each of the chair control section 111, instrument control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 is installed in the medical device 1 in advance. Alternatively, it may be modularized so that it can be attached to the medical device 1 as an option.

Each of the control device 100, chair control section 111, appliance control section 211, display control section 221, panel control section 222, sound control section 321, basin control section 121, and lighting control section 122 communicates with each other by CAN communication. . In CAN communication, device-related data including log data in each control unit is converted into a communication packet, and the control units communicate with each other. Note that in communication between control units, as long as the device-related data is transmitted and received, the device-related data does not necessarily need to be converted into a communication packet.

Further, the basin unit 12 includes a storage section 123 that stores device-related data including log data within the medical device 1 .

The storage unit 123 communicates with each of the chair control unit 111, appliance control unit 211, display control unit 221, panel control unit 222, sound control unit 321, basin control unit 121, and lighting control unit 122 by CAN communication. By doing so, device-related data is collected and accumulated from each control unit. The storage unit 123 may be configured with a memory such as a ROM or RAM mounted on a substrate (not shown), or may be configured with a nonvolatile storage medium such as a memory card.

The control device 100 controls the recording of medical data regarding a series of medical treatments, and also controls the medical treatment device 1 . The control device 100 includes a communication device 101, a calculation device 102, and a recording device 103.

The communication device 101 transmits and receives data by communicating with each of the tray camera 51, patient camera 52, and general camera 53. The communication device 101 acquires image-related data from each camera by communicating with each camera through wired or wireless LAN (Local Area Network) communication. Note that the communication device 101 may acquire image-related data from each camera in other formats. For example, the communication device 101 may acquire image-related data temporarily stored in a nonvolatile storage medium such as a memory card taken out from each camera. The communication device 101 acquires device-related data accumulated in the storage section 123 of the basin unit 12 by communicating with the basin unit 12 through wired or wireless LAN communication. Note that the basin unit 12 is not limited to collecting and accumulating medical care-related data, and other components such as the chair 11, the instrument control device 21, the display control device 22, and the sound control device 32 may accumulate device-related data. Alternatively, the control device 100 may acquire device-related data from these other configurations.

The computing device 102 is a computing entity that executes various processes (for example, recording control processing and learning processing that will be described later) by executing various programs (for example, a control program 140 and an estimation program 141 that will be described later). . Computing device 102 is one embodiment of a "computer." The arithmetic device 102 includes, for example, a CPU, an FPGA (Field-Programmable Gate Array), and a GPU (Graphics Processing Unit).

Note that the arithmetic device 102 may be composed of at least one of a CPU, an FPGA, and a GPU, or may be composed of a CPU and an FPGA, an FPGA and a GPU, a CPU and a GPU, or a CPU, an FPGA, and a GPU. may be done. Further, the arithmetic device 102 may be configured with a processing circuitry.

The recording device 103 is an embodiment of a "recording section". The recording device 103 provides a recording area for storing program codes, work memory, etc. when the arithmetic device 102 executes an arbitrary program. The recording device 103 may be configured with a volatile memory device such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory), or a non-volatile memory device such as a hard disk or SSD (Solid State Drive). or The recording device 103 records image-related data (tray image data, patient image data, whole image data) acquired from each camera (tray camera 51, patient camera 52, whole body camera 53), and also records the image related data (tray image data, patient image data, whole image data) acquired from each camera (tray camera 51, patient camera 52, whole body camera 53), and also records the image related data (tray image data, patient image data, whole image data) acquired from each camera (tray camera 51, patient camera 52, whole body camera 53), and also records the image related data (tray image data, patient image data, whole image data) acquired from each camera (tray camera 51, patient camera 52, whole body camera 53). 123 is recorded. That is, the recording device 103 records clinical data (image-related data, device-related data) regarding a series of medical treatments.

The recording device 103 is not limited to that included in the control device 100. For example, an in-hospital server communicably connected to the medical device 1 may include the recording device 103, or an external server installed outside the medical treatment space where the medical device 1 is installed may include the recording device 103. Good too. Furthermore, the recording device 103 may exist in a cloud computing mode in which each of the plurality of control devices 100 included in each of the plurality of medical treatment devices 1 can communicate with each other. In this way, image-related data and device-related data acquired from a plurality of medical devices 1 can be uniformly stored and managed by the recording device 103.

The recording device 103 stores an estimation model 161, a control program 140, an estimation program 141, and an OS (Operating System) 142.

The estimation model 161 is used to estimate control data of the medical device 1 based on at least one of image-related data and device-related data included in the medical care data. The estimation model 161 is optimized (adjusted) by performing machine learning based on at least one of the image-related data and the device-related data, and can improve the estimation accuracy of the control data of the medical device 1.

Note that the processing in which the arithmetic device 102 of the control device 100 controls recording of medical data by the recording device 103 is also referred to as "recording control processing," and the arithmetic device 102 of the control device 100 learns the estimation model 161 using medical data. The process is also referred to as "learning process." Furthermore, the estimated model 161 that has been optimized through the learning process is also particularly referred to as a "trained model." That is, in this embodiment, while the pre-learning estimation model 161 and the trained estimation model 161 are collectively referred to as the "estimation model", the trained estimation model 161 is also particularly referred to as the "trained model". to be called.

The control program 140 includes a program for the arithmetic device 102 to execute recording control processing. The estimation program 141 includes a program for the arithmetic device 102 to execute a learning process.

The server device 200 includes a recording device 203. The recording device 203, like the recording device 103, is an embodiment of a “recording unit”. The recording device 203 provides a recording area for storing various data. The recording device 203 is, for example, configured with a volatile memory device such as DRAM or SRAM, or a nonvolatile memory device such as a hard disk or SSD. The recording device 203 acquires medical data (image-related data, device-related data) regarding a series of medical treatments from the recording device 103, and records the acquired medical data.

In this embodiment, the computing device 102 executes recording control processing to cause the recording device 103 to record medical data. Further, the computing device 102 transmits the medical data recorded in the recording device 103 to the server device 200 using the communication device 101. The server device 200 acquires the medical data transmitted from the control device 100 and causes the recording device 203 to record the acquired medical data. Therefore, when the computing device 102 executes the recording control process, it also controls the recording of medical data by the recording device 203.

Note that the recording device 203 acquires image-related data (tray image data, patient image data, whole image data) from each camera (tray camera 51, patient camera 52, whole camera 53) without going through the recording device 103. Alternatively, the device-related data may be acquired from the storage section 123 of the basin unit 12. In this case, the computing device 102 may control the recording of medical data by the recording device 203 of the server device 200 by executing recording control processing. Then, the computing device 102 may acquire the medical care data recorded in the recording device 203 of the server device 200 and execute the learning process of the estimation model 161 using the acquired medical data. Note that the recording device 203, which is an embodiment of the “recording unit”, may be provided in the control device 100 of the medical treatment device 1. In this case, the recording device 103 of the control device 100 may have the function of the recording device 203.

<Position data>
With reference to FIGS. 6 to 8, position data of the operator and the patient acquired by the control device 100 based on the imaging information obtained by the general camera 53 will be described. FIG. 6 is a diagram for explaining an example of position data acquired by control device 100 according to the present embodiment. FIG. 7 is a diagram for explaining an example of detection points of position data. FIG. 8 is a diagram for explaining an example of position coordinates corresponding to position data.

In the control device 100, the arithmetic device 102 generates position data for specifying the positions of persons included in the photographed image, such as the surgeon and patient during treatment, based on the overall image data acquired from the recording device 103 or the recording device 203. The generated position data is recorded on the recording device 103.

Specifically, as shown in FIG. 6, the recording device 103 collectively records position data for each person, such as person A, person B, and person C. For example, person A corresponds to a dentist, person B corresponds to a dental assistant, and person C corresponds to a patient. The recording device 103 records the position data generated by the arithmetic device 102 in association with time data (time stamp). The recording device 103 records the position data newly generated by the calculation device 102 when the position data newly generated by the calculation device 102 is different from the recorded position data. That is, in this embodiment, the recording device 103 does not record new position data unless the position of the person changes, but records new position data on the condition that the position of the person changes. Note that the recording device 103 may record the position data generated by the arithmetic device 102 in association with a time stamp at predetermined time intervals (for example, 1 second) regardless of whether the position data changes or not. .

The position data includes three-dimensional (X, Y, Z) position coordinates for each detection point determined for the person to be detected. As shown in FIGS. 6 and 7, for example, the detection points include the right wrist, right elbow, right shoulder, head, left shoulder, left elbow, left wrist, right ear, and left ear of the person (dentist 3 in this example). , including right eye, left eye, nose, mouth, hip, right knee, left knee, right ankle, and left ankle. Note that the control device 100 may acquire position data for points other than the detection points shown in FIG. 6, or may acquire position data only for some of the detection points shown in FIG. .

Note that when the control device 100 cannot specify some of the predetermined detection points, the control device 100 may cause the recording device 103 to record the position data corresponding to the previously specified detection points as the current position data. For example, if a person to be photographed enters a blind spot of the general camera 53, the control device 100 may not be able to identify some detection points. The position data corresponding to some of the detection points that are missing this time may be complemented using the position data corresponding to the detection points specified before entering the blind spot of 53.

Position data in the case where the overall camera 53 is composed of a ToF type camera will be explained. For example, the positional coordinates corresponding to the positional data set the origin of the X and Y coordinates to the upper left corner of the photographed image obtained by the general camera 53 as shown in FIG. Furthermore, as shown in FIG. 8, the position coordinates corresponding to the position data have the origin of the Z coordinate as the position where the general camera 53 is placed.

When the arithmetic device 102 acquires a captured image as shown in FIG. 7 from the general camera 53, it identifies three people, person A, person B, and person C, by image recognition. The computing device 102 identifies detection points such as the right wrist and right elbow for each person by image recognition, and identifies the position coordinates of each identified detection point. Each person present in the medical treatment space, such as a dentist, a dental assistant, and a patient, has a roughly determined location (home position). Therefore, the computing device 102 can identify whether each person is a dentist, a dental assistant, or a patient based on the position data of each of the plurality of people shown in the captured image of the general camera 53. good. Further, each person present in the medical treatment space, such as a dentist, a dental assistant, and a patient, generally wears a certain color or shape. For example, a dentist usually wears a white coat. For this reason, the computing device 102 may identify whether each person is a dentist, a dental assistant, or a patient based on the clothes of each of the plurality of people shown in the captured image of the general camera 53. .

For example, focusing on person A (dentist 3), the computing device 102 calculates P1 (right wrist), P2 (right elbow), P3 (right shoulder), P4 (head), P5 (left shoulder), P6 (left elbow). ), and P7 (left wrist) are identified, and the positional information of each of the detection points P1 to P7 is identified. For example, as shown in FIG. 8, the calculation device 102 specifies X1 as the X coordinate and Y1 as the Y coordinate, based on the captured image, regarding the position coordinates of P1. Further, the calculation device 102 identifies Z1 as the Z coordinate based on the reflected light of the light emitted from the origin of the Z coordinate toward P1 by the general camera 53. Specifically, the calculation device 102 calculates the distance L between the origin of the Z coordinate and P1 based on the time from when light is irradiated by the general camera 53 until the reflected light of that light returns. . Then, the calculation device 102 draws a perpendicular line from P1 to the Z axis, and uses trigonometric functions based on the length of the perpendicular line and the distance L to specify Z1 as the Z coordinate.

The arithmetic device 102 causes the recording device 103 to record the position coordinates of P1 identified as described above as position data of P1. At this time, if the newly acquired position data of P1 is different from the position data of P1 already recorded in the recording device 103, the calculation device 102 causes the recording device 103 to record the newly acquired position data of P1 together with a time stamp. . On the other hand, if the newly acquired position data of P1 is the same as the position data of P1 already recorded in the recording device 103, the arithmetic device 102 does not cause the recording device 103 to record the newly acquired position data of P1.

In this way, the computing device 102 does not cause the recording device 103 to record new position data each time the person's position is detected, but records new position data only when the person's position data changes. Since the data is recorded by the recording device 103, an increase in the amount of data recorded by the recording device 103 can be suppressed as much as possible.

In this way, the control device 100 acquires position data of each detection point for specifying the positions of the surgeon and patient during treatment, based on the imaging information acquired from the general camera 53. The control device 100 associates the acquired position data with time data (time stamp), and causes the recording device 103 to accumulate and record the data as the entire image data. The entire image data including the position data recorded on the recording device 103 is transmitted to the server device 200 and recorded by the recording device 203.

As a result, the designer of the estimation model 161 can perform machine learning on one estimation model 161 using position data, and then use the same position data recorded in the recording device 103 or the recording device 203 to perform the same estimation. When subjecting the model 161 to machine learning again, the time data can be used as a clue to specify the position data used during past machine learning. Furthermore, when performing machine learning using the same clinical data among multiple estimation models 161, the designer of the estimation model 161 uses the time data as a clue and uses the clinical data used during machine learning for one estimation model 161. can be specified during machine learning of other estimation models 161.

Furthermore, by recording and retaining the position data of a person in a series of medical treatments in the recording device 103 or the recording device 203, when the estimation model 161 is subjected to machine learning again, it is possible to avoid performing the same medical treatment again. , it is possible to reproduce the medical treatment of the time. Furthermore, in the future, if the accuracy of position detection at each detection point improves, it will be difficult to obtain the same position data even if a series of medical treatments similar to those at the time were performed; Even in such a case, by recording and preserving the position data of the person in the series of medical treatments at that time in the recording device 103 or the recording device 203, when the estimation model 161 is subjected to machine learning again, the medical treatment at that time can be used. Can be reproduced.

Note that the location data of the person at the time of medical treatment is not limited to the data recorded in the recording device 103 or the recording device 203, but also the raw video data used to specify the location data of the person (for example, tray image data). , patient image data, whole image data, etc.) may be recorded and left in the recording device 103 or the recording device 203 along with the person's position data. In this way, if the accuracy of the position detection section 157 improves dramatically in the future, for example by repeating learning or developing a new AI algorithm, then the highly accurate position detection section 157 can 157 allows raw video data to be analyzed. As a result, it is possible to obtain position data that is even more accurate than when raw video data was acquired, and it is possible to record and retain this highly accurate position data in the recording device 103 or the recording device 203. can.

Further, the control device 100 can identify the motions of a person's hands, arms, head, waist, legs, etc. by virtually connecting each detection point with a straight line or the like.

As a result, the control device 100 can identify the person's movements more accurately by detecting the positions of detection points such as joints, eyes, nose, mouth, ears, and head where the movements of the person are likely to be reflected. be able to.

Note that the method for specifying the position coordinates of the detection point described above is an example, and the control device 100 may specify the position coordinates of the detection point using other methods. Further, the control device 100 may specify the position coordinates of not only a person involved in a series of medical treatments but also a key item involved in a series of medical treatments. In this way, the control device 100 can perform machine learning on the estimation model 161 using the position coordinates of the identified key item.

In the present embodiment, the calculation device 102 calculates the Z coordinate based on the photographing information acquired from the general camera 53, and then records the Z coordinate together with the X coordinate and the Y coordinate in the recording device 103 or the recording device 203. Met. However, instead of the Z coordinate, the calculation device 102 may cause the recording device 103 or the recording device 203 to record a depth frame, which is a frame of the distance to the object shown in the photographed image, together with the image frame. In this way, even after a series of medical treatments have been completed, the Z coordinate can be calculated by analyzing the depth frame and image frame recorded on the recording device 103 or the recording device 203.

<Functional configuration of control device>
[Functional configuration of the control device at the recording stage]
FIG. 9 is a block diagram showing the functional configuration of the control device 100 according to the present embodiment at the stage of recording medical data. As shown in FIG. 9, the control device 100 includes an input section 150, a communication section 110, a recording section 130, and a control section 120 as main functional sections. Note that the input unit 150 and the communication unit 110 are functional units of the communication device 101 , the recording unit 130 is a functional unit of the recording device 103 , and the control unit 120 is a functional unit of the computing device 102 .

The input unit 150 includes image-related data (tray image data, patient image data, and overall image data) captured by each of the tray camera 51, patient camera 52, and overall camera 53, and device-related data obtained from the medical treatment device 1. are input in chronological order. Note that depth frames, which are frames of the distance to the object shown in the captured image, may be input to the input unit 150 in chronological order.

The recording unit 130 records medical data (image-related data, device-related data) input from the input unit 150 under the control of the control unit 120.

The control unit 120 controls the recording unit 130 to cause the recording unit 130 to record the medical data (image-related data, device-related data) input from the input unit 150.

Specifically, the control unit 120 causes the recording unit 130 to start recording the medical care data regarding the series of medical treatments input from the input unit 150 when the start condition for the series of medical treatments is satisfied. The "start condition" is a condition that, when met, is assumed to start a series of medical treatments.

For example, the start condition includes that the height of the seat 11c of the chair 11 has reached a predetermined height or higher. When a series of medical treatments starts, it is assumed that the patient is sitting on the chair 11 and the seat 11c is raised. Therefore, the control unit 120 determines whether the height of the seat sheet 11c is equal to or higher than a predetermined height based on at least one of the image-related data and the device-related data, and When it is determined that the height is equal to or higher than a predetermined height, the recording unit 130 is caused to start recording medical data. For example, the control unit 120 specifies the operation of the seat 11c based on the device-related data (log data) acquired from the chair control unit 111, so that the height of the seat 11c becomes equal to or higher than a predetermined height. It can be determined whether or not. Alternatively, the control unit 120 determines whether the height of the seat seat 11c is equal to or higher than a predetermined height by specifying the movement of the seat seat 11c based on the overall image data acquired from the overall camera 53. can do.

The start condition includes that the angle of the backrest 11b of the chair 11 (for example, the angle with respect to the vertical direction) becomes equal to or greater than a predetermined angle. When a series of medical treatments starts, it is assumed that the patient is sitting on the chair 11 and the backrest 11b is tilted backward so that the patient is in a backward leaning posture. Therefore, the control unit 120 determines whether the angle of the backrest 11b is greater than or equal to a predetermined angle based on at least one of the image-related data and the device-related data, and determines whether the angle of the backrest 11b is greater than or equal to the predetermined angle. If it is determined that this has occurred, the recording unit 130 is caused to start recording the medical data. For example, the control unit 120 determines whether the angle of the backrest 11b is equal to or greater than a predetermined angle by specifying the movement of the backrest 11b based on device-related data (log data) acquired from the chair control unit 111. be able to. Alternatively, the control unit 120 can determine whether the angle of the backrest 11b is equal to or greater than a predetermined angle by specifying the movement of the backrest 11b based on the overall image data acquired from the overall camera 53.

The start condition includes that the driving of the medical instrument 15 is turned on. When a series of medical treatments starts, it is assumed that the driving of the medical instrument 15 is turned on. Therefore, the control unit 120 determines whether or not the drive of the medical instrument 15 is turned on based on at least one of the image-related data and the device-related data, and determines whether the drive of the medical instrument 15 is turned on. If determined, the recording unit 130 is caused to start recording the medical data. For example, the control unit 120 determines whether or not the drive of the medical instrument 15 is turned on by specifying the state of the medical instrument 15 based on the device-related data (log data) acquired from the instrument control unit 211. be able to. Alternatively, the control unit 120 determines whether the driving of the medical instrument 15 is turned on by specifying the state of the medical instrument 15 based on the overall image data obtained from the overall camera 53 or the patient image data obtained from the patient camera 52. It can be determined whether or not.

The start condition includes that the lighting device 19 is turned on. When a series of medical treatments starts, it is assumed that the lighting device 19 is turned on. Therefore, the control unit 120 determines whether or not the lighting device 19 is powered on based on at least one of image-related data and device-related data, and determines whether the lighting device 19 is powered on. If determined, the recording unit 130 is caused to start recording the medical data. For example, the control unit 120 determines whether or not the power of the lighting device 19 is turned on by specifying the state of the lighting device 19 based on device-related data (log data) acquired from the lighting control unit 122. be able to. Alternatively, the control unit 120 can determine whether the lighting device 19 is powered on by specifying the state of the lighting device 19 based on the overall image data acquired from the overall camera 53.

The start condition includes detection of a person (for example, a patient) sitting on the chair 11. When a series of medical treatments begins, it is assumed that a person will be seated on the chair 11. Therefore, the control unit 120 determines whether or not a person is seated on the chair 11 based on the image-related data, and when it is determined that a person is seated on the chair 11, the control unit 120 records medical data in the recording unit 130. Let it start. For example, the control unit 120 can determine whether a person is seated on the chair 11 by specifying the movement of the person in the medical treatment space based on the overall image data acquired from the overall camera 53. Note that the seat 11c of the chair 11 may be provided with a seating sensor that detects whether a person is sitting.

The start condition includes detection of a medical instrument in the medical treatment space. When a series of medical treatments begins, it is assumed that medical instruments will be used. Therefore, the control unit 120 determines whether or not a medical instrument exists in the medical treatment space based on at least one of image-related data and device-related data, and when it is determined that a medical instrument exists in the medical treatment space. Then, the recording unit 130 is caused to start recording the medical data. For example, the control unit 120 identifies the presence or absence of medical instruments based on the overall image data obtained from the overall camera 53, the patient image data obtained from the patient camera 52, or the tray image data obtained from the tray camera 51. It is possible to determine whether medical instruments are present in the medical treatment space. More specifically, the control unit 120 determines whether or not a medical instrument has been placed on the tray 30 based on the tray image data acquired from the tray camera 51, and determines that a medical instrument has been placed on the tray 30. In this case, the recording unit 130 is caused to start recording the medical data.

The starting condition includes detection of a person (eg, patient, dentist, dental assistant) in the treatment space. When a series of medical treatments begins, it is assumed that people will enter the medical treatment space from outside. For this reason, the control unit 120 determines whether or not a person exists in the medical treatment space based on the image-related data, and starts recording medical data in the recording unit 130 when it is determined that a person exists in the medical treatment space. let For example, the control unit 120 can determine whether a person is present in the medical treatment space by specifying the presence or absence of a person in the medical treatment space based on the overall image data acquired from the general camera 53.

Further, if the chair 11 includes a bed in which a portion corresponding to the backrest 11b and a portion corresponding to the seat sheet 11c are integrated, the starting condition is that the height of the bed of the chair 11 is equal to or higher than a predetermined height. Including what has become. When a series of medical treatments begins, it is assumed that the patient is lying on the bed and the bed is raised. Therefore, the control unit 120 determines whether the height of the bed is equal to or higher than the predetermined height based on at least one of the image-related data and the device-related data, and determines whether the height of the bed is equal to or higher than the predetermined height. If it is determined that the above has been reached, the recording unit 130 is caused to start recording the medical data. For example, the control unit 120 determines whether the height of the bed is equal to or higher than a predetermined height by specifying the movement of the bed based on the device-related data (log data) acquired from the chair control unit 111. be able to. Alternatively, the control unit 120 can determine whether the height of the bed is equal to or higher than a predetermined height by specifying the movement of the bed based on the overall image data acquired from the overall camera 53.

A plurality of the above-mentioned starting conditions may be combined. For example, even if the height of the seat 11c of the chair 11 is above a predetermined height, the height of the seat 11c may be set above the predetermined height simply due to maintenance of the chair 11. is assumed. Therefore, when the height of the seat 11c of the chair 11 is equal to or higher than a predetermined height and it is detected that a person is sitting on the chair 11, the control unit 120 records medical data on the recording unit. It may start at 130. Alternatively, even if the angle of the backrest 11b of the chair 11 is at least a predetermined angle, it is conceivable that the angle of the backrest 11b is at least the predetermined angle simply for maintenance of the chair 11. Therefore, when the angle of the backrest 11b of the chair 11 is equal to or greater than a predetermined angle and it is detected that a person is sitting on the chair 11, the control unit 120 causes the recording unit 130 to start recording medical data. You can.

In this way, the start conditions are that the height of the seat 11c of the chair 11 has become a predetermined height or more, that the height of the bed of the chair 11 has become a predetermined height or more, and that the backrest 11b of the chair 11 has reached a predetermined height. The angle of the medical instrument 15 is turned on, the lighting device 19 is turned on, a person is detected sitting on the chair 11, and the medical treatment space This includes at least one of the following: that the medical instrument 15 is detected in the medical treatment space, and that a person is detected in the medical treatment space.

Note that the medical device 1 (control device 100) may include a recording start switch for starting recording of medical data. The start condition may include the user operating a recording start switch. The medical device 1 (control device 100) may include a gesture sensor that detects a user's gesture (for example, hand gesture). The start condition may include that the gesture sensor detects a gesture by the user. The medical device 1 (control device 100) may include a voice sensor that detects the user's voice. The starting condition may include the audio sensor detecting the user's voice.

The start condition may include that image-related data has been input. That is, the control device 100 may set the recording flag to ON when image-related data is input or when image recognition is performed based on the input image-related data.

The control unit 120 causes the recording unit 130 to end the recording of the medical care data regarding the series of medical treatments input from the input unit 150 when the conditions for ending the series of medical treatments are satisfied. The "termination condition" is a condition that, when met, indicates the end of a series of medical treatments.

For example, the termination condition includes that the height of the seat 11c of the chair 11 has become less than a predetermined height. At the end of a series of medical treatments, it is assumed that the seat 11c is lowered while the patient is sitting on the chair 11. Therefore, the control unit 120 determines whether the height of the seat sheet 11c is less than a predetermined height based on at least one of the image-related data and the device-related data, and When it is determined that the height has become less than the predetermined height, the recording unit 130 is caused to finish recording the medical data. For example, the control unit 120 specifies the operation of the seat 11c based on the device-related data (log data) acquired from the chair control unit 111, so that the height of the seat 11c becomes less than a predetermined height. It can be determined whether or not. Alternatively, the control unit 120 determines whether the height of the seat seat 11c becomes less than a predetermined height by specifying the movement of the seat seat 11c based on the overall image data acquired from the overall camera 53. can do.

The termination condition includes that the angle of the backrest 11b of the chair 11 (for example, the angle with respect to the vertical direction) has become less than a predetermined angle. At the end of a series of medical treatments, it is assumed that the patient is sitting on the chair 11 and the backrest 11b is tilted forward so that the patient is in a forward leaning posture. Therefore, the control unit 120 determines whether the angle of the backrest 11b becomes less than a predetermined angle based on at least one of image-related data and device-related data, and determines whether the angle of the backrest 11b becomes less than a predetermined angle. If it is determined that this has occurred, the recording unit 130 is caused to finish recording the medical data. For example, the control unit 120 determines whether the angle of the backrest 11b is less than a predetermined angle by specifying the movement of the backrest 11b based on the device-related data (log data) acquired from the chair control unit 111. be able to. Alternatively, the control unit 120 can determine whether the angle of the backrest 11b is less than a predetermined angle by specifying the movement of the backrest 11b based on the overall image data acquired from the overall camera 53.

The termination condition includes that the driving of the medical instrument 15 is turned off. When a series of medical treatments is completed, it is assumed that the driving of the medical instrument 15 is turned off. Therefore, the control unit 120 determines whether or not the drive of the medical instrument 15 is turned off based on at least one of the image-related data and the device-related data, and determines whether the drive of the medical instrument 15 is turned off. If determined, the recording unit 130 is caused to finish recording the medical data. For example, the control unit 120 determines whether or not the driving of the medical instrument 15 is turned off by specifying the state of the medical instrument 15 based on the device-related data (log data) acquired from the instrument control unit 211. be able to. Alternatively, the control unit 120 determines whether the driving of the medical instrument 15 is turned off by specifying the state of the medical instrument 15 based on the overall image data obtained from the overall camera 53 or the patient image data obtained from the patient camera 52. It can be determined whether or not.

The termination condition includes that the lighting device 19 is powered off. When a series of medical treatments ends, it is assumed that the lighting device 19 will be powered off. Therefore, the control unit 120 determines whether or not the power of the lighting device 19 is turned off based on at least one of the image-related data and the device-related data, and determines whether the power of the lighting device 19 is turned off. If determined, the recording unit 130 is caused to finish recording the medical data. For example, the control unit 120 determines whether the power of the lighting device 19 is to be turned off by specifying the state of the lighting device 19 based on device-related data (log data) acquired from the lighting control unit 122. be able to. Alternatively, the control unit 120 can determine whether or not the power of the lighting device 19 is to be turned off by specifying the state of the lighting device 19 based on the overall image data acquired from the overall camera 53.

The termination condition includes detecting that a person (for example, a patient) has left the chair 11 . It is assumed that a person leaves the chair 11 when a series of medical treatments ends. Therefore, the control unit 120 determines whether or not a person has left the chair 11 based on the image-related data, and when it is determined that a person has left the chair 11, the control unit 120 records the medical data in the recording unit. It ends at 130. For example, the control unit 120 can determine whether a person has left the chair 11 by specifying the movement of the person in the medical treatment space based on the overall image data acquired from the overall camera 53.

The termination condition includes that the medical instrument 15 is not detected in the medical treatment space for a predetermined period of time or more. When a series of medical treatments ends, it is assumed that the medical instruments 15 are put away and are not used for a predetermined period of time. For this reason, the control unit 120 determines whether or not the medical instrument 15 exists in the medical treatment space for a predetermined time or more based on at least one of the image-related data and the device-related data, and If it is determined that the medical data does not exist for a predetermined period of time or longer, the recording unit 130 ends recording of the medical data. For example, the control unit 120 can identify the presence or absence of the medical instrument 15 based on the overall image data obtained from the overall camera 53, the patient image data obtained from the patient camera 52, or the tray image data obtained from the tray camera 51. , it can be determined whether the medical instrument 15 is present in the medical treatment space for a predetermined period of time or longer. More specifically, the control unit 120 determines whether the medical instrument has been removed from the tray 30 based on the tray image data acquired from the tray camera 51, and determines whether the medical instrument has been removed from the tray 30 for a predetermined period of time or more. If it is determined that the medical data has been recorded, the recording unit 130 is caused to finish recording the medical data.

Termination conditions include that no person (eg, patient, dentist, dental assistant) is detected in the treatment space. At the end of a series of medical treatments, it is assumed that people will leave the medical treatment space. Therefore, the control unit 120 determines whether or not there is a person in the medical treatment space based on the image-related data, and if it is determined that there is no person in the medical treatment space, the control unit 120 records the medical data in the recording unit 130. Terminate it. For example, the control unit 120 can determine whether a person is present in the medical treatment space by specifying the presence or absence of a person in the medical treatment space based on the overall image data acquired from the general camera 53.

Furthermore, if the chair 11 includes a bed in which a portion corresponding to the backrest 11b and a portion corresponding to the seat seat 11c are integrated, the end condition is that the height of the bed of the chair 11 is less than a predetermined height. Including what has become. At the end of a series of medical treatments, it is assumed that the bed will be lowered with the patient lying on the bed. Therefore, the control unit 120 determines whether the height of the bed is less than the predetermined height based on at least one of the image-related data and the device-related data, and determines whether the height of the bed becomes less than the predetermined height. If it is determined that the amount has become less than 1, the recording unit 130 is caused to finish recording the medical data. For example, the control unit 120 determines whether the height of the bed becomes less than a predetermined height by specifying the movement of the bed based on the device-related data (log data) acquired from the chair control unit 111. be able to. Alternatively, the control unit 120 can determine whether the height of the bed becomes less than a predetermined height by specifying the movement of the bed based on the overall image data acquired from the overall camera 53.

A plurality of the above termination conditions may be combined. For example, even if the height of the seat 11c of the chair 11 is less than a predetermined height, it is assumed that the patient may be temporarily away from the chair 11 simply for maintenance of the chair 11 or for taking X-rays. . Therefore, when the height of the seat 11c of the chair 11 is less than a predetermined height and no person or medical equipment is detected in the medical treatment space, the control unit 120 stops recording medical data in the recording unit 130. You may let them. Alternatively, even if the angle of the backrest 11b of the chair 11 is less than a predetermined angle, it may be assumed that the patient is temporarily away from the chair 11 simply for maintenance of the chair 11 or to take an X-ray. Therefore, when the height of the seat 11c of the chair 11 is less than a predetermined height and no person or medical equipment is detected in the medical treatment space, the control unit 120 stops recording medical data in the recording unit 130. You may let them.

In this way, the termination conditions are that the height of the seat 11c of the chair 11 has become less than the predetermined height, that the height of the bed of the chair 11 has become less than the predetermined height, and that the backrest 11b of the chair 11 has become less than the predetermined height. angle has become less than a predetermined angle, the drive of the medical instruments 15 has been turned off, the power of the lighting device 19 has been turned off, a person's separation from the chair 11 has been detected, This includes at least one of the fact that the medical instrument 15 is not detected in the space for a predetermined period of time or more, and that a person is not detected in the medical care space.

Note that the medical device 1 (control device 100) may include a recording end switch for ending recording of medical data. The start condition may include the user operating a recording end switch. The recording start switch may be the same as the recording end switch, or may be different. The medical device 1 (control device 100) may include a gesture sensor that detects a user's gesture (for example, hand gesture). The termination condition may include that the gesture sensor detects a gesture by the user. The medical device 1 (control device 100) may include a voice sensor that detects the user's voice. The termination condition may include the voice sensor detecting the user's voice.

As described above, since the control device 100 can perform machine learning on the estimation model 161 using the medical care data that started recording at the start timing of the series of medical treatments, it is independent of the series of medical treatments before the start of the series of medical treatments. It is possible to prevent the estimation model 161 from performing machine learning using useless data.

The control unit 120 causes the recording unit 130 to end the recording of the medical care data regarding the series of medical treatments input from the input unit 150 when the conditions for ending the series of medical treatments are satisfied. As a result, the control device 100 can perform machine learning on the estimation model 161 using the medical care data that has been recorded at the end timing of a series of medical treatments. It is possible to prevent the estimation model 161 from being subjected to machine learning using such data.

Furthermore, by starting recording medical data at the start of a series of medical treatments and finishing recording medical data at the end of a series of medical treatments, it is possible to avoid cases where there is insufficient medical data for the series of medical treatments actually performed. The medical data does not include unnecessary data unrelated to the series of medical treatments. Therefore, the control device 100 can reproduce the operations of the medical device 1 and the dentist's actions at the time when the series of medical treatments were performed, based on the medical care data recorded in the recording unit 130. The control device 100 can improve the accuracy of machine learning of the estimation model 161 by using medical care data recorded for each series of medical treatments.

The communication unit 110 outputs the medical data recorded in the recording unit 130 to the server device 200. Note that the control device 100 may output the medical data to the server device 200 while the medical data is being recorded to the recording unit 130 (during a series of medical treatments), or may output the medical data to the recording unit 130. The medical data may be output to the server device 200 after data recording is completed (after a series of medical treatments are completed).

The server device 200 includes a recording unit 230. The recording unit 230 is a functional unit of the recording device 203. The server device 200 causes the recording unit 230 to record the medical data output from the communication unit 110.

The medical care data recorded in at least one of the recording unit 130 and the recording unit 230 as described above is used as learning data for the estimation model 161.

FIG. 10 is a diagram for explaining the data structure of learning data. As shown in FIG. 10, the recording unit 130 and the recording unit 230 record one set of medical treatment data for each series of medical treatment.

Specifically, as shown in FIG. 10, each of the recording unit 130 and the recording unit 230 stores identification data assigned to a series of medical treatments, the start timing (start date and time) of a series of medical treatments, and the end date and time of a series of medical treatments. Medical data (tray image data, patient image data, whole image data, device-related data) is recorded by associating the timing (end date and time) with identifiable time data (time stamp). Note that each of the recording unit 130 and the recording unit 230 may record a depth frame that is a frame of the distance to the object shown in the photographed image as medical data.

In this way, each of the recording unit 130 and the recording unit 230 records medical care data in a manner that can be identified by identification data for each series of medical treatments. As a result, when one estimation model 161 is subjected to machine learning and then machine learning is performed again on the same estimation model 161 using the same medical data, the identification data is used as a clue to identify the medical treatment data used in the past. can do. Furthermore, when performing machine learning using the same medical data between multiple estimation models 161, using the identification data as a clue, the clinical data used during machine learning of one estimation model 161 can be used for machine learning of other estimation models 161. can be specified at times. Therefore, the designer of the estimation model 161 can perform machine learning efficiently.

In this way, each of the recording unit 130 and the recording unit 230 records medical care data in a manner that allows the times at which a series of medical treatments were performed to be specified using time data. This allows the designer of the estimation model 161 to perform machine learning on one estimation model 161, and then use the time data as a clue when performing machine learning on the same estimation model 161 again using the same clinical data. , it is possible to specify the medical treatment data used in the past, and also to specify the time of a series of medical treatments performed at that time. Furthermore, when performing machine learning using the same clinical data among multiple estimation models 161, the designer of the estimation model 161 uses the time data as a clue and uses the clinical data used during machine learning for one estimation model 161. can be specified during machine learning of the other estimation model 161, and also the times of a series of medical treatments performed at that time can be specified. Therefore, the designer of the estimation model 161 can perform machine learning efficiently.

Furthermore, each of the recording unit 130 and the recording unit 230 may record an ID as identification information that allows individual medical data to be identified in association with the medical data. Further, each of the recording unit 130 and the recording unit 230 may record contents related to a series of medical treatments (for example, the name of the surgeon, treatment details, other special notes, etc.) in association with the medical treatment data. In this way, the designer of the estimation model 161 can use the above-mentioned ID or content related to a series of medical treatments as clues to identify medical treatment data used in the past, improving the efficiency of machine learning. do.

[Functional configuration of control device at learning stage]
FIG. 11 is a block diagram showing the functional configuration of control device 100 in the learning stage according to the present embodiment. As shown in FIG. 11, the control device 100 includes a recording section 130, a position detection section 157, an object detection section 152, a conversion section 154, a synchronization section 155, and a segmentation section 156 as main functional sections. , a generation section 160, and an output section 180. Note that the output unit 180 is a functional unit of the communication device 101, and the position detection unit 157, object detection unit 152, conversion unit 154, synchronization unit 155, segmentation unit 156, and generation unit 160 are This is the functional part of

The position detection section 157 includes an XY detection section 151 and a Z detection section 153. The XY detection unit 151 performs image recognition based on the overall image data of the overall camera 53 acquired from the recording unit 130 or the recording unit 230 of the server device 200 to obtain positional data (X coordinates, Y coordinate) is detected. As explained in FIG. 8, the Z detection unit 153 detects the person based on the X and Y coordinates of the person identified by the Detect position data (Z coordinate) of For example, based on the overall image data of the overall camera 53, the position detection unit 157 detects the position of the person at predetermined timings (tC1, tC2, tC3, ... shown in FIG. 16, which will be described later), as shown in FIG. 16, which will be described later. Detect position data (X coordinate, Y coordinate, Z coordinate).

Here, the configuration of the XY detection section 151 will be explained in more detail. As shown by the broken line C in the figure, the XY detection unit 151 includes an estimated model 1511. Furthermore, the estimation model 1511 includes a neural network 1512 and parameters 1513 used by the neural network 1512. The parameters 1513 include weighting coefficients used in calculations by the neural network 1512 and determination values used in estimation determination.

The neural network 1512 is an image recognition method using deep learning, such as a convolution neural network (CNN), a recurrent neural network (RNN), or a LSTM network (Long Short Term Memory Network). A well-known neural network used in processing is applied.

The XY detection unit 151 calculates position data of persons such as the operator and the patient (X coordinates of each detection point) based on the overall image data acquired from the recording unit 130 or the recording unit 230 and the estimation model 1511 including the neural network 1512. , Y coordinate).

In the learning stage, the estimation model 1511 uses the person's position data (X coordinates, Y coordinates of each detection point) associated with the whole image data and the person's position data (X coordinates of each detection point) using the whole image data. Optimization (adjustment) is performed by learning based on the estimation results of the coordinates and Y coordinates.

Specifically, when the estimation model 1511 receives whole image data as training data, the neural network 1512 estimates the detection points of the person shown in the photographed image based on the whole image data. Note that the detection points estimated by the estimation model 1511 are determined in advance as shown in FIG. When the estimation model 1511 estimates a detection point for each person, it estimates position data (X coordinate, Y coordinate) for each estimated detection point. Note that the origin of the position data is determined in advance as shown in FIGS. 7 and 8.

The estimation model 1511 determines whether or not its own estimation result matches the person's position data, which is the correct data associated with the input whole image data, and does not update the parameter 1513 if both match. On the other hand, if the two do not match, the parameter 1513 is updated so that the two match, thereby optimizing the parameter 1513. Note that the above-described learning of the estimation model 1511 may be performed not only in the learning stage but also in the operational stage.

Note that the XY detection unit 151 is not limited to one that detects the position data of a person through AI image recognition using a neural network. For example, the XY detection unit 151 may detect the position data of the person by comparing the photographed image with a template prepared in advance, such as by known pattern matching. Further, the function of the XY detection unit 151 may not be possessed by the control device 100, but may be possessed by the general camera 53, or by an edge computer disposed near the general camera 53. good. In this case, detection results by an edge computer or the like are input to the input unit 150 as image-related data.

The object detection unit 152 performs image recognition based on the tray image data of the tray camera 51 acquired from the recording unit 130 or the recording unit 230 to detect the presence or absence, shape, type, etc. of medical instruments placed on the tray 30. can do. For example, the object detection unit 152 detects a predetermined timing (tA1 shown in FIG. 14, which will be described later) as shown in FIG. , tA2, tA3, ...), the presence or absence, shape, type, etc. of the medical instruments placed on the tray 30 are estimated. For example, in the tray image data shown in FIG. 14, data "0" is stored in the recording area corresponding to the medical instruments present on the tray 30, and data "0" is stored in the recording area corresponding to the medical instruments not present on the tray 30. 1" data is stored.

Furthermore, the object detection unit 152 can detect the position of the medical instrument in the patient's oral cavity by performing image recognition based on patient image data from the patient camera 52 acquired from the recording unit 130 or the recording unit 230. For example, the object detection unit 152 detects the data at predetermined timings (tB1, tB2, tB3, etc. shown in FIG. 15, which will be described later), as shown in FIG. In addition, the position of medical instruments within the patient's oral cavity is estimated. For example, in the patient image data shown in FIG. 15, in the positional relationship between the medical instrument and the patient, if both are not located within a predetermined range, data of "0" is stored in the recording area; If it is located, data "1" is stored in the recording area.

Although not shown, the object detection section 152, like the XY detection section 151, has an estimation model including a neural network and parameters. Then, the object detection unit 152 estimates the presence or absence, shape, type, etc. of a medical instrument, and estimates the position of the medical instrument in the patient's oral cavity, through AI image recognition using a neural network.

For example, the object detection unit 152 has an estimation model including a known neural network used in image recognition processing using deep learning, such as a CNN, RNN, and LSTM network. In the learning stage, the estimation model is based on the existence, shape, and type of medical instruments associated with the tray image data, and the estimation results of the presence, shape, and type of medical instruments using the tray image data. Optimized (adjusted) by learning. Furthermore, in the learning stage, the estimation model uses the position of the medical instrument in the patient's oral cavity associated with the patient image data, and the estimation result of the position of the medical instrument in the patient's oral cavity using the patient image data. It is optimized (adjusted) by learning based on the following. Note that the above-described learning of the estimation model is not limited to the learning stage, but may also be performed during the operation stage.

Note that the object detection unit 152 is limited to detecting the presence, shape, type, etc. of medical instruments through AI image recognition using a neural network, and detecting the position of medical instruments in the patient's oral cavity. do not have. For example, the object detection unit 152 can detect the presence, shape, and type of medical instruments by comparing a photographed image with a template prepared in advance, as in the case of well-known pattern matching. The position of a medical instrument may also be detected.

The conversion unit 154 converts the device-related data acquired from the recording unit 130 or the recording unit 230 into a predetermined format. Specifically, the conversion unit 154 converts the device-related data acquired from the recording unit 130 or the recording unit 230 into a format suitable for synchronizing with the image-related data also acquired from the recording unit 130 or the recording unit 230. . For example, the conversion unit 154 converts the format of the device-related data so that the time axis of the device-related data matches the time axis of the image-related data. Alternatively, the conversion unit 154 converts the format of the device-related data so that the time axis of the device-related data is aligned with a common time axis that is aligned with the time axis of the image-related data.

The synchronization unit 155 chronologically synchronizes the image-related data including the detection results by the position detection unit 157 and the object detection unit 152 and the device-related data converted by the conversion unit 154. For example, as shown in FIGS. 18 and 19, which will be described later, the synchronization unit 155 synchronizes image-related data (tray image data, patient image data, whole image data) and device-related data in accordance with the time axis of the synchronization time. Synchronization is performed at predetermined timings (T1, T2, T3, . . . shown in FIGS. 18 and 19, which will be described later). Data obtained by synchronizing image-related data and device-related data is also referred to as "synchronized data." Note that as the synchronization time used when generating the synchronization data, an elapsed time from when a series of medical treatment start conditions were satisfied or an absolute time that can be specified from a timestamp (for example, real time time) may be used.

The segmentation unit 156 segments the synchronous data by separating image-related data and device-related data at predetermined timing. Specifically, the segmentation unit 156 records that a predetermined event has occurred at a predetermined timing with respect to the image-related data and device-related data (i.e., synchronized data) synchronized by the synchronization unit 155. The synchronous data is segmented as follows. "The occurrence of a predetermined event" includes, for example, starting or stopping the driving of medical instruments 15 such as handpieces, starting or stopping lighting of the lighting device 19, starting or ending water supply/drainage in the basin unit 12, or Events related to medical treatment are included, such as when a medical instrument on the tray 30 is taken out or when a medical instrument is placed on the tray 30.

Furthermore, the segmentation unit 156 records the continuation of a predetermined event at a predetermined timing with respect to the image-related data and device-related data (synchronized data) synchronized by the synchronization unit 155. Segment the synchronous data. “Continuation of a predetermined event” includes, for example, the continuation of the driving state of the medical instrument 15 such as a handpiece, the continuation of the lighting state of the lighting device 19, the continuation of water supply/drainage in the basin unit 12, and the continuation of the operation of the medical instrument 15 such as a handpiece. This includes the continuation of events related to medical treatment, such as the continuation of the state in which medical instruments exist or the continuation of the state in which they do not exist.

For example, the segmentation unit 156 segments the synchronized data for each period by dividing the image-related data and device-related data at predetermined timings. As for the segmentation method, flag data may be set at each predetermined timing, or synchronized data may be moved or copied to a predetermined recording area for each segmented period.

The "predetermined timing" at which the synchronized data is segmented by the segmentation unit 156 may be a timing that occurs periodically at predetermined time intervals (for example, every 5 seconds).

Note that the "predetermined timing" at which the synchronized data is divided by the segmentation unit 156 is defined as "0" or "1" data stored in the image-related data (tray image data, patient image data) and in the device-related data. It may be determined based on stored data of "0" or "1".

For example, the "predetermined timing" at which the synchronized data is divided by the segmentation unit 156 is data included in the device-related data that drives the chair 11 of the medical device 1 and data that drives the medical instrument 15 of the medical device 1. Based on at least one of data for driving the lighting device 19 included in the medical treatment device 1 and data for driving the basin unit 12 included in the medical treatment device 1, the occurrence or continuation of driving of these medical treatment devices is determined. It may be timing. For example, as shown in FIG. 19, at timing T3 when the pick-up of the medical instrument 15 switches from "0" to "1", timing T6 when the pick-up of the medical instrument 15 switches from "1" to "0", etc. may be separated.

Note that even if the segmentation unit 156 cannot segment the synchronous data based on device-related data, it can also segment the synchronous data based on image-related data such as tray image data. For example, as shown in FIG. 18, at timing T8 when the determination of the presence or absence of the cleaning needle 309 and the cleaning syringe 310 is switched from "0" to "1", the determination of the presence or absence of the cleaning needle 309 and the cleaning syringe 310 is "1". The synchronized data may be separated at timing T10 when the value changes from "0" to "0". That is, the segmentation unit 156 divides the image-related data and the device-related data into a predetermined pattern based on data indicating the occurrence of an event related to patient treatment that is included in at least one of the image-related data and the device-related data. You can also segment based on timing.

In this way, when segmenting the synchronized data, the segmentation unit 156 can segment the control data by combining the image-related data and the device-related data, even if there is insufficient data for each of the image-related data and device-related data alone. The synchronized data may be separated at appropriate timings for estimating.

Note that if the image capturing timing in the image-related data and the log data acquisition timing in the device-related data are synchronized in advance, the conversion unit 154 and the synchronization unit 155 may not be used. In this case, the segmentation unit 156 directly uses the image-related data and device-related data acquired from the recording unit 130 or the recording unit 230 instead of the synchronization data, and divides the image-related data and the device-related data into predetermined information. It may be segmented by dividing at the timing of .

The generation unit 160 generates an estimation result regarding medical treatment based on the image-related data and device-related data segmented by the segmentation unit 156. For example, the generation unit 160 estimates control data for the medical device 1 based on the image-related data and the device-related data, and generates the estimated control data.

The generation unit 160 has an estimation model 161. Furthermore, the estimation model 161 includes a neural network 162 and parameters 163 used by the neural network 162. The parameters 163 include weighting coefficients used in calculations by the neural network 162 and determination values used in estimation determination.

In the learning stage, the estimation model 161 is trained based on the image-related data and device-related data segmented by the segmentation unit 156, and the estimation results of control data using the image-related data and the device-related data. It is optimized (adjusted) by

For example, when whole image data (position data of a person) is input as training data, the estimation model 161 uses a neural network 162 to grasp the order and posture of the person's actions based on the whole image data. 1 and estimates the control data of the medical device 1.

The estimation model 161 determines whether or not its own estimation result matches the control data of the medical device 1, which is the correct data associated with the input overall image data, and if they match, it changes the parameter 163. The parameter 163 is optimized by not updating the parameter 163, but by updating the parameter 163 so that the two match if they do not match.

In this way, in the learning stage, the control device 100 selects the image-related data such as tray image data, patient image data, and whole image data, and the device-related data acquired by the medical device 1 such as log data. The estimation model 161 including the neural network 162 is subjected to machine learning so that the control data can be estimated based on at least one of the data.

Note that the control device 100 may perform machine learning on the estimation model 161 including the neural network 162 not only in the learning stage but also in the operational stage. For example, in the operation stage, when the medical device 1 operates based on the control data generated by the generation unit 160, a user such as a surgeon inputs correct data to cause the medical device 1 to perform a different operation. , the estimation model 161 may be optimized (readjusted) by learning based on the estimation result and the correct data. In this way, since the control device 100 performs machine learning even during the operation stage, the more a user such as a surgeon uses it, the more the control data estimation accuracy improves.

Note that the estimation model 161 of the generation unit 160 is not limited to a supervised learning algorithm, and may perform machine learning using a known algorithm such as an unsupervised learning algorithm or a reinforcement learning algorithm. For example, when using an unsupervised learning algorithm, the estimation model 161 is based on image-related data, the types of medical instruments used during treatment, the timing of their use, the positional relationship between the medical instruments and the patient's oral cavity, and the The operator's and patient's actions and the log data of each part of the medical treatment device 1 are classified (clustered), and based on the classification results, the medical procedure (for example, the content of the next treatment to be performed during the medical treatment) is estimated. Good too.

The output unit 180 outputs the estimation result (for example, control data) generated by the generation unit 160 to a controlled object in the medical device 1 or the like. The controlled objects include at least one of the chair 11, the medical instrument 15, the lighting device 19, the basin unit 12, the foot controller 16, the display 17, the operation panel 18, and the speaker 35.

The control data includes data indicating the start of driving the controlled object, data indicating the end of driving the controlled object, data indicating prohibition of driving the controlled object, data indicating continuation of driving the controlled object, data indicating the drive amount for the controlled object, and drive. It includes at least one of data indicating an upper limit value regarding the amount, data indicating a lower limit value regarding the drive amount, data indicating a driving time for the controlled object, and data regarding the status of the controlled object.

[Functional configuration of control device during operation stage]
FIG. 12 is a block diagram showing the functional configuration of control device 100 according to the present embodiment at an operation stage. As shown in FIG. 12, in the control device 100 in the operation stage, each of the XY detection section 151, the object detection section 152, and the generation section 160 has been trained. Control device 100 further includes an input section 150. The input unit 150 is a functional unit of the communication device 101.

In the operation stage, image-related data is input from each camera to the input section 150 in real time while the surgeon is treating a patient, and further, device-related data is input from the basin unit 12 to the input section 150.

The XY detection unit 151 performs image recognition based on the overall image data of the overall camera 53 input from the input unit 150 to detect positional data (X coordinates, Y coordinates) of persons such as the operator and the patient. The Z detection unit 153 calculates the position data (Z coordinate) of the person based on the X and Y coordinates of the person identified by the XY detection unit 151 and the distance L to the person calculated from the reflected light from infrared rays or the like. To detect.

The object detection unit 152 performs image recognition based on the tray image data of the tray camera 51 input from the input unit 150 to detect the presence or absence, shape, type, etc. of medical instruments placed on the tray 30. Further, the object detection unit 152 detects the position of the medical instrument in the patient's oral cavity by performing image recognition based on patient image data from the patient camera 52 input from the input unit 150. The object detection unit 152 may detect the position of a key item related to medical treatment, such as a medical instrument, by performing image recognition based on the overall image data of the overall camera 53 input from the input unit 150.

The conversion unit 154 converts the device-related data input from the basin unit 12 to the input unit 150 into a predetermined format.

The synchronization unit 155 generates synchronization data by chronologically synchronizing the image-related data including the detection results by the position detection unit 157 and the object detection unit 152 and the device-related data converted by the conversion unit 154.

The segmentation unit 156 segments the synchronized data by separating the image-related data and device-related data input from the input unit 150 at a predetermined timing.

The generation unit 160 generates an estimation result regarding medical treatment based on the image-related data and device-related data segmented by the segmentation unit 156 and the learned estimation model 161 including the neural network 162. For example, the generation unit 160 estimates control data for the medical device 1 based on the image-related data and the device-related data, and generates the estimated control data.

The output unit 180 outputs the estimation result (for example, control data) generated by the generation unit 160 to a controlled object in the medical device 1 or the like.

In this way, the control device 100 uses image-related data such as tray image data, patient image data, and whole image data, device-related data such as log data acquired by the medical treatment device 1, and estimation including the neural network 162. Control data for controlling the medical treatment device 1 is generated based on the model 161. As a result, the medical device 1 is controlled by the control device 100 prior to the actions of an operator such as a dentist or a dental assistant, so convenience for the operator can be improved.

Note that the control device 100 is not limited to generating control data for controlling the medical device 1 based on both the image-related data and the device-related data and the estimation model 161. The control device 100 may generate control data for controlling the medical treatment device 1 based on the image-related data and the estimation model 161 without using the device-related data.

For example, the control device 100 detects position data of people such as a surgeon and a patient from the overall image data acquired by the overall camera 53, and uses the estimation model 161 to determine the actions each person performs based on the position data of each person. By understanding the order and posture of the patient, the control details of the medical device 1 may be read in advance, and the control data of the medical device 1 may be estimated.

For example, the control device 100 detects the position of the medical instrument in the patient's oral cavity from patient image data acquired by the patient camera 52, and uses the estimation model 161 to detect the position of the medical instrument in the patient's oral cavity. By understanding the order and posture of the operations performed by the patient, the control details of the medical device 1 may be read in advance and the control data of the medical device 1 may be estimated.

For example, the control device 100 detects the presence, shape, and type of medical instruments placed on the tray 30 from the tray image data acquired by the tray camera 51, and uses the estimation model 161 to detect the presence, shape, and type of medical instruments placed on the tray 30. By understanding the order and posture of the operator's actions based on the presence, shape, and type of instruments, the control details of the medical device 1 may be read in advance and the control data of the medical device 1 may be estimated.

For example, the control device 100 may pre-read the control content of the medical device 1 from device-related data and estimate the control data of the medical device 1.

<Example of estimation of control data>
An example of estimation of control data by the control device 100 will be described with reference to FIGS. 13 to 20. Note that FIGS. 13 to 20 show examples of root canal treatment.

During medical treatment, one or more medical instruments placed on the tray 30 are photographed by the tray camera 51 at predetermined timings (tA1, tA2, tA3, . . . ) depending on the shutter timing.

For example, FIG. 13 is a diagram for explaining an example of an image taken by the tray camera 51 during root canal treatment. As shown in FIG. 13, at timing tA2 when examination is being performed as a treatment included in root canal treatment, the operator uses tweezers 314, mirror 316, and deep needle 318, so the image captured by tray camera 51 is The tweezers 314, mirror 316, and deep needle 318 are not shown in the image. At timing tA3 when pulp extraction is being performed as a treatment included in root canal treatment, the file 305 and vacuum 315 are used by the operator, so the file 305 and vacuum 315 are not displayed in the captured image of the tray camera 51. At timing tA5 when root canal length measurement and enlargement are being performed as a procedure included in root canal treatment, the root canal length measurement device 303 and file 305 are used by the operator, so the root canal is not included in the image taken by the tray camera 51. Pipe length measuring device 303 and file 305 are not displayed. At timing tA7 when disinfection and cleaning are performed as a procedure included in root canal treatment, the operator uses the cleaning needle 309, the cleaning syringe 310, and the vacuum 315, so the image taken by the tray camera 51 does not include the following: Cleaning needle 309, cleaning syringe 310, and vacuum 315 are not shown. At timing tA9 when root canal filling is being performed as a procedure included in root canal treatment, the mirror 316 and the root canal material injector 319 are used by the operator, so the image taken by the tray camera 51 does not include the mirror 316 and the root canal. Tubing syringe 319 is not shown. At timing tA11 when stuffing and capping are performed as a treatment included in root canal treatment, the temporary sealant filler 311, tweezers 314, and mirror 316 are used by the operator, so the image taken by the tray camera 51 does not include the following: Temporary sealant filler 311, tweezers 314, and mirror 316 are not displayed.

In this way, during medical treatment, the presence, absence, shape, type, etc. of the medical instruments placed on the tray 30 are displayed using images taken by the tray camera 51.

When the above-described image data captured by the tray camera 51 (image-related data as tray image data) is input to the input unit 150, the control device 100 determines that a series of medical treatment start conditions are satisfied and that the treatment is terminated. The input tray image data is recorded on the recording device 103 on the condition that the condition is not satisfied.

During medical treatment, the patient's oral cavity is photographed by the patient camera 52 at predetermined timings (tB1, tB2, tB3, . . . ) depending on the shutter timing.

Then, when the data of the image taken by the patient camera 52 (image-related data as patient image data) is input to the input unit 150, the control device 100 determines that a series of medical treatment start conditions are satisfied and an end condition is satisfied. On the condition that this is not true, the input patient image data is recorded in the recording device 103.

During medical treatment, the overall camera 53 photographs the medical treatment space, the actions of the surgeon and the patient, the operation of the medical treatment apparatus 1, etc. at predetermined timings depending on the shutter timing.

The control device 100 determines that when the data of images captured by the general camera 53 (image-related data as the overall image data) is input to the input unit 150, a series of medical treatment start conditions are satisfied and an end condition is satisfied. On the condition that this is not true, the inputted entire image data is recorded on the recording device 103.

During medical treatment, the chair 11, basin unit 12, lighting device 19, instrument control device 21, and display included in the medical treatment device 1 are activated at predetermined timings (tD1, tD2, tD3, ...) that depend on the timing of acquiring log data. Log data indicating the operation and control of at least one of the control devices 22 is output from the medical treatment device 1.

For example, FIG. 17 is a diagram for explaining an example of device-related data acquired from the medical treatment device 1. As shown in FIG. 17, the device-related data includes the presence or absence of a pickup, the rotational speed of the air turbine, the rotational speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the operation of the vacuum syringe, and the operation of the Saraiba ejector for the medical instrument 15. , and log data representing the operating state of water injection, etc. Contains log data representing the seat height, back plate angle (backrest angle), and operating state of the chair 11, etc. For the lighting device 19, log data representing the lighting state and luminous intensity is included, and for the basin unit 12, log data representing the presence or absence of a cup is included. Contains log data representing such things.

In the example of FIG. 17, data indicating the degree of magnitude, such as the rotation speed of the air turbine, the rotation speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the amount of depression of the foot controller 16, and the emission intensity of the lighting device 19, is shown. is included in medical care-related data (log data). The medical care-related data includes the rotational speed of the air turbine, the rotational speed of the micromotor, the oscillation intensity of the ultrasonic scaler, the amount of depression of the foot controller 16, and the value (for example, absolute value) of the light emission intensity of the lighting device 19. It's okay to stay. The depression pattern of the foot controller 16 may not be included in the medical care-related data. For example, the control device 100 may identify the depression pattern of the foot controller 16 by analyzing changes in the amount of depression of the foot controller 16 over time.

Note that the log data is not limited to that output from the medical device 1 at predetermined timings (tD1, tD2, tD3, ...) that depend on the acquisition timing of the log data, but is also based on the occurrence of an event (for example, the movement of the chair 11). It may also be outputted from the medical device 1 every time the medical device 15 changes its operation, changes the operation of the medical instrument 15, etc.).

When device-related data is input to the input unit 150, the control device 100 records the input device-related data on the condition that a series of treatment start conditions are met and end conditions are not met. The device 103 records the information.

In the learning stage or the operating stage, the control device 100 estimates control data based on the medical care data recorded in the recording device 103 or the recording device 203.

Specifically, when the control device 100 acquires tray image data from the recording device 103 or the recording device 203, the object detection unit 152 extracts feature amounts included in the image.

For example, FIG. 14 is a diagram for explaining an example of tray image data acquired from photographic information of the tray camera 51. As shown in FIG. 14, the control device 100 estimates the presence or absence, shape, type, etc. of medical instruments placed on the tray 30 at predetermined timings (tA1, tA2, tA3, ...), and Data "0" is stored in the recording area corresponding to the medical instruments that are present on the tray 30, and data "1" is stored in the recording area corresponding to the medical instruments that are not present on the tray 30. Thereby, as shown in FIG. 14, the control device 100 can obtain tray image data (image-related data) that can distinguish the presence or absence of medical instruments at each predetermined timing.

When the control device 100 acquires patient image data from the recording device 103 or the recording device 203, the object detection unit 152 extracts feature amounts included in the image.

For example, FIG. 15 is a diagram for explaining an example of patient image data acquired from imaging information of the patient camera 52. As shown in FIG. 15, the control device 100 estimates the position of the medical instrument in the patient's oral cavity at each predetermined timing (tB1, tB2, tB3, ...), and estimates the position of the medical instrument and the patient's lips, or Regarding the positional relationship between the instrument and the patient's cheek, if both are not located within a predetermined range, "0" is stored in the recording area, and if both are located within the predetermined range, "1" is stored in the recording area. Store data. Thereby, as shown in FIG. 15, the control device 100 can obtain patient image data (image-related data) that can specify the position of the medical instrument in the patient's oral cavity at each predetermined timing.

When the control device 100 acquires the entire image data from the recording device 103 or the recording device 203, the position detection unit 157 extracts feature amounts included in the image to generate position data of persons such as the surgeon and the patient. do. If the position data newly generated by the position detection unit 157 is different from the position data already recorded in the recording device 103, the control device 100 causes the recording device 103 to record the position data newly generated by the position detection unit 157. .

For example, FIG. 16 is a diagram for explaining an example of position data acquired from photographing information of the general camera 53. As shown in FIG. 16, the control device 100 records the generated person's position data in association with time data (time stamp) as whole image data (image-related data). In the example shown in FIG. 16, position data of person A (dentist) is recorded in the recording device 103 at predetermined timings (tC1, tC2, tC3, . . . ). Thereby, the control device 100 can obtain whole image data (image-related data) including positional data that can specify the operator's actions and patient's actions at each predetermined timing. In particular, if the control device 100 knows the positional relationship between the person and the medical device 1 in the medical treatment space based on the positional data recorded in the recording device 103, the control device 100 can avoid unnecessary contact between the person and the medical device 1, for example. Even in such a case, driving of the medical device 1 can be stopped based on the control data.

When the control device 100 acquires the device-related data from the recording device 103 or the recording device 203, the conversion unit 154 converts the device-related data into a format suitable for synchronization with the image-related data.

Next, the control device 100 causes the synchronization unit 155 to synchronize the image-related data shown in FIGS. 14 to 16 and the device-related data shown in FIG. 17 at predetermined timings (T1, T2, T3,...).

For example, FIGS. 18 and 19 are diagrams for explaining examples of synchronous data. As shown in FIGS. 18 and 19, the synchronization data includes image-related data and device-related data that are synchronized at predetermined timings (T1, T2, T3, ...) according to the synchronization time axis. Contains data.

For example, data at tA3 and tA4 in the tray image data shown in FIG. 14 is associated with data at timings (T3 to T5) corresponding to tA3 and tA4 in the synchronization data shown in FIG. The data of tB3 and tB4 in the patient image data shown in FIG. 15 are associated with the data of the timing (T3 to T5) corresponding to tB3 and tB4 in the synchronization data shown in FIG. The data at tC3 in the entire image data (position data) shown in FIG. 16 is associated with the data at the timing corresponding to tC3 in the synchronization data shown in FIG. 18.

Next, the control device 100 causes the segmentation unit 156 to segment the image-related data and device-related data included in the synchronized data shown in FIGS. 18 and 19 by separating them at a predetermined timing.

For example, the control device 100 segments the synchronization data at timings that are periodically visited at predetermined time intervals (for example, every 5 seconds).

Alternatively, the control device 100 can control the data of "0" or "1" stored in the image-related data (tray image data, patient image data) and the data of "0" or "1" stored in the device-related data. Segment synchronized data based on

Here, even if there is no change in each of the tray image data, patient image data, whole image data, and device-related data alone, there may be a change in the data when other data is referred to. That is, the control device 100 can separate the synchronized data at an appropriate timing by referring to and combining a plurality of pieces of data from each of the tray image data, patient image data, whole image data, and device-related data.

For example, as shown in FIG. 18, in the tray image data, each of the data T3 to T7 is the same for the file 305, but the data of each of T3 to T5 is the same for the vacuum. Therefore, by simply referring to the tray image data, the control device 100 may decide whether to segment the synchronization data at T3 to T5 or T3 to T7. However, as shown in FIG. 19, in the device-related data, each of the data T3 to T5 is the same for the medical instrument and the foot controller 16. Therefore, the control device 100 segments the synchronization data at T3 to T5, which are expected to have a high probability, based on the tray image data and device related data.

In this way, when segmenting the synchronized data, the control device 100 can segment the synchronous data even if there is insufficient data for each of the tray image data, patient image data, whole image data, and device-related data alone. By referring to the data, you can separate the synchronized data at the appropriate timing.

As described above, during medical treatment, tray image data, patient image data, whole image data, and device-related data are each input to the control device 100 in chronological order. The control device 100 generates synchronization data based on these input data and segments the synchronization data. The control device 100 can estimate control data by learning the estimation model 161 included in the generation unit 160 based on the synchronization data.

For example, in the case of root canal treatment, the control device 100 inputs synchronized data corresponding to T1 and T2 in "examination", inputs synchronized data corresponding to T3 to T5 in "pulp extraction", and inputs synchronized data corresponding to T3 to T5 in "root canal length measurement". In ``, synchronous data corresponding to T6 and T7 is input, synchronous data corresponding to T8 and T9 is input in ``cleaning/disinfection'', synchronous data corresponding to T10 and T11 is input in ``root canal filling'', In "filling/covering", synchronization data corresponding to T12 and T13 is input.

For example, in the case of root canal treatment, the control device 100 learns the above-mentioned synchronization data, so that when synchronization data corresponding to T1 and T2 is input, the control device 100 thereafter performs the synchronization data corresponding to T3 to T5. It is estimated that such synchronization data will be input, and the control data for "extraction" will be generated. When the synchronous data corresponding to T3 to T5 is input, the control device 100 estimates that synchronous data corresponding to T6 and T7 will be input thereafter, and performs the process for "root canal length measurement". Generates control data. When the synchronization data corresponding to T6 and T7 is input, the control device 100 estimates that the synchronization data corresponding to T8 and T9 will be input after that, and the control device 100 assumes that the synchronization data corresponding to T8 and T9 will be input. Generate control data. When the synchronous data corresponding to T8 and T9 is input, the control device 100 estimates that synchronous data corresponding to T10 and T11 will be input thereafter, and performs the process for "root canal filling". Generate control data. When the synchronization data corresponding to T10 and T11 is input, the control device 100 estimates that synchronization data corresponding to T12 and T13 will be input after that, and performs a process for "stuffing/covering". Generate control data.

As described above, the control device 100 uses so-called AI technology such as a neural network to estimate control data based on at least one of tray image data, patient image data, whole image data, and device-related data. The control data can be estimated by finding the characteristics for

FIG. 20 is a timing chart for explaining the recording timing of medical data. As shown in FIG. 20, the control device 100 records medical data (tray image data, patient image data, whole image data, device-related data) in the recording device 103 when a series of medical treatment start conditions are satisfied. Let it start.

For example, as shown in FIGS. 17 and 20, in the device-related data, the chair 11 is operating at tD1. The control device 100 detects that when the chair 11 is operated, the height of the seat 11c of the chair 11 becomes a predetermined height or more, or the angle of the backrest 11b of the chair 11 becomes a predetermined angle or more. Then, the recording device 103 starts recording the medical data.

The control device 100 causes the recording device 103 to end the recording of medical care data when a series of medical treatment end conditions are satisfied.

For example, as shown in FIGS. 17 and 20, in the device-related data, the chair 11 is operating at tD15. By operating the chair 11, the control device 100 indicates that the height of the seat 11c of the chair 11 has become less than a predetermined height, or that the angle of the backrest 11b of the chair 11 has become less than a predetermined angle. When detected, the recording device 103 ends recording of the medical data.

While the recording device 103 is recording medical data, the control device 100 acquires tray image data from the tray camera 51 at predetermined timings (tA1, tA2, tA3, ...), and transfers the acquired tray image data to the recording device. 103 to record.

While the recording device 103 is recording medical data, the control device 100 acquires patient image data from the patient camera 52 at predetermined timings (tB1, tB2, tB3, ...), and transfers the acquired patient image data to the recording device. 103 to record.

While the recording device 103 is recording medical data, the control device 100 acquires whole image data from the whole camera 53 at a predetermined timing, and causes the recording device 103 to record the acquired patient image data. Further, the control device 100 generates position data of persons such as the operator and the patient based on the acquired overall image data. For example, when the control device 100 generates position data of person A (dentist) at predetermined timings (tC1, tC2, tC3, . . . ), the control device 100 causes the recording device 103 to record the generated position data.

While the recording device 103 is recording medical data, the control device 100 acquires device-related data from the medical device 1 at predetermined timings (tD1, tD2, tD3, ...), and transfers the acquired device-related data to the recording device. 103 to record.

The control device 100 generates synchronized data by synchronizing the image-related data and device-related data recorded in the recording device 103 as described above at predetermined timings (T1, T2, T3, ...). do. Then, the control device 100 generates control data for the medical treatment device 1 using the generated synchronization data.

<Control device processing>
[Recording control processing]
FIG. 21 is a flowchart for explaining an example of the recording control process executed by the control device 100. Each step (hereinafter referred to as "S") shown in FIG. 21 is realized by the arithmetic unit 102 of the control device 100 executing the OS 142 and the control program 140.

As shown in FIG. 21, the control device 100 determines whether a series of medical treatment start conditions are satisfied (S1). If the start condition is not satisfied (NO in S1), the control device 100 ends this process.

On the other hand, if the start condition is satisfied (YES in S1), the control device 100 allows the recording device 103 to record by setting the recording flag to ON (S2). That is, the control device 100 causes the recording device 103 to start recording medical data.

The control device 100 determines whether image-related data has been input (S3). When the image-related data is input (YES in S3), the control device 100 causes the recording device 103 to record the input image-related data (S4).

If image-related data has not been input (NO in S3), or after the process in S4, control device 100 determines whether device-related data has been input (S5). When the device-related data is input (YES in S5), the control device 100 causes the recording device 103 to record the input device-related data (S6).
If no device-related data has been input (NO in S5), or after the process in S6, the control device 100 determines whether a series of treatment termination conditions have been met (S7). If the termination condition is not satisfied (NO in S7), the control device 100 returns to the process in S3.

On the other hand, if the termination condition is satisfied (YES in S7), the control device 100 prohibits recording by the recording device 103 by setting the recording flag to OFF (S8). That is, the control device 100 causes the recording device 103 to finish recording the medical data. After that, the control device 100 ends this process.

In this way, by using the control method defined by the control program 140, the control device 100 starts recording medical data at the start timing of a series of medical treatments, and records medical data at the end timing of a series of medical treatments. can be terminated. This prevents the medical data from being insufficient for a series of medical treatments actually performed, or from including useless data unrelated to the series of medical treatments. Therefore, the control device 100 can record medical care data that can be easily utilized as learning data for machine learning of the estimation model 161.

[Learning process]
FIG. 22 is a flowchart for explaining an example of a learning process executed by the control device 100. Each step (hereinafter referred to as "S") shown in FIG. 22 is realized by the arithmetic unit 102 of the control device 100 executing the OS 142 and the estimation program 141.

As shown in FIG. 22, the control device 100 acquires medical data from the recording device 103 or the recording device 203 (S11). The control device 100 determines whether image-related data has been acquired (S12). When the control device 100 acquires image-related data (YES in S12), it performs image recognition (S13). For example, when the entire image data is input, the control device 100 uses the position detection unit 157 to detect the position of the person based on the entire image data and generates position data. When patient image data or tray image data is input, the control device 100 uses the object detection unit 152 to detect the presence, shape, type, etc. of an object such as a medical instrument.

If the control device 100 has not acquired image-related data (NO in S12), or after the process in S13, it determines whether device-related data has been acquired (S14). When the control device 100 acquires the device-related data (YES in S14), the conversion unit 154 converts the device-related data into a predetermined format (S15).

If the control device 100 has not acquired the device-related data (NO in S14), or after the process in S15, the synchronization unit 155 generates synchronized data by synchronizing the image-related data and the device-related data. (S16).

Next, the control device 100 segments the synchronous data by having the segmentation unit 156 separate the image-related data and the device-related data at a predetermined timing based on the synchronous data (S17).

Next, the control device 100 uses the generation unit 160 to estimate control data based on the segmented synchronization data, and generates the estimated control data (S18). Next, the control device 100 outputs the generated control data to the outside (S19).

Next, the control device 100 performs machine learning on the estimation model 161 based on the generated control data and the control data that is correct data associated with the medical care data (S20), and ends this process.

In this way, the control device 100 can perform machine learning on the estimation model 161 for estimating control data using the medical care data recorded in the recording device 103 or the recording device 203. At this time, since the medical care data recorded in the recording device 103 or the recording device 203 is recorded corresponding to the start timing and end timing of a series of medical treatments, the control device 100 Based on the medical care data recorded in 203, the operations of the medical device 1 and the dentist's actions at the time when the series of medical treatments were performed can be reproduced. The control device 100 can improve the accuracy of machine learning of the estimation model 161 by using medical care data recorded for each series of medical treatments. Furthermore, the control device 100 can improve the estimation accuracy of the estimation model 161 by repeatedly using medical care data recorded for each series of medical treatments to re-learn the estimation model 161.

<Main disclosure>
As described above, this embodiment includes the following disclosures.

[Configuration 1]
The control device (100) includes an input section (150) into which medical data is input, and a control section (120) that controls a recording section (130) that records the medical data. The control unit (120) causes the recording unit (130) to start recording the medical care data input to the input unit (150) when a series of medical treatment start conditions are satisfied.

This allows the control device 100 to start recording medical data at the start timing of a series of medical treatments, thereby preventing recording of useless data unrelated to a series of medical treatments before the series of medical treatments starts. It is possible to record clinical data that is easy to utilize.

[Configuration 2]
The starting conditions are that the height of the seat (11c) of the chair (11) for a series of medical treatments has reached a predetermined height or higher, and that the height of the bed for a series of medical treatments has reached a predetermined height or higher. The angle of the backrest (11b) of the chair (11) has reached a predetermined angle or more, the driving of the medical instruments has been turned on, the power of the lighting device (19) has been turned on, and the chair (11) has turned on. (11) includes at least one of the following: a person is detected sitting in the medical treatment space; a medical instrument is detected in the medical treatment space; and a person is detected in the medical treatment space.

As a result, the control device 100 can start recording medical data when the start condition for inferring the start of a series of medical treatments is satisfied, so that the control device 100 can record medical data at an appropriate timing when a series of medical treatments starts. can be started.

[Configuration 3]
The control unit (120) causes the recording unit (130) to terminate recording of the medical care data input to the input unit (150) when a series of medical treatment termination conditions are satisfied.

As a result, the control device 100 can finish recording the medical data at the end timing of the series of medical treatments, thereby preventing recording of useless data unrelated to the series of medical treatments after the series of medical treatments has ended. It is possible to record clinical data that is easy to utilize.

[Configuration 4]
The termination conditions are that the height of the seat (11c) of the chair (11) for a series of medical treatments has become less than a predetermined height, and that the height of the bed for a series of medical treatments has become less than a predetermined height. The angle of the backrest (11b) of the chair (11) has become less than a predetermined angle, the drive of the medical instruments has been turned off, the power of the lighting device (19) has been turned off, and the chair (11) has turned off. (11) This includes at least one of the following: a person's departure from the seat is detected, a medical instrument is not detected in the medical treatment space for a predetermined period of time or more, and a person is not detected in the medical treatment space.

As a result, the control device 100 can end the recording of medical data when the end condition for estimating the end of a series of medical treatments is satisfied, so that the control device 100 can record the medical data at an appropriate timing when the series of medical treatments ends. can be terminated.

[Configuration 5]
The control unit (120) causes the recording unit (130) to record medical treatment data in association with identification data assigned to a series of medical treatments.

As a result, the control device 100 can record medical care data in a manner that can be identified by the identification data for each series of medical treatments. When performing learning or machine learning using the same medical data among a plurality of estimation models 161, efficient machine learning can be performed.

[Configuration 6]
The control unit (120) associates the times at which the series of medical treatments were performed with identifiable time data, and causes the recording unit (130) to record the medical treatment data.

As a result, the control device 100 can record the medical care data in a manner that can specify the time during which a series of medical treatments were performed. When relearning is performed after the estimation model 161 has been trained, or machine learning is performed using the same medical data among a plurality of estimation models 161, efficient machine learning can be performed.

[Configuration 7]
The medical care data includes image-related data captured by cameras (51, 52, 53) in the medical treatment space.

Thereby, the control device 100 can perform machine learning on the estimation model 161 using image-related data obtained by photographing a series of medical treatments.

[Configuration 8]
Medical data is collected from a series of medical chairs (11), medical instruments (15), lighting devices (19), water supply/drainage devices (12), foot controllers (16), displays (17), and operation panels (18). ) at least one of the past and present control data.

As a result, the control device 100 can transmit information such as data for driving the chair 11, data for driving the medical equipment 15, data for driving the lighting device 19, and data for driving the water supply/drainage device (basin unit) 12 during medical treatment. By using the generated data, it is possible to compensate for the lack of image-related data for performing machine learning on the estimation model 161, and it is possible to output estimation results with higher accuracy.

[Configuration 9]
The medical care data recorded by the recording unit (130) is learning data for machine learning of the estimation model (161).

Thereby, the control device 100 can perform machine learning on the estimation model 161 using the recorded medical care data.

[Configuration 10]
A control method for controlling the recording of medical data related to a series of medical treatments by a computer (102) includes steps in which medical data is input (S3, S5), and a step in which it is determined whether conditions for starting a series of medical treatments are satisfied. (S1), a step (S2) of causing the recording unit (130) to start recording input medical data when a series of medical treatment start conditions are met, and a step (S2) when a series of medical treatment termination conditions are satisfied. and a step of causing the recording unit (130) to finish recording the input medical data.

This allows the control device 100 to start recording medical data at the start timing of a series of medical treatments, thereby preventing recording of useless data unrelated to a series of medical treatments before the series of medical treatments starts. It is possible to record clinical data that is easy to utilize.

[Configuration 11]
The control program (140) includes steps (S3, S5) in which medical care data is input into the computer (102), a step (S1) in which it is determined whether or not a series of medical treatment start conditions are satisfied, and a series of steps (S1) in which medical treatment data is input into the computer (102). A step (S2) of causing the recording unit (130) to start recording the input medical data when the medical treatment start condition is met; and causing the recording unit (130) to complete the recording.

This allows the control device 100 to start recording medical data at the start timing of a series of medical treatments, thereby preventing recording of useless data unrelated to a series of medical treatments before the series of medical treatments starts. It is possible to record clinical data that is easy to utilize.

[Configuration 12]
The data structure of the learning data used by the computer (102) for machine learning of the estimation model (161) includes medical data regarding a series of medical treatments, identification data assigned to the series of medical treatments, and data regarding the series of medical treatments. Contains time data (time stamp) that can identify the time at which the event took place.

This allows the designer of the estimation model 161 to efficiently train the estimation model 161 after machine learning, or when performing machine learning on multiple estimation models 161 using the same clinical data. It can be made to learn.

<Modified example>
The present invention is not limited to the above embodiments, and can be further modified and applied in various ways. Modifications applicable to the present invention will be described below.

[About the camera]
In this embodiment, one tray camera 51, one patient camera 52, and one general camera 53 are each attached to the medical device 1, but a plurality of these cameras may be attached to the medical device 1. .

The tray camera 51, the patient camera 52, and the general camera 53 are not limited to fixed types, and may be movable types that follow the movement of the tray table 13 on which the tray 30 is placed or the movement of the operator.

The tray camera 51, the patient camera 52, and the overall camera 53 may be mounted on the medical device 1, or may be cameras whose optical axes are directed from the periphery of the medical device 1 toward the medical device 1. Furthermore, in a medical treatment space that can be understood by images captured by cameras, a plurality of cameras may be installed so that there are no blind spots in imaging. Furthermore, the control device 100 may generate the control data using an image taken by an intraoral camera mounted on a medical instrument, a microscope, or a camera mounted on glasses used by a surgeon.

By analyzing the image captured by the tray camera 51 through image recognition, the control device 100 not only identifies the presence, shape, and type of medical instruments placed on the tray 30, but also identifies the presence, shape, and type of medical instruments placed on the tray 30. The medical instrument held in the operator's hand, or the medical instrument placed in the operator's hands, such as a dentist or dental assistant, may be identified.

[About the control device]
The control device 100 may be included in the basin unit 12 and included in the medical treatment device 1. Alternatively, the control device 100 may be a computer such as an in-hospital server that is separate from the medical device 1 and is communicably connected to the medical device 1. Furthermore, the control device 100 may exist in the form of cloud computing, such as an external server installed outside the medical treatment space where the medical treatment device 1 is installed. For example, the server device 200 may have the functions of the control device 100.

The generation unit 160 of the control device 100 is not limited to estimating control data of the medical device 1 using the estimation model 161. For example, the generation unit 160 may estimate the next operation (control content) of the medical device 1 using the estimation model 161, and generate control data corresponding to the estimation result. Alternatively, the generation unit 160 may estimate the next movement of the person using the estimation model 161 and generate control data corresponding to the estimation result. Alternatively, the generation unit 160 may estimate the medical treatment procedure (for example, the content of the next treatment to be performed during the medical treatment) using the estimation model 161, and generate control data corresponding to the estimation result.

Based on the estimation results of the estimation model 161, the control device 100 is not limited to directly controlling the medical device 1, but can also display a warning image on a display separate from the medical device 1, or output a warning sound to a speaker. Alternatively, the operator may operate the medical device 1 (for example, perform a stop operation) by doing so. That is, the control device 100 may not directly control the medical device 1, but may indirectly control the medical device 1 by guiding the surgeon.

[About the recording section]
The control device 100 is not limited to one that controls the recording device 103. For example, the control device 100 may control the recording device 203 of the server device 200. Specifically, the arithmetic unit 102 (control unit 120) of the control device 100 causes the recording device 203 to start recording medical data when a series of treatment start conditions are met, and when a series of treatment termination conditions are met. If this is true, the recording device 203 may end recording of the medical data.

Furthermore, the "recording unit" is not limited to the recording device 103 included in the control device 100 and the recording device 203 included in the server device 200, but may be a recording device included in an in-hospital server communicably connected to the medical treatment device 1. However, it may be a recording device (for example, a large-capacity storage such as a data center) provided in an out-of-hospital server installed outside the medical treatment space where the medical device 1 is installed. Further, the "recording unit" may exist in the form of cloud computing with which each of the plurality of control devices 100 that acquires medical data from each of the plurality of medical devices 1 can communicate. In this way, the medical data acquired from multiple medical devices 1 can be uniformly accumulated and managed by the "recording section", which improves the accuracy of machine learning of the estimation model 161, and further improves the accuracy of the machine learning of the estimation model 161. A plurality of estimation models 161 having estimation accuracy can be generated.

The recording unit 130 records image-related data (whole image data, patient image data, tray image data) including images taken by each camera as is. In other words, the recording unit 130 records the moving images or images taken by each camera as they are. However, the control unit 120 may compress or process these raw image-related data acquired from each camera and record the compressed or processed data in the recording unit 130. Alternatively, the control unit 120 may detect features from these raw image-related data obtained from each camera, and cause the recording unit 130 to record encoded data of the detected features.

For example, the control device 100 may provide at least one of the position detection section 157 and the object detection section 152 shown in FIG. 11 between the input section 150 and the recording section 130 shown in FIG. In this way, the control unit 120 records the position data of the person detected by the position detection unit 157 in the recording unit 130 based on the overall image data of the overall camera 53 in accordance with the start and end of a series of medical treatments. can be done. Based on the tray image data of the tray camera 51, the control unit 120 records data such as the presence or absence, shape, and type of medical instruments detected by the object detection unit 152 to the recording unit 130 in accordance with the start and end of a series of medical treatments. can be recorded. Based on patient image data from the patient camera 52, the control unit 120 records data such as the position of medical instruments in the patient's oral cavity detected by the object detection unit 152 to the recording unit 130 at the start and end of a series of medical treatments. can be recorded.

The control device 100 according to the present embodiment controls the medical treatment device 1 used by an operator to perform dental treatment on a patient. It may also be something that controls the device.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the above description, and it is intended that all changes within the scope and meanings equivalent to the claims are included. Note that the configurations exemplified in this embodiment and the configurations exemplified in the modified examples can be combined as appropriate.

1 Medical equipment, 2 Patient, 3 Dentist, 4 Dental assistant, 5 Pole, 6 Arm, 11 Chair, 11a Headrest, 11b Backrest, 11c Seat, 11d Foot rest, 12 Basin unit, 12a Bowl, 12b Cup stand , 12c water tap, 13 tray table, 14 instrument holder, 15 medical instruments, 16 foot controller, 17 display, 18 operation panel, 19 lighting device, 21 instrument control device, 22 display control device, 30 tray, 32 sound control device, 35 speaker, 51 tray camera, 52 patient camera, 53 general camera, 100 control device, 101 communication device, 102 arithmetic device, 103, 203 recording device, 110 communication section, 111 chair control section, 120 control section, 121 bassin control section , 122 illumination control section, 123 storage section, 130, 230 recording section, 140 control program, 141 estimation program, 150 input section, 151 XY detection section, 152 object detection section, 153 Z detection section, 154 conversion section, 155 synchronization section, 156 segmentation section, 157 position detection section, 160 generation section, 161, 1511 estimation model, 162, 1512 neural network, 163, 1513 parameter, 180 output section, 200 server device, 211 appliance control section, 221 display control section , 222 Panel control unit, 301 Rubber dam moisture proof set, 302 Rubber dam sheet, 303 Root canal length measuring device, 304 Bar set, 305 File, 306 Opposite corner of mouth, 307 Faul clip, 308 Brooch, 309 Cleaning needle, 310 Cleaning syringe, 311 Temporary sealant filling device, 312 Cleanser, 313 Turbine, 314 Tweezers, 315 Vacuum, 316 Mirror, 317 Excavator, 318 Needle, 319 Root canal material injector, 321 Sound control unit, 1000 Control system.

Claims (10)

A control device that controls recording of dental treatment data regarding a series of dental treatments,
an input unit into which the dental treatment data is input;
and a control unit that controls a recording unit that records the dental treatment data,
The control unit causes the recording unit to start recording the dental treatment data input to the input unit when the series of dental treatment start conditions is satisfied ,
The start condition is a condition for inferring the start of the series of dental treatments; The height of the bed for medical treatment has exceeded a predetermined height, the angle of the backrest of the chair has exceeded a predetermined angle, the driving of medical instruments has been turned on, and the power of the lighting device has been turned on. , a person is detected sitting on the chair, the medical equipment is detected in the medical treatment space, and a person is detected in the medical treatment space. ,Control device. The control device according to claim 1 , wherein the control unit causes the recording unit to end recording of the dental treatment data input to the input unit when a condition for ending the series of dental treatment is satisfied. The termination conditions include that the height of the seat of the chair has become less than a predetermined height, that the height of the bed has become less than a predetermined height, and that the backrest of the chair has become less than a predetermined height. that the angle of the medical instrument has become less than a predetermined angle, that the drive of the medical instrument has been turned off, that the power of the lighting device has been turned off, that a person has left the chair , 3. The control device according to claim 2 , wherein the control device includes at least one of: the medical instrument is not detected in the medical treatment space for a predetermined period of time or more, and a person is not detected in the medical treatment space. The control device according to any one of claims 1 to 3 , wherein the control unit causes the recording unit to record the dental treatment data in association with identification data assigned to the series of dental treatments. According to any one of claims 1 to 4 , the control unit causes the recording unit to record the dental treatment data by associating the time when the series of dental treatments were performed with identifiable time data. Control device as described. The control device according to any one of claims 1 to 5 , wherein the dental treatment data includes image-related data obtained by photographing the treatment space with a camera. The dental treatment data includes past and present control data on at least one of the chair , the medical equipment, the lighting device, the water supply/drainage device, the foot controller, the display, and the operation panel. The control device according to any one of claims 1 to 6 , comprising: The control device according to any one of claims 1 to 7 , wherein the dental treatment data recorded by the recording unit is learning data for machine learning of an estimation model. A control method for controlling the recording of dental treatment data regarding a series of dental treatments by a computer,
the step of inputting the dental treatment data;
a step of determining whether the series of dental treatment start conditions are satisfied;
a step of causing a recording unit to start recording the inputted dental treatment data when the start condition is satisfied;
the step of causing the recording unit to terminate recording of the inputted dental treatment data when the series of termination conditions for the dental treatment is satisfied ;
The start condition is a condition for inferring the start of the series of dental treatments; The height of the bed for medical treatment has exceeded a predetermined height, the angle of the backrest of the chair has exceeded a predetermined angle, the driving of medical instruments has been turned on, and the power of the lighting device has been turned on. , a person is detected sitting on the chair, the medical instrument is detected in the medical treatment space, and a person is detected in the medical treatment space. Hmm, control method. A control program that controls recording of dental treatment data regarding a series of dental treatments,
to the computer,
the step of inputting the dental treatment data;
a step of determining whether the series of dental treatment start conditions are satisfied;
a step of causing a recording unit to start recording the inputted dental treatment data when the start condition is satisfied;
causing the recording unit to terminate recording of the input dental treatment data when the series of dental treatment termination conditions are satisfied ;
The start condition is a condition for inferring the start of the series of dental treatments; The height of the bed for medical treatment has exceeded a predetermined height, the angle of the backrest of the chair has exceeded a predetermined angle, the driving of medical instruments has been turned on, and the power of the lighting device has been turned on. , a person is detected sitting on the chair, the medical equipment is detected in the medical treatment space, and a person is detected in the medical treatment space. , control program.

Images