KR102013806B1 - Method and apparatus for generating artificial data - Google Patents

Abstract

Provided is a method for generating artificial data performed by a computer. The method comprises steps of: obtaining a background image and an object image; and generating artificial data by combining the background image and the object image, wherein the background image comprises a specific area inside the body photographed by an endoscope camera, and the object image comprises a surgical tool or blood. In addition, the artificial data is generated to approximate an actual surgical image based on a placement relationship between the surgical tool or blood in the object image and the specific area inside the body in the background image.

Description

Artificial data generation method and apparatus {METHOD AND APPARATUS FOR GENERATING ARTIFICIAL DATA}

The present invention relates to a method and apparatus for generating artificial data.

In the surgical procedure, there is a need for the development of technologies that can provide information to assist the surgeon's surgery. In order to provide information to assist the operation, the operation should be recognizable.

Therefore, there is a need for the development of a technology that allows a computer to recognize surgical behavior from a surgical image.

In recent years, deep learning has been widely used to analyze medical images. Deep learning is defined as a set of machine learning algorithms that attempts to achieve high levels of abstraction (summarizing key content or functions in large amounts of data or complex data) through a combination of several nonlinear transformations. Deep learning can be seen as a field of machine learning that teaches computers how people think in a large framework.

When learning using such deep learning and machine learning, a large amount of learning data is required. However, since the surgical image is obtained during the actual surgical procedure, the amount of data is small and it is difficult to secure various types of surgical images.

The problem to be solved by the present invention is to provide an artificial data generating method and apparatus.

An object of the present invention is to provide a method and apparatus for generating artificial data from an actual surgical image.

The problem to be solved by the present invention is to provide a method and apparatus for generating artificial data close to the actual surgical image through learning.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method of generating artificial data performed by a computer according to an embodiment of the present invention includes obtaining a background image and an object image, and generating artificial data by combining the background image and the object image. The background image includes a specific area inside the body taken by the endoscope camera, and the object image includes a surgical tool or blood, and the artificial data includes a surgical tool or blood in the object image, and the It is characterized in that it is generated to approximate the actual surgical image based on the arrangement relationship between specific regions within the body in the background image.

In an embodiment of the present invention, the generating of the artificial data may generate the artificial data by using a generational model trained on the basis of the actual surgical image.

In an embodiment of the present invention, the generating of the artificial data may include determining whether the artificial data is real data or artificial data by using a discriminative model trained to discriminate whether the artificial data is real data or artificial data. And re-learning the generation model according to the determination result.

In one embodiment of the present invention, the differential model may be learned using the artificial data and the object segmentation in the actual surgical image corresponding to the artificial data.

In one embodiment of the present invention, the method may further include constructing a training data set for surgical image recognition based on new artificial data generated from the retrained generation model.

In an embodiment of the present disclosure, the acquiring of the background image and the object image may include obtaining the object image from a first real surgery image including an object and a background, and including a second real image including an object and a background. The background image may be obtained from a surgical image.

In an embodiment of the present disclosure, the acquiring of the background image and the object image may include obtaining the object image from a first real surgery image including an object and a second real surgery image including an object and a background. The background image can be obtained from the.

In an embodiment of the present disclosure, the obtaining of the background image and the object image may include obtaining the object image from a first real surgery image including an object and a background, and including a background. The background image can be obtained from the.

In an embodiment of the present disclosure, the acquiring of the background image and the object image may include obtaining the object image from a first real surgery image including an object and from the second real surgery image including a background. A background image can be obtained.

An apparatus according to an embodiment of the present invention includes a memory for storing one or more instructions, and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions, thereby executing a background image and Acquiring an object image, and generating artificial data by combining the background image and the object image, wherein the background image includes a specific area inside the body photographed by an endoscope camera, and the object The image may include a surgical instrument or blood, and the artificial data may be generated to approximate an actual surgical image based on an arrangement relationship between the surgical instrument or blood in the object image and a specific region inside the body in the background image. It is characterized by.

A computer program according to an embodiment of the present invention is combined with a computer, which is hardware, and stored in a computer-readable recording medium to perform the artificial data generation method.

According to the present invention, artificial data close to the actual surgical image can be generated through learning.

According to the present invention, learning data can be constructed using artificial data.

According to the present invention, a sufficient learning effect can be obtained by providing artificially generated surgical images together with actual surgical images, such as artificial intelligence, deep learning, and machine learning, which require a large amount of learning data.

According to the present invention, an artificially generated surgical image including various surgical information that cannot be obtained during actual surgery can be secured.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a flowchart illustrating a method of generating artificial data according to an embodiment of the present invention.
2 is a view for explaining a method of generating artificial data based on learning according to an embodiment of the present invention.
3 to 6 are diagrams for explaining embodiments of a method for generating artificial data based on learning according to an embodiment of the present invention.
7 is a schematic diagram of a system capable of performing robotic surgery in accordance with one embodiment of the present invention.
8 is a diagram schematically illustrating a configuration of an apparatus 700 for performing an artificial data generating method according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

As used herein, the term "part" or "module" refers to a hardware component such as software, FPGA, or ASIC, and the "part" or "module" plays certain roles. However, "part" or "module" is not meant to be limited to software or hardware. The “unit” or “module” may be configured to be in an addressable storage medium or may be configured to play one or more processors. Thus, as an example, a "part" or "module" may include components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, Procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. Functions provided within components and "parts" or "modules" may be combined into smaller numbers of components and "parts" or "modules" or into additional components and "parts" or "modules". Can be further separated.

As used herein, the term "computer" includes all the various devices capable of performing arithmetic processing to provide a result to a user. For example, a computer can be a desktop PC, a notebook, as well as a smartphone, a tablet PC, a cellular phone, a PCS phone (Personal Communication Service phone), synchronous / asynchronous The mobile terminal of the International Mobile Telecommunication-2000 (IMT-2000), a Palm Personal Computer (PC), a Personal Digital Assistant (PDA), and the like may also be applicable. In addition, when a head mounted display (HMD) device includes a computing function, the HMD device may be a computer. Also, the computer may correspond to a server that receives a request from a client and performs information processing.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating a method of generating artificial data according to an embodiment of the present invention.

Although the method of FIG. 1 is described as being performed by a computer for convenience of description, the subject of each step is not limited to a specific device but may be used to encompass a device capable of performing computing processing. That is, in the present embodiment, the computer may mean an apparatus capable of performing the artificial data generating method according to the embodiment of the present invention.

Referring to FIG. 1, in the method of generating artificial data according to an embodiment of the present invention, obtaining a background image and an object image (S100), and combining the background image and an object image to generate artificial data (S110). ) May be included. Hereinafter, a detailed description of each step will be described.

The computer may acquire a background image and an object image (S100).

The background image and the object image may be part or the entire image area included in the actual surgery image. The actual surgery image may be data obtained by the actual medical staff performing the surgery. For example, medical staff may perform surgery directly on a patient, or perform minimally invasive surgery using a surgical robot, a laparoscope, an endoscope, or the like. At this time, it is possible to obtain a variety of information about the surgical operation performed in such a surgical process or surgical instruments, surgical sites related to the surgical operation. For example, an actual surgical image obtained by photographing organs, surgical instruments, and the like in the body may be obtained in a surgical procedure, or data recorded on a surgical operation performed in the surgical procedure may be acquired.

In one embodiment, the background image may include a specific area inside the body taken by the endoscope camera. For example, the image data may be image data obtained by photographing a specific region including organs (eg, liver, heart, uterus, brain, breast, abdomen, etc.), blood vessels, and tissues in the body.

In one embodiment, the object image may be image data including surgical instruments or blood (including bleeding out of the blood vessels due to blood vessel damage).

According to an embodiment, the computer may acquire an object image or a background image from an actual surgical image including an object and a background (ie, a background including a specific area inside the body). Alternatively, the computer may acquire the object image from the actual surgical image including only the object or the background image from the actual surgical image including only the background. Thereafter, the computer may generate various artificial data using the object image or the background image obtained from the actual surgical image. Specific embodiments thereof will be described with reference to FIGS. 3 to 6.

In addition, according to an embodiment, the computer may acquire the actual surgical image in units of frames or in units of sequences, which are sets of consecutive frames. For example, when generating artificial data about an object including a surgical tool, the computer may generate an artificial data by acquiring an actual surgical image by one frame and extracting an object image including the surgical tool therefrom. have. Alternatively, when generating artificial data about an object including blood or bleeding, the computer acquires an actual surgical image in a sequence unit consisting of a plurality of frames and extracts an object image including blood or bleeding from the artificial data. Can be generated. In this case, the computer inputs the actual surgical image including the object image in sequence units (ie, continuous frames) to the generator to be described later, and accordingly, the generator inputs artificial data in sequence units (ie, continuous frames). Can be generated and printed. In the case of an object such as blood or bleeding, it is difficult to see the meaning of the object by looking at only one frame image, and thus it may be meaningful to be generated as a set of consecutive frames instead of one frame. Accordingly, in the present invention, artificial data may be generated in units of frames or sequences according to the type of object. In addition, according to the present invention, even when generating artificial data corresponding to a specific surgical operation (for example, cutting, grabbing, etc.), object images are acquired in sequence units as in the case of generating artificial data including blood or bleeding. Artificial data can be generated.

In addition, according to the present invention, artificial data may be generated by acquiring an actual surgical image in a frame unit or a sequence unit as a change of an object occurs. In one embodiment, when the computer acquires a real surgery image that changes in a specific object, for example, to acquire a real surgery image performing a specific surgery operation according to the movement of the surgical tool or the surgical tool in a continuous frame In this case, the computer may continuously extract an object image including a specific object (eg, a surgical tool) from consecutive frames to generate artificial data reflecting a change in movement of the specific object (eg, a surgical tool). In this case, the artificial data may be generated in a continuous frame according to a change of a specific object. In addition, the computer may extract a background image from the actual surgical image in response to the movement change of a specific object (eg, a surgical tool). For example, the computer continuously extracts the object image and the background image according to the change of the movement of a specific object (for example, a surgical tool), and generates continuous artificial data by sequentially combining the continuous object image and the background image. You may. Therefore, by extracting and reflecting the background image corresponding to the change of the object, it is possible to generate more artificial data.

The computer may generate artificial data by combining the background image and the object image (S110).

In one embodiment, the computer is based on the placement relationship between objects in the object image (e.g. surgical instruments, blood, bleeding, etc.) and specific areas within the body (e.g. organs, blood vessels, tissues, etc.) within the background image. As such, artificial data may be generated by combining the object image and the background image to approximate the actual surgical image.

The arrangement relationship may be information derived based on information such as a shape, a location, a direction or a camera photographing direction, a type and a number of backgrounds and objects in each image.

In generating artificial data, the computer may generate artificial data through learning by using a generator and a discriminator. This will be described in detail with reference to FIG. 2.

2 is a view for explaining a method of generating artificial data based on learning according to an embodiment of the present invention.

In an embodiment, in generating artificial data by combining the object and the background based on the arrangement relationship between the object in the object image and the background in the background image, the computer may generate the artificial data closer to the actual surgical image. Learning can be done. In this case, the computer may perform the learning by using the generator 200 and the discriminator 210. For example, the generator 200 and the discriminator 210 may operate by competing with each other using a generative adversarial network (GAN) based learning scheme.

Referring to FIG. 2, the generator 200 trains a generation model using a real surgery image, and based on the learned generation model, artificial data close to the actual input data (actual surgery image) 220. And 230.

In an embodiment, the generator 200 may generate artificial data by combining a background image and an object image obtained from each actual surgical image using a generation model.

The discriminator 210 trains a discrimination model to discriminate whether the artificial data 230 generated by the generator 200 is real data or artificial data. At this time, the discriminator 210 may train the discrimination model to discriminate the authenticity of the artificial data 230 based on the actual surgical image 240 corresponding to the artificial data. The discriminator 210 may retrain the generated model based on the learned discrimination model.

In one embodiment, the discriminator 210 may determine whether the artificial data 230 generated by the generator 200 is real data or artificial data using the discrimination model. If the discriminator 210 determines that the artificial data is not real, that is, the generator 200 does not deceive the discriminator 210, the generator 200 may not deceive the discriminator 210. You can retrain the generation model in a way that reduces. The generator 200 may generate improved artificial data through relearning the generation model. On the contrary, in the case where it is determined that the discriminator 210 is the actual data, that is, when the discriminator 210 is deceived by the generator 200, the discriminator 210 reduces the discrimination model in the direction of reducing the error rate (error). You can relearn. This process is repeated in the generator 200 and the discriminator 210, thereby making artificial data close to reality.

Therefore, when the artificial data approximating the actual surgical image is generated through the mutual learning between the generator 200 and the discriminator 210, the computer may construct a training data set for learning a learning model for recognizing a surgical image. .

Surgical images can only be obtained through actual surgical procedures, and often contain only limited scenes. In actual operation, surgery is performed on a specific surgical site, and since surgery is performed using a surgical tool determined according to the surgical site or the surgical purpose, it is difficult to secure a surgical image including various surgical tools or surgical sites. . Therefore, when learning is performed using such a surgical image, it is difficult to construct sufficient learning data using only the actual surgical image obtained during the actual surgery. Thus, in the present invention, by generating artificial data almost no difference from the actual surgical image, it can be provided as the learning data of the learning model to perform the learning using the surgical image.

3 to 6 are diagrams for explaining embodiments of a method for generating artificial data based on learning according to an embodiment of the present invention.

3 to 6, various embodiments in which a computer acquires a background image and an object image to generate artificial data will be described.

3 is a first embodiment of generating artificial data using first and second actual surgical images including an object and a background.

Referring to FIG. 3, the computer may acquire a first real surgery image 300 including an object and a background, and a second real surgery image 310 including an object and a background. In this case, the computer may extract the object image or the background image included in each of the first real surgery image 300 and the second real surgery image 310, and combine them to generate artificial data.

In an embodiment, the computer may extract the background image in the first real surgery image 300 and the object image in the second real surgery image 310. The computer may generate artificial data 330 close to the actual surgical image by using the generated model (ie, the generator) 320 to extract the extracted background image and the object image. For example, based on the learning in the generation model 320, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding. In this case, the computer may provide an object segmentation 311 in the second real surgery image 310 to the generation model 320 together with the second real surgery image 310 as a correct answer. Since the generation model 320 is trained through the object segmentation 311 which is the correct answer, it is possible to generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 330 generated by the generation model 320 to the discrimination model (ie, the discriminator) 350. The differential model 350 may determine the authenticity of the artificial data 330 based on the actual surgical image 340 corresponding to the artificial data 330. In this case, the computer may additionally provide the object segmentation 331 in the artificial data 330 and the object segmentation 341 in the actual surgical image 340 to learn the differential model 350.

The computer may relearn the generation model 320 or the differential model 350 according to the determination result of the differential model 350. For example, when it is determined that the artificial data 330 is artificial data rather than actual data, the computer may relearn the generated model 320. On the contrary, when it is determined that the artificial data 330 is real data, the computer may relearn the differential model 350.

In another embodiment, the computer may extract the object image in the first real surgery image 300, extract the background image in the second real surgery image 310, and generate artificial data. Since a detailed process thereof is similar to that described with reference to FIG. 3, the description thereof will be omitted.

According to the above-described embodiment, artificial data is generated by using an actual surgical image including different objects, which is performed on rare objects (eg, surgical instruments, blood or bleeding) that do not appear well at the time of surgery. You can accumulate images. In addition, manual labeling can replace expensive segmentation data with artificial data, thereby saving costs.

4 is a second embodiment of generating artificial data using a first real surgery image including only an object and a second real surgery image including both an object and a background.

Referring to FIG. 4, the computer may acquire a first real surgery image 400 including only an object and a second real surgery image 410 including both an object and a background. In this case, the computer may extract the object image from the first real surgery image 400 and the background image from the second real surgery image 410. The computer may generate artificial data 430 close to the actual surgical image by using the generation model (ie, the generator) 420 of the extracted background image and the object image.

For example, based on the training in the generation model 420, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

The computer may provide the artificial data 430 generated by the generation model 420 to the discrimination model (ie, the discriminator) 450. The differential model 450 may determine the authenticity of the artificial data 430 based on the actual surgical image 440 corresponding to the artificial data 430. In this case, the computer may additionally provide the object segmentation 431 in the artificial data 430 and the object segmentation 441 in the actual surgical image 440 to learn the differential model 450.

The computer may relearn the generation model 420 or the differential model 450 according to the determination result of the differential model 450. For example, when it is determined that the artificial data 430 is artificial data rather than real data, the computer may relearn the generated model 320. On the contrary, when it is determined that the artificial data 430 is real data, the computer may relearn the differential model 450.

According to the embodiment described above, flexibility for generating artificial data can be increased, and surgical images for various objects can be accumulated. For example, various types of surgical tool images, surgical images with different positions, directions, and arrangements of surgical tools can be obtained.

FIG. 5 is a third embodiment of generating artificial data using a first real surgery image including only a background and a second real surgery image including both an object and a background.

Referring to FIG. 5, the computer may acquire a first real surgery image 500 including only a background and a second real surgery image 510 including both an object and a background. In this case, the computer may extract the background image from the first real surgery image 500 and the object image from the second real surgery image 510. The computer may generate artificial data 530 close to the actual surgical image by using the generated model (ie, generator) 520 of the extracted background image and the object image.

For example, based on the learning in the generation model 520, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

In this case, the computer may provide the object segmentation 511 in the second actual surgery image 510 with the second actual surgery image 510 as a correct answer to the generation model 520. Since the generation model 520 is trained through the object segmentation 511 which is the correct answer, the generation model 520 may generate artificial data closer to the actual surgical image.

The computer may provide the artificial data 530 generated by the generation model 520 to the discrimination model (ie, discriminator) 550. The differential model 550 may determine the authenticity of the artificial data 530 based on the actual surgical image 540 corresponding to the artificial data 530. In this case, the computer may additionally provide an object segmentation 531 in the artificial data 530 and an object segmentation 541 in the actual surgical image 540 to learn the differential model 550.

The computer may relearn the generation model 520 or the differential model 550 according to the determination result of the differential model 550. For example, when it is determined that the artificial data 530 is artificial data rather than actual data, the computer may relearn the generated model 520. On the contrary, if it is determined that the artificial data 530 is real data, the computer may relearn the differential model 550.

According to the above-described embodiments, it is possible to increase flexibility for generating artificial data and to obtain an image of operating various types of internal organs, blood vessels, tissues, etc. on a specific object (eg, a surgical tool). For example, it is possible to obtain a surgical image for operating different body parts for one surgical tool.

FIG. 6 is a fourth embodiment of generating artificial data using a first real surgery image including only a background and a second real surgery image including only an object.

Referring to FIG. 6, the computer may acquire a first real surgery image 600 including only a background and a second real surgery image 610 including only an object. In this case, the computer may extract the background image from the first real surgery image 600 and the object image from the second real surgery image 610. The computer may generate artificial data 630 close to the actual surgical image by using the generated model (ie, generator) 620 of the extracted background image and the object image.

For example, based on the learning in the generation model 620, the computer may use the background in the background image (ie, organs, blood vessels, tissues, etc., present in a specific area of the body) and the objects in the object image (ie, surgical instruments). Background image and object image can be combined based on the arrangement relationship between the blood, blood, and bleeding.

The computer may provide the artificial data 630 generated by the generation model 620 to the discrimination model (ie, discriminator) 650. The differential model 650 may determine the authenticity of the artificial data 630 based on the actual surgical image 640 corresponding to the artificial data 630. In this case, the computer may additionally provide the object segmentation 631 in the artificial data 630 and the object segmentation 641 in the actual surgical image 640 to learn the differential model 650.

The computer may relearn the generation model 620 or the differential model 650 according to the determination result of the differential model 650. For example, when it is determined that the artificial data 630 is artificial data rather than real data, the computer may relearn the generated model 620. On the contrary, when it is determined that the artificial data 630 is real data, the computer may relearn the differential model 650.

According to the above-described embodiment, a surgical image including various objects and various internal body spaces may be acquired. For example, various types of surgical images may be obtained by freely arranging information on necessary surgical tools or organs, blood vessels, tissues, and the like.

5 and 6, the artificial data is generated by acquiring an actual surgical image (background image) including only a background. In this case, the actual surgical image including only the background does not include an object such as a surgical tool and includes only a background image. In this case, the computer may acquire image frames photographed at a specific position inside the body by the endoscope camera, and remove the object image such as a surgical tool from the image frames to generate a background image including only the background area. Alternatively, the computer may use an image frame obtained by the endoscope entering the body during the operation using the endoscope. In this case, an image frame having only a background image not including an object such as a surgical tool may be obtained. Alternatively, the computer may extract an image frame including only a background from an image frame obtained by a camera entering the body during minimally invasive surgery such as laparoscopic or robotic surgery. In the case of minimally invasive surgery, the operation of the first area is completed and the camera and the surgical tool are also moved from the first area to the second area when the operation is performed by moving to the second area. At this time, there is a constraint that the surgical tool does not move when the camera moves due to the nature of the surgery. Accordingly, when moving from the first area to the second area, the camera moves first and then the surgical tool moves, so that the camera moved first to the second area may acquire an image frame without the surgical tool.

The computer may construct the training data using artificial data generated from the above embodiments. For example, artificial data may be used as training data for training a training model for image recognition. Therefore, according to an embodiment of the present invention, by providing artificial surgical images generated with artificial surgery, deep learning, machine learning, etc. which requires a large amount of learning data together with the actual surgical images, sufficient learning effects can be obtained.

7 is a schematic diagram of a system capable of performing robotic surgery in accordance with one embodiment of the present invention.

According to FIG. 7, the robotic surgical system includes a medical imaging apparatus 10, a server 100, a control unit 30 provided in an operating room, a display 32, and a surgical robot 34. According to an embodiment, the medical imaging apparatus 10 may be omitted in the robot surgery system according to the disclosed embodiment.

In one embodiment, surgical robot 34 includes imaging device 36 and surgical instrument 38.

In one embodiment, the robot surgery is performed by the user controlling the surgical robot 34 using the control unit 30. In one embodiment, the robot surgery may be automatically performed by the controller 30 without the user's control.

The server 100 is a computing device including at least one processor and a communication unit.

The controller 30 includes a computing device including at least one processor and a communication unit. In one embodiment, the control unit 30 includes hardware and software interfaces for controlling the surgical robot 34.

The imaging device 36 includes at least one image sensor. That is, the imaging device 36 includes at least one camera device and is used to photograph an object, that is, a surgical site. In one embodiment, the imaging device 36 includes at least one camera coupled with a surgical arm of the surgical robot 34.

In an embodiment, the image photographed by the photographing apparatus 36 is displayed on the display 340.

In one embodiment, surgical robot 34 includes one or more surgical tools 38 that can perform cutting, clipping, fixing, grabbing operations, and the like, of the surgical site. Surgical tool 38 is used in conjunction with the surgical arm of the surgical robot 34.

The controller 30 receives information necessary for surgery from the server 100 or generates information necessary for surgery and provides the information to the user. For example, the controller 30 displays the information necessary for surgery, generated or received, on the display 32.

For example, the user performs the robot surgery by controlling the movement of the surgical robot 34 by manipulating the control unit 30 while looking at the display 32.

The server 100 generates information necessary for robotic surgery using medical image data of an object previously photographed from the medical image photographing apparatus 10, and provides the generated information to the controller 30.

The controller 30 displays the information received from the server 100 on the display 32 to provide the user, or controls the surgical robot 34 by using the information received from the server 100.

In one embodiment, the means that can be used in the medical imaging apparatus 10 is not limited, for example, other various medical image acquisition means such as CT, X-Ray, PET, MRI may be used.

As described above, when performing a robotic operation, data including various surgical information may be acquired in a surgical image photographed in a surgical process or a control process of a surgical robot. In one embodiment, in the present invention, artificial data may be generated by acquiring the object image or the background image as described above based on the surgical information (ie, the surgical image) obtained in the robot surgery. Of course, in generating artificial data according to an embodiment of the present invention, utilizing the surgical image obtained in the robot surgery process is just one example, the present invention is not limited thereto. In another embodiment, minimally invasive surgery using a laparoscope, endoscope, or the like, and a surgical image obtained when a medical staff directly operates on a patient may also be used to generate artificial data.

8 is a diagram schematically illustrating a configuration of an apparatus 700 for performing an artificial data generating method according to an embodiment of the present invention.

Referring to FIG. 8, the processor 710 may include a connection passage (eg, a bus or the like) that transmits and receives signals with one or more cores (not shown) and a graphic processor (not shown) and / or other components. ) May be included.

The processor 710 according to an embodiment executes one or more instructions stored in the memory 720 to perform the artificial data generation method described with reference to FIGS. 1 to 6.

For example, the processor 710 may acquire a background image and an object image by executing one or more instructions stored in the memory 720, and generate artificial data by combining the background image and the object image.

Meanwhile, the processor 710 may include random access memory (RAM) and read-only memory (ROM) for temporarily and / or permanently storing a signal (or data) processed in the processor 710. , Not shown) may be further included. In addition, the processor 710 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processor, a RAM, and a ROM.

The memory 720 may store programs (one or more instructions) for processing and controlling the processor 710. Programs stored in the memory 720 may be divided into a plurality of modules according to their functions.

The artificial data generation method according to an embodiment of the present invention described above may be implemented as a program (or an application) and stored in a medium to be executed in combination with a computer which is hardware.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. The software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (11)

In the method for generating artificial data performed by a computer,
Obtaining a background image and an object image; And
Generating artificial data by combining the background image and the object image;
The background image includes a specific area inside the body taken by the endoscope camera,
The object image includes a surgical instrument or blood,
The artificial data is generated to approximate the actual surgical image based on the arrangement relationship between the surgical tool or blood in the object image and a specific region inside the body in the background image, and is used as learning data for learning a learning model. Artificial data generation method characterized in that the. The method of claim 1,
Generating the artificial data,
Artificial data generation method characterized in that to generate the artificial data using a generation model (Generative Model) learned based on the actual surgical image. The method of claim 2,
Generating the artificial data,
Determining whether the artificial data is real data or artificial data by using a discrimination model trained to discriminate whether the actual data is artificial data or artificial data; And
And if it is determined that the artificial data is not the actual data according to the determination result, re-learning the generation model. The method of claim 3,
The differentiation model,
The artificial data generation method characterized in that the learning using the artificial data and the object segmentation (segmentation) in the actual surgical image corresponding to the artificial data. The method of claim 3,
And constructing a training data set for surgical image recognition based on the new artificial data generated from the retrained generation model. The method of claim 1,
Acquiring the background image and the object image,
And obtaining the object image from the first real surgical image including an object and a background, and obtaining the background image from a second real surgical image including an object and a background. The method of claim 1,
Acquiring the background image and the object image,
And obtaining the object image from the first real surgical image including an object, and obtaining the background image from a second real surgical image including an object and a background. The method of claim 1,
Acquiring the background image and the object image,
And obtaining the object image from the first real surgery image including an object and a background, and obtaining the background image from a second real surgery image including a background. The method of claim 1,
Acquiring the background image and the object image,
And obtaining the object image from the first real surgical image including an object, and obtaining the background image from the second real surgical image including a background. Memory for storing one or more instructions; And
A processor for executing the one or more instructions stored in the memory,
The processor executes the one or more instructions,
Obtaining a background image and an object image; And
Performing artificial data by combining the background image and the object image;
The background image includes a specific area inside the body taken by the endoscope camera,
The object image includes a surgical instrument or blood,
The artificial data is generated to approximate the actual surgical image based on the arrangement relationship between the surgical tool or blood in the object image and a specific region inside the body in the background image, and is used as learning data for learning a learning model. Apparatus characterized in that the. A computer program, coupled to a computer, which is hardware, stored on a recording medium readable by a computer so as to perform the method of claim 1.

Images