KR20220161597A - Apparatus and method for predicting position informaiton according to movement of tool - Google Patents

Abstract

Provided is a computing device, which includes: a processor; and a memory comprising one or more instructions implemented to be executed by the processor. The processor performs image processing on a first image having a point reached as the object moves to generate first location information of the point, and extracts second location information of a point to be reached as the object moves in a second image paired with the first image based on the first location information based on a machine learning model.

Description

Location information prediction method and apparatus according to object movement

The present application relates to a method for predicting a location of an object, and more particularly, to a method and apparatus capable of accurately predicting a location to which an object will later arrive according to the movement of the object.

Recently, with the development of artificial intelligence learning models, many machine learning models are being used to read images in the medical field. For example, learning models such as Convolutional Neural Networks, Deep neural networks, Recurrent Neural Networks, and Deep Belief Networks are used for detection, classification, and feature detection of medical images. It is applied to learning.

On the other hand, in the medical field where a biopsy or procedure using an endoscope is performed, the position of the lesion inside the human body is recognized by the camera of the endoscope, and when the lesion is extracted or excised using an endoscopic instrument such as forceps, Endoscopic instruments are not properly guided to the location, causing a perforation in the inner wall of the object to be examined.

In contrast, when an endoscopic instrument is used, it is necessary to accurately predict the position according to the movement of the endoscopic instrument based on a mechanical design or a machine learning model that can guide the precise position of the lesion.

Korean Patent No. 10-2167655

An object of the present application is to provide an apparatus and method capable of predicting positional information to be reached by a mechanism in the future according to the movement of the mechanism in an image.

The objects of the present application are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

A computing device according to one aspect of the present invention includes a processor and a memory including one or more instructions implemented to be executed by the processor, wherein the processor has a point reached by a movement of an object. Image processing is performed on one image to generate first location information of the point, and based on a machine learning model, based on the first location information, the object is displayed in a second image paired with the first image. According to the movement, second location information of a point to be reached is extracted.

A computing device according to another aspect of the present invention includes a processor and a memory including one or more instructions implemented to be executed by the processor, wherein the processor includes an input first image and a pair of the first image. , and collects a second image including pixel information of a first marker indicating a point reached as the object moves, and includes pixel information of the second marker based on a machine learning model using the first image. generating a reconstructed image, and matching pixel information of the second marker of the reconstructed image with pixel information of the first marker of the second image based on the machine learning model.

An apparatus for predicting location information based on object motion according to an aspect of the present invention includes a processor and a memory including one or more instructions implemented to be executed by the processor, wherein the processor includes a machine learning model based on A display unit extracts positional information of a point to be reached as the object moves in the biometric image, displays the positional information in the biometric image, and displays the location information.

According to an embodiment of the present invention, there is an effect of predicting positional information of a point reaching a tissue in the human body according to the movement of an object such as an endoscopy instrument based on a machine learning model.

Effects of the present application are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically illustrating an exemplary configuration of an apparatus for generating an image having location information of an object according to an embodiment of the present invention.
2 is a block diagram of a processor for recognizing location information of an object in an image according to an embodiment of the present invention.
3 is a diagram illustratively illustrating a process of extracting location information of an object in a biometric image by a computing device according to an embodiment of the present invention.
4 is a diagram illustrating another process of extracting location information of an object in a biometric image by means of a computing device according to an embodiment of the present invention.
5 is a diagram exemplarily illustrating another process of extracting location information of an object in a biometric image by a computing device according to an embodiment of the present invention.
6 is a flowchart exemplarily illustrating a method for predicting location information according to motion of an object according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the accompanying drawings are only described in order to more easily disclose the contents of the present invention, and those skilled in the art can easily understand that the scope of the present invention is not limited to the scope of the accompanying drawings. You will know.

In addition, terms used in the detailed description and claims of the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the detailed description and claims of the present invention, terms such as "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, It should be understood that it does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In the detailed description and claims of the present invention, terms such as "learning" or "learning" refer to performing machine learning through procedural computing, such as human educational activities. It should be understood that it is not intended to refer to mental processes.

"Real-time image data" used in the detailed description and claims of the present invention can be defined as including a single image (still image) or continuous images (moving image), and has the same meaning as "image" or "image data". can be expressed

The term "image" used in the detailed description and claims of the present invention can be defined as a digital copy or imitation of the shape or specific characteristics of a person or object, and the image is a JPEG image, PNG image, GIF image, It may be, but is not limited to, a TIFF image or any other digital image format known in the art. Also, "image" may be used in the same sense as "photograph".

The "property" used in the claims of the detailed description of the present invention may be defined as a group of one or more descriptive characteristics of an object capable of recognizing a label or displacement of an object recognizable within image data, and "property" can be expressed as a numerical feature.

The apparatus, method, and device disclosed in the present invention may be applied to and used for any real-time biological tissue image capable of supporting diagnosis of a medical image or a disease state of the inside of the abdominal cavity, but is not limited thereto, and is not limited thereto, and time-sequential computed tomography (CT) ), magnetic resonance imaging (MRI), computed radiography, magnetic resonance, vascular endoscopy, optical coherence tomography, color flow Doppler, cystoscopy, diaphanography, echocardiography, fluoresosin angiography ), laparoscopy, magnetic resonance angiography, positron emission tomography, single photon emission computed tomography, X-ray angiography, nuclear medicine, biomagnetic imaging, culposcopy, double Doppler, digital microscope, endoscope, laser , surface scanning, magnetic resonance spectroscopy, radiographic imaging, thermal imaging, and radiofluorescence.

Moreover, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

1 is a diagram schematically illustrating an exemplary configuration of an apparatus for generating an image having location information of an object according to an embodiment of the present invention.

Referring to FIG. 1, an apparatus 100 according to an embodiment of the present invention is a computing device (Computing Device, 110), a display device (Display Device, 130), an examination capable of inspecting or performing a procedure on the inside of a human body, such as an endoscope. Device 150 may be included. The computing device 110 includes a processor 111, a memory unit 113, a storage device 115, an input/output interface 117, a network adapter 118, and a display adapter. (Display Adapter, 119), and a system bus (System bus, 112) that connects various system components including a processor to the memory unit 113, but is not limited thereto. In addition, the apparatus 100 for generating an image having location information of an object may include a system bus 112 for transferring information as well as other communication mechanisms.

A system bus or other communication mechanism may include a processor, a memory that is a computer-readable recording medium, a short-range communication module (eg, Bluetooth or NFC), a network adapter including a network interface or mobile communication module, and a display device (eg, CRT or LCD, etc.), input devices (eg, keyboard, keypad, virtual keyboard, mouse, trackball, stylus, touch sensing means, etc.), and/or subsystems.

The processor 111 may be a processing module that automatically processes using the machine learning model 13, and may be a CPU, an application processor (AP), a microcontroller, etc., but is not limited thereto.

The processor 111 may communicate with the display adapter 119 , for example, a hardware controller for a display device, and display an operation of the real-time bio-image tissue recognition apparatus and a user interface on the display device 130 .

The processor 111 controls the operation of the real-time biometric image tissue recognition device according to an embodiment of the present invention to be described below by accessing the memory unit 113 and executing one or more sequences of instructions or logic stored in the memory unit. .

These instructions may be read into memory from static storage or other computer readable media such as a disk drive. In other embodiments, hard-wired circuitry may be used in place of or combined with software instructions to implement the present disclosure. Logic may refer to any medium that participates in providing instructions to the processor and may be loaded into the memory unit 113 .

System bus 112 is one of several possible types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. indicates an abnormality. For example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standard Association (VESA) local bus, Accelerated Graphics Port (AGP) Buses and Peripheral Component Interconnects (PCI), PCI-Express buses, Personal Computer Memory Card Industry Association (PCMCIA), and Universal Serial Bus (USB).

The system bus 112 can be implemented as a wired or wireless network connection. Processor (111), mass storage device (Mass Storage Device), operating system (Operating System, 113c, 115a), imaging software (Imaging Software, 113b, 115b), imaging data (Imaging Data, 113a, 115c), network A subsystem including an adapter (Network Adapter, 118), system memory, input/output interface (Input/Output Interface, 117), display adapter (119), and display device (Display Device, 130) Each may be included in one or more remote computing devices at physically separate locations, and may be connected through these types of buses in efficiently executing a distributed system.

Transmission media including the wires of the bus may include coaxial cable, copper wire, and optical fibers. In one example, transmission media may take the form of acoustic or light waves generated during radio wave communication or infrared data communication.

Device 100 according to an embodiment of the present invention transmits messages, data, information, and commands including one or more programs (ie, application code) over a network link and network adapter 118, and may receive.

The network adapter 118 may include a separate or integrated antenna for enabling transmission and reception over a network link. The network adapter 118 may access a network and communicate with a remote computing device. The network may include, but is not limited to, at least one of a LAN, WLAN, PSTN, and cellular phone network.

The network adapter 118 may include at least one of a network interface and a mobile communication module for accessing the network. The mobile communication module can access a mobile communication network for each generation (for example, a 2G to 5G mobile communication network).

The program code may be executed by the processor 111 when received and/or may be stored for execution in a disk drive of the memory unit 113 or in a non-volatile memory of a type other than the disk drive.

The computing device 110 may be a variety of computer readable recording media. A readable medium can be any of a variety of media that can be accessed by a computing device, including, for example, volatile or non-volatile media, removable media, and non-removable media. removable media), but is not limited thereto.

The memory unit 113 may store an operating system, drivers, application programs, data, and databases necessary for the operation of the biometric tissue recognition device according to an embodiment of the present invention, but is not limited thereto. In addition, the memory unit 113 may include computer readable media in the form of volatile memory such as RAM (Random Access Memory), ROM (Read Only Memory) and non-volatile memory such as flash memory, and may also include a disk drive. For example, it may include, but is not limited to, a hard disk drive, a solid state drive, an optical disk drive, and the like. In addition, the memory unit 113 and the storage device 115 typically store data such as imaging data (Imaging Data, 113a, 115a) such as a biometric image of an object, which can be immediately accessed to be operated by the processor 111. It may include program modules such as imaging software 113b and 115b and operating systems 113c and 115c.

The machine learning model 13 may be inserted into the processor, memory unit 113 or storage device 115 . The machine learning model at this time includes an autoencoder model, and the autoencoder may be a variational autoencoder having a neural network. In addition, the machine learning model may include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc., which are one of the machine learning algorithms. Not limited.

The examination device 150 may include various devices capable of examining the inside of the human body, such as a flexible endoscope, a rigid endoscope, and a cystoscope, and may include a camera 151 capable of visually photographing and confirming the inside of the human body and a human body for tissue examination. An endoscopic instrument (Tool, 153) for excising or excising internal tissue, and a light attached to the endoscopic instrument 153 to guide the endoscopic instrument 153 so that the endoscopic instrument 153 can reach the exact position of the tissue A guider (Light guider, 155) may be included.

In an embodiment, the endoscopic instrument 153 is an object for recognizing location information of a point reaching a specific position of a tissue in the human body by a user's manipulation, and the object may be composed of various components constituting the endoscopic instrument, According to the movement of the object, the apparatus according to an embodiment of the present invention predicts a position where the object reaches the tissue.

The camera unit 151 includes an image sensor (not shown) that captures an image of an object and photoelectrically converts the image into an image signal, and captures a biometric image of tissue in the human body in real time. The captured biometric image (image data) is provided to the processor 111 through the input/output interface 117 and processed based on the machine learning model 13 or stored in the memory unit 113 or storage device 115. can

The target object (eg, an endoscopic instrument) includes forceps, clamps, scissors, and the like, and may be used to examine tissues inside the human body or to excise or excise tissues of a target for examination.

The light guider 155 is attached to the endoscopic instrument and guides the movement direction of the endoscopic instrument. The light guider includes a light source (not shown) therein so that the light generated from the light source targets the tissue lesion so that the endoscopic instrument can accurately reach the location of the lesion. At this time, a marker X such as a light spot is formed by light on the tissue lesion. The biometric image taken by the camera according to the turn-on/turn-off of the light source attached to the light guider is an image in which tissue lesions appear along with the object or a marker (X) is formed along with the object. It may include images showing tissue lesions.

The apparatus 100 for generating an image having location information of an object according to the present invention is not limited to laptop computers, desktop computers, and servers, and is implemented in a computing device or system capable of processing data and executing arbitrary commands. It can be, and can be implemented in other computing devices and systems through the Internet network. Also, the device 100 may be implemented in a variety of ways, including software including firmware, hardware, or a combination thereof. For example, functions to execute in various ways may be performed by components implemented in various ways, including discrete logic components, one or more application specific integrated circuits (ASICs), and/or program control processors.

2 is a block diagram of a processor for recognizing location information of an object in an image according to an embodiment of the present invention.

Referring to FIG. 2 , the processor 600 may be the processor 111 of FIG. 1 , receives training data to train machine learning models 211a, 213a, 215a, and 230a, and based on the received training data Attribute information of learning data can be extracted. The learning data may be biometric image data (plural biometric image data or single biometric image data) or attribute information data extracted from biometric image data.

In one embodiment, the attribute information extracted from the biometric image data may be label information for classifying objects detected in the biometric image data. For example, the label may be a category classified by organs in the body, such as the liver, pancreas, and gallbladder, represented in the biometric image data, or a category classified by tissues in the body, such as blood vessels, lymph, and nerves. It may be a category classified as a lesion of an internal tissue such as a tumor. In one embodiment, the label information may include location information of an object (eg, lesion), and the location information of the object may be expressed in 2-dimensional coordinates (x,y) or 3-dimensional coordinates (x,y,z). can Also, the label information may include size information of an object (eg, a lesion), and a weight or order may be assigned to the label based on the weight and meaning of the object recognized in the biometric image data.

The processor 600 may include a data processing unit 210 and an attribute information model learning unit 230 .

The data processing unit 210 receives the bio image data and the attribution information data of the bio image for the attribution information model learning unit 230 to learn, and uses the received bio image data and the attribution information data of the bio image to learn the attribution information model. Transforming into suitable data or image processing can be performed. The data processor 210 may include a label information generator 211 , a data generator 213 , and a feature extractor 215 .

The label information generator 211 may generate label information corresponding to the received biometric image data using the first machine learning model 211a. The label information may be information about one or more categories determined according to objects recognized in the received biometric image data. In one embodiment, the label information may be stored in the memory unit 113 or the storage device 115 together with information on biometric image data corresponding to the label information.

The data generating unit 213 may generate data to be input to the attribute information model learning unit 230 including the machine learning model 230a. The data generator 213 may generate input data to be input to the third machine learning model 230a based on a plurality of frame data included in the received biometric image data using the second machine learning model 213a. have. The frame data may refer to each frame constituting the biometric image, RGB data or pixel data of each frame constituting the biometric image, and data obtained by extracting characteristics of each frame or to each frame. It may refer to data expressing characteristics of a vector.

The attribute information model learning unit 230 includes a third machine learning model 230a, and transmits data including image data and label information generated and extracted by the label information generator 211 and the data generator 213 to the third machine learning model 230a. Attribute information of the biometric image data may be extracted through fusion learning by inputting the data to the machine learning model 230a. Attribute information refers to information related to characteristics of a target image recognized from the biometric image data. For example, the attribute information may be a label for classifying a subject within the biometric image data, for example, lesion information such as a polyp. If an error occurs in the attribute information extracted by the attribute information model learning unit, coefficients or connection weight values used in the third machine learning model 230a may be updated.

3 is a diagram illustratively illustrating a process of extracting location information of an object in a biometric image by a computing device according to an embodiment of the present invention.

Referring to FIG. 3 , the image data set learned by the computing device includes a first image 40 having a mark X indicating a point at which the object 40a moves and reaches the lesion of the tissue and the object 10a. ) may include a second image 10 that does not have a mark indicating a point at which the tissue lesion is moved.

The first image 40 and the second image 10 may be generated by the device 100 according to one embodiment of the present invention described above with reference to FIG. 1 . For example, when an operator manipulates an object 10a or 40a, which is an endoscopic instrument, to reach a tissue lesion, an attached light source may be turned on to point a point at which the object is reached in order to guide a movement path of the object. A marker (X) may be formed in the lesion. When the camera captures this, it is expressed as the first image 40 . In contrast, when a light source attached to guide a movement path of an object is not turned on, a marker capable of pointing to a point to which the object is reached is not formed on the lesion. When the camera captures this, it is expressed as the second image 10 . Accordingly, the first image 40 and the second image 10 form a pair of images that correspond to each other except for differences in markers.

The first image 40 may be subjected to image processing by a processor 700 such as a digital signal processor (DSP) to generate label information 50 from the first image 40 . At this time, the label information 50 may include a category for classifying an object having a marker recognized in the biometric image described above, location information of the object, size or shape of the object, bitmap information of the first image, and the like. The label information 50 may include first location information of a location of an object where a marker is formed, that is, a point where the object 40a will reach a tissue in the future.

The second image 10 is input to the machine learning model 710, and the processor 700 may extract attribute information 30 of the input second image based on the machine learning model 710 included therein. have. In this case, the attribute information 30 may include second location information of a point where the object will reach the tissue in the future according to the location of the object 10a shown in the second image 10, the direction and angle the object points, and the like. .

In the second image 10 , the second location information of the point where the object will reach the tissue may be extracted in various ways. For example, using the position values (x ₂ , y ₂ ) of the first location information of the point where the object in the first image 40 paired with the second image 10 reaches the tissue and the machine learning model In the generated second image 10, the location value of the first location information and the location value of the second location information are compared with the location values (x ₁ , y ₁ ) of the second location information of the point where the object will reach the tissue When this difference is found, the machine learning model is repeatedly learned using a back-propagation method, and second location information may be extracted.

4 is a diagram illustrating another process of extracting location information of an object in a biometric image by means of a computing device according to an embodiment of the present invention.

Referring to FIG. 4 , since an image data set learned by a computing device is similar to the image data set of FIG. 3 , a description thereof will be omitted. The second location information of the point where the object 10a will reach the tissue in the second image 10 may be extracted by image processing the first image and the second image as a bitmap and comparing them.

Specifically, the first image 40 paired with the second image 10 is image-processed by the processor 700 to generate a first bitmap 40b, and the second image 10 is also a machine learning model. Based on step 710, the image is processed by the processor 700 to generate a second bitmap 10b.

If a loss or error occurs by comparing the bit values of the first bitmap 40b generated by the processor and the second bitmap 10b generated based on the machine learning model, respectively, backpropagation ) method, the machine learning model is repeatedly learned to extract the second location information.

3 and 4, the machine learning model 710, although not shown, can be executed by being input to a computer-readable recording medium, input into the memory unit 113 or storage device 115, and by the processor 700. It can be operated and executed.

In this way, the extraction of attribute information of the object from the biometric images can be performed by a computing device, and the computing device is a device that receives a biometric image dataset captured in real time as learning data and learns as a result of performing a machine learning model. data can be generated. In describing each operation belonging to the method according to the present embodiment, if the description of the subject is omitted, it will be understood that the subject of the corresponding operation is the computing device.

As in the above embodiments, it can be clearly understood that the operation and method of the present invention can be achieved through a combination of software and hardware or can be achieved only by hardware. Objects of the technical solution of the present invention or parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable recording medium. The machine-readable recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software.

Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler.

The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa. The hardware device may include a processor such as a CPU or GPU coupled to a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and may transmit and receive signals with external devices. A communication unit may be included. In addition, the hardware device may include a keyboard, mouse, and other external input devices for receiving commands written by developers.

5 is a diagram exemplarily illustrating another process of extracting location information of an object in a biometric image by a computing device according to an embodiment of the present invention.

Referring to FIG. 5, the image data set learned by the computing device includes a first image 60 that does not have a mark indicating a point at which the object 60a moves and reaches the tissue lesion, and the tissue lesion that is moved by the object 70a. A second image 70 having a mark X indicating a point where the lesion is reached may be included. The first image 60 and the second image 70 may be generated by the device 100 according to an embodiment of the present invention in the same way as the method described above with reference to FIG. 3 .

The first image 60 may be input to the processor 700 and a restored image 80 may be generated based on the machine learning model 710 . The machine learning model 710 may include a U-net, an autoencoder, or a deconvolution neural network. As an example, an autoencoder may be implemented in a probabilistic manner to describe an object in a latent space as a Variational Autoencoder.

Accordingly, the variational autoencoder can describe a probability distribution for each latent attribute. A variational autoencoder may use the encoder 11 and decoder 13 during workflow operation. When high-dimensional data is input to the encoder 11 of the autoencoder, the encoder performs encoding to convert the high-dimensional data into a low-dimensional latent variable (Z). In the embodiment, the high-dimensional data may be the first image 60, but any data may be used as long as it is data capable of restoring the mark of the second image. The decoder 13 may output restored high-dimensional data by decoding the latent variable Z. In an embodiment, the reconstructed high-dimensional data may be a reconstructed image including pixel information. In this case, the pixel information is information of a pixel where the marker X is generated, and the pixel information may include gray value 1 or color 1 data of the corresponding pixel.

The loss calculator (710a) can calculate the difference between the comparison data stored in the memory (not shown) and the restored high-dimensional data using a loss function, and the autoencoder uses back-propagation. It can be repeatedly learned to minimize the loss function using In the embodiment, the comparison data may be a pair with the first image 60 and the second image 70 including pixel information on which the marker X is formed. Calculation of the difference between the comparison data and the reconstructed high-dimensional data is calculated by comparing the marker pixel information generated in the first image with the marker pixel information in the second image, and specifically, the gradation in the marker pixels of the first image and the second image. It can be calculated by comparing value or color data.

In FIG. 5 , although not shown, the machine learning model 710 can be input to and executed on a computer-readable recording medium, input to the memory unit 113 or storage device 115, and operated and executed by the processor 700. can

6 is a flowchart exemplarily illustrating a method for predicting location information according to motion of an object according to another embodiment of the present invention.

Referring to FIG. 6 , a first image having a mark X indicating a point at which a tissue lesion is reached as an object moves, and a first image having a mark X indicating a point at which a tissue lesion is reached as an object moves The second image is taken by an apparatus for generating an image having location information of an object according to an embodiment of the present invention.

In step S610, a first image and a second image paired with the first image may be collected. The first image and the second image may be moving image images of organs or tissues inside the human body captured in real time using a camera such as a flexible endoscope or laparoscopic endoscope. Anything can apply.

In step S630, the first image may be image-processed by the processor to generate first location information of a point where a mark is formed. At this time, the first location information may include a two-dimensional plane coordinate value. Also, the first location information may be generated based on a bitmap of the first image generated through image processing by the processor.

In step S650, based on the machine learning model, second location information of a point to be reached according to the movement of the object in the second image may be extracted. The machine learning model may be composed of a neural network for extracting location information of points. At this time, the second location information may be extracted based on the previously generated first location information. In addition, the second location information may be extracted from a bitmap of a second image generated by image processing by a processor based on a machine learning model.

Accordingly, based on the extracted second location information, a point to be reached according to the motion of the object in the image may be predicted.

Location information extraction according to the movement of the object may be performed by a computing device, which is a device that receives the above-described first image and second image as learning data, and generates learned data as a result of performing the machine learning model. can do. In describing each operation belonging to the method according to the present embodiment, if the description of the subject is omitted, it will be understood that the subject of the corresponding operation is the computing device.

As in the above embodiments, it can be clearly understood that the operation and method of the present invention can be achieved through a combination of software and hardware or can be achieved only by hardware. Objects of the technical solution of the present invention or parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable recording medium. The machine-readable recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software.

Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler.

The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa. The hardware device may include a processor such as a CPU or GPU coupled to a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and may transmit and receive signals with external devices. A communication unit may be included. In addition, the hardware device may include a keyboard, mouse, and other external input devices for receiving commands written by developers.

As described above, the embodiments according to the present invention have been reviewed, and the fact that the present invention can be embodied in other specific forms in addition to the above-described embodiments without departing from the spirit or scope is apparent to those skilled in the art. It is self-evident to Therefore, the embodiments described above are to be regarded as illustrative rather than restrictive, and thus the present invention is not limited to the above description, but may vary within the scope of the appended claims and their equivalents.

110: computing device
111,600,700: processor
113: memory unit
115: storage device
117: I/O interface
118: network adapter
119: display adapter
130: display device
10,60: 1st image
40,80: 2nd image

Claims (10)

processor; and
a memory comprising one or more instructions implemented to be executed by the processor, the processor comprising:
image processing of a first image having a point reached as the object moves to generate first location information of the point;
A computing device that extracts second location information of a point to be reached as the object moves in a second image paired with the first image based on the first location information based on a machine learning model. According to claim 1,
The second location information of the point is repeatedly learned and extracted by the machine learning model so that a location value of the first location information and a location value of the second location information are the same. According to claim 1,
A computing device characterized in that a marker is displayed at a point reached by the movement of the object. According to claim 1,
The first location information and the second location information are each a two-dimensional coordinate value, characterized in that the computing device. According to claim 1,
The first location information is generated based on a bitmap of the first image generated through image processing by the processor,
The second location information is extracted from a bitmap of the second image generated by image processing by the processor based on the machine learning model. According to claim 5,
The second location information is repeatedly learned and extracted by the machine learning model so that the bitmap of the first image and the bitmap of the second image are the same. processor; and
a memory comprising one or more instructions implemented to be executed by the processor, the processor comprising:
Based on a machine learning model, extracting positional information of a point to be reached as the object moves within the biometric image;
A display unit for displaying the location information in the biometric image
A device for predicting location information according to object movement. processor; and
a memory comprising one or more instructions implemented to be executed by the processor, the processor comprising:
Collecting an input first image and a second image including pixel information of a first marker that is paired with the first image and indicates a point to be reached as the object moves;
Using the first image, a reconstructed image including pixel information of a second marker is generated based on a machine learning model;
A computing device configured to match pixel information of the second marker of the reconstructed image with pixel information of the first marker of the second image based on the machine learning model. According to claim 8,
The machine learning model comprises a U-net, an autoencoder, and a deconvolution neural network. According to claim 8,
The pixel information of the first marker and the second marker includes gradation values or color data of pixels where the markers are displayed.

Images