WO2015163171A1

WO2015163171A1 - Image processing apparatus and method and surgical operation system

Info

Publication number: WO2015163171A1
Application number: PCT/JP2015/061311
Authority: WO
Inventors: 真人山根; 恒生林
Original assignee: ソニー株式会社
Priority date: 2014-04-24
Filing date: 2015-04-13
Publication date: 2015-10-29
Also published as: JPWO2015163171A1; US20190387135A1; US10440241B2; JP6737176B2; EP3136719A4; US20170046847A1; EP3136719A1; US11245816B2; CN106233719B; CN106233719A

Abstract

This technique relates to an image processing apparatus, an image processing method and a surgical operation system for reducing the latency, thereby enabling a captured image to be displayed in nearly real time. A DMA controller (51) of a CPU (31) divides image data, which have been inputted via an interface card (34), by a number of GPU cards (35-1, 35-2) in the horizontal direction and allocates the thus divided image data to the GPU cards (35-1, 35-2). In each of the GPU cards (35-1, 35-2), the image data are subjected to a time division process in the vertical direction. In this way, multiple GPU cards (35-1, 35-2) are used to enhance the speed of a process related to the displaying of image data, thereby reducing the latency, thereby achieving a high speed displaying. This technique can be applied to an endoscopic camera, a surgical operation microscope and the like.

Description

Image processing apparatus and method, and surgical system

The present technology relates to an image processing apparatus and method, and a surgical system, and more particularly, to an image processing apparatus and method, and a surgical system that can realize image display with low latency.

In recent years, endoscopic surgery has been performed in place of conventional laparotomy in medical practice. As an image processing apparatus used in endoscopic surgery or the like, it is particularly required to realize image display with low latency.

On the other hand, a technique has been proposed that enables a captured image to be displayed at high speed with a minimum time lag.

For example, a technique of displaying an image at high speed by dividing an image in the vertical direction and using a plurality of processors to perform parallel processing for each divided area has been proposed (see Patent Document 1).

JP-A-2-040688

However, in the technique described in Patent Document 1 described above, when an image is divided into line units, in the case of a configuration having independent processing memories, such as GPU (Graphics Processing Unit), overlap is made in line units. And the overhead becomes large.

As a result, since the overhead is processed, the number of processing lines as a whole increases, so that the amount of calculation increases and the processing speed cannot be improved.

The present technology has been made in view of such a situation. In particular, the image is divided in the horizontal direction and assigned to a plurality of processors, and each processor performs time-division processing on the assigned area in the vertical direction. In addition, for each area divided in the vertical direction, the largest overhead is set in the head area and sequentially processed so that the captured image can be displayed at high speed.

An image processing apparatus according to an aspect of the present technology includes a plurality of arithmetic processing units that perform time-division processing on a range obtained by dividing an image obtained by imaging a patient's surgical part in a vertical direction, and the arithmetic processing unit includes: The image divided in the horizontal direction is time-divided in the vertical direction by the number of the arithmetic processing units and processed.

The plurality of arithmetic processing units are configured by a plurality of GPUs (Graphical Processing Unit), and the arithmetic processing unit can perform processing on the image divided in the horizontal direction by the number of GPUs. .

The process applied to the image can be a process of applying an n-stage filter.

The n stages of filters sequentially process a range obtained by dividing the image in the vertical direction by time division sequentially from the uppermost stage range in the vertical direction downward.

Based on the amount of processing performed on the image calculated based on the number of divisions in the horizontal direction and the number of divisions in the vertical direction of the image, and the processing speed of the arithmetic processing unit, A timing control unit for controlling the timing may be further included.

Of the range in which the image is time-divided in the vertical direction, the processing range in the first period includes a reference pixel required for processing in the second period after the first period. can do.

The arithmetic processing unit may include a memory for buffering a processing result, and in the processing of the second period, from the processing result of the first period buffered in the memory, An arithmetic process can be executed using a processing result corresponding to the reference pixel.

The arithmetic processing unit may include a memory for buffering a processing result, and the vertical stage processing by the filter at each stage in a range obtained by dividing the image in the vertical direction. The range may be a range including the number of reference pixel lines required for the processing of the filter in the processing range of the second stage or less in the vertical direction, and the arithmetic processing unit may include processing by the filter. In the processing using the reference pixel, the calculation processing is performed using the processing result corresponding to the reference pixel from the processing result of the filter processing up to the previous stage buffered in the memory. It can be made to execute.

The arithmetic processing unit can perform at least an enlargement process on an image obtained by imaging the surgical site of the patient.

The image obtained by imaging the surgical site of the patient can be an image taken by an endoscope.

The image obtained by imaging the surgical site of the patient can be an image taken by a microscope.

An image processing method according to an aspect of the present technology is an image processing method for an image processing apparatus including a plurality of arithmetic processing units that perform time-division processing on an image obtained by imaging a patient's surgical part for each range divided in the vertical direction. The arithmetic processing unit performs processing by time-dividing the image divided in the horizontal direction in the vertical direction by the number of the arithmetic processing units.

The image can be an image captured by an endoscope.

The image can be an image taken with a microscope.

An operation system according to an aspect of the present technology includes an imaging device that images a surgical part of a patient, and a plurality of arithmetic processing units that perform processing in a time-sharing manner for each range obtained by dividing an image captured by the imaging device in a vertical direction The arithmetic processing unit includes an image processing apparatus that performs time-division processing on the image divided in the horizontal direction by the number of the arithmetic processing units.

In one aspect of the present technology, processing is performed in a time-sharing manner for each range in which an image obtained by imaging a patient's operative part is vertically divided by a plurality of arithmetic processing units, and by the number of the arithmetic processing units, The image divided in the horizontal direction is processed by time division in the vertical direction.

According to one aspect of the present technology, display processing of captured images can be realized with low latency, and the captured images can be displayed at high speed in real time.

It is a block diagram explaining the composition of the 1 embodiment of the image processing device to which this art is applied. It is a figure explaining the process by the image processing apparatus of FIG. It is a figure explaining the difference between the conventional image processing and the image processing of this technique. FIG. 2 is a diagram illustrating that the image processing apparatus of FIG. 1 performs parallel processing by dividing the number of GPU cards in the horizontal direction. FIG. 2 is a diagram for explaining that the image processing apparatus of FIG. 1 divides in the horizontal direction by the number of GPU cards and performs parallel processing and time-division processing in the vertical direction. It is a figure which shows an example of the filter which processes an image. It is a figure explaining the relationship between the attention pixel and reference pixel in a filter process. It is a figure explaining the overhead generate | occur | produced when filtering in each process area | region. It is a figure explaining an example of the concrete overhead generate | occur | produced when filtering in each process area | region. It is a figure explaining the setting method of the processing range in each filter process in the image processing apparatus of FIG. 3 is a flowchart for explaining low latency display processing by the image processing apparatus of FIG. 1. It is a figure explaining the method of calculating the processing amount of a horizontal direction. It is a figure explaining the method of calculating the processing amount of a perpendicular direction. It is a figure explaining that processing time changes according to a processing range. It is a figure explaining adjusting the number of lines output in a processing range, and equalizing processing time for every processing range. It is a figure explaining equalizing the processing time for every processing range by performing the processing for which real-time property is not calculated | required when the processing time difference which arises for every processing range arises in the vacant time. And FIG. 11 is a diagram illustrating a configuration example of a general-purpose personal computer.

<Configuration example of image processing apparatus>
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus to which the present technology is applied.

The image processing apparatus 11 in FIG. 1 receives input of image data captured by an imaging apparatus such as a camera (not shown), performs various processes, and then outputs the image data to a display apparatus such as a display (not shown). Is displayed.

More specifically, the image processing apparatus 11 includes a CPU (Central Processing Unit) 31, a main memory 32, a bus 33, an IF (Interface) card 34, and GPU (Graphical Processing Units) cards 35-1 and 35-2. ing. Note that the GPU cards 35-1 and 35-2 are simply referred to as the GPU card 35 unless otherwise distinguished, and the other configurations are also referred to in the same manner.

A CPU (Central Processing Unit) 31 controls the overall operation of the image processing apparatus 11. In addition, the CPU 31 includes a DMA (Direct Memory Access) controller 51. Note that DMA here refers to Direct Memory Access, and is directly managed between the IF card 34, the main memory 32, and the GPU card 35 via the bus 33 without being directly managed by the CPU 31. It represents the operation of transferring data. That is, the DMA controller 51 controls the transfer source, the transfer destination, and the transfer timing in the DMA transfer operation that is not directly managed by the CPU 31.

More specifically, the DMA controller 51 temporarily stores image data supplied as an input signal from a camera (not shown) via the IF card 34 and the bus 33 in the main memory 32. The DMA controller 51 also selects the main memory 32 according to the image data stored in the main memory 32, the processing capabilities of the processors 92-1 and 92-1 of the GPU cards 35-1 and 35-2, and the processing contents. The image data stored in is divided. Further, the DMA controller 51 assigns a timing for reading the divided image data for each range and a timing for storing the processed image data again. Further, the DMA controller 51 sequentially supplies the image data divided at the assigned timing to the GPU cards 35-1 and 35-2 and stores the processed image data in the main memory 32 sequentially. Then, the DMA controller 51 outputs the processed image data stored in the main memory 32 to the display (not shown) as an output signal via the bus 33 and the IF card 34 for display.

The IF (Interface) card 34 includes a camera IF 71, a display IF 72, and a PCIe (Peripheral Component Interconnect Express) bridge 73. When the camera IF 71 of the IF card 34 receives image data supplied as an input signal from a camera (not shown) under the management of the DMA controller 51, the image data is supplied to the main memory 32 via the PCIe bridge 73 and the bus 33. To do. The display IF 72 of the IF card 34 outputs processed image data supplied from the main memory 32 via the bus 33 and the PCIe bridge 73 to the display (not shown) under the management of the DMA controller 51. Output as a signal.

The GPU cards 35-1 and 35-2 include PCIe bridges 91-1 and 91-2, processors 92-1 and 92-2, and memories 93-1 and 93-2, respectively. The GPU card 35 temporarily stores image data supplied from the main memory 32 via the bus 33 and the PCIe bridge 91 under the management of the DMA controller 51 of the CPU 31 in the memory 93. Then, the processor 91 performs predetermined processing while sequentially reading the image data stored in the memory 93, buffers the processing result in the memory 93 as necessary, and also via the PCIe bridge 91 and the bus 33. Output to CPU 31. In FIG. 1, an example in which there are two GPU cards 35 is shown, but two or more GPU cards may be used.

<About image processing>
Next, image processing by the image processing apparatus 11 of FIG. 1 will be described with reference to FIG.

In FIG. 2, as indicated by an arrow from the upper left, image data (leftmost part), for example, consisting of a pixel array of a Bayer array, captured by a camera (not shown), is defect correction processing and RAWNR (Noise Reduction). ) Processing, an R (red) image, a G (green) image, and a B (blue) image (RGB image in the figure) are generated by demosaic processing. Further, the demosaiced R (red) image, G (green) image, and B (blue) image are subjected to an image quality enhancement process and then subjected to an enlargement process to form R ( A red) image, a G (green) image, and a B (blue) image are generated. The R (red) image, G (green) image, and B (blue) image generated in this way are output as output signals to a display unit such as a display (not shown) and displayed.

<Low latency>
When the image processing as described above is executed in units of frames as in the prior art, the processing becomes a time chart as shown in the upper part of FIG. Here, it is assumed that only two types of processes A and B are performed on the image. In the upper part of FIG. 3, it is assumed that there is one GPU card identical to the GPU card 35.

That is, from time t0 to t1, image data supplied as an input signal by the DMA controller 51 is written and stored in the main memory 32 as indicated by “DMA INPUT # 0” in the drawing.

From time t1 to t2, as indicated by “Kernel A # 0” in the figure, the DMA controller 51 supplies the image data stored in the main memory 32 to the GPU card 35, and the processor 92 of the GPU card 35 Thus, process A is executed.

From time t3 to t4, as indicated by “Kernel B # 0” in the figure, the DMA controller 51 supplies the image data stored in the main memory 32 to the GPU card 35, and the processor 92 of the GPU card 35 Thus, the process B is executed, and the process result is returned to the main memory 32.

From time t5 to t6, as indicated by “DMA OUTPUT # 0” in the figure, the DMA controller 51 reads out and outputs the image data subjected to the processes A and B stored in the main memory 32. Is done.

In this case, if one GPU card 35 processes the entire frame and then tries to display it, it will not be displayed as an image until the processing result for one frame is generated, and the processing time will be enormous. And the latency becomes large, and there is a risk of delay in display.

Therefore, in the image processing apparatus 11 of FIG. 1, as shown in the lower part of FIG. 3, the frame is divided into several ranges in the vertical direction, and the processing is divided into small parts to reduce the latency. . The lower part of FIG. 3 shows an example in which processing is performed when each of the frames is divided into three to form image data # 0 to # 2.

That is, in the lower part of FIG. 3, image data # 0 supplied as an input signal by the DMA controller 51 is written to the main memory 32 as indicated by “DMAｔINPUT よう # 0” in the figure from time t21 to t22. And memorized.

At time t31 to t32, the image data # 0 stored in the main memory 32 is supplied to the GPU card 35 by the DMA controller 51, as indicated by “processing A # 0” in the figure, and the GPU card 35 Processing A is executed by the processor 92.

At this time, image data # 1 supplied as an input signal by the DMA controller 51 at time t22 to t23 in parallel with the processing of “Processing A # 0”, as indicated by “DMA INPUT # 1” in the drawing. Is stored in the main memory 32.

At times t33 to t34, as indicated by “Processing B B # 0” in the figure, the DMA controller 51 supplies the image data stored in the main memory 32 to the GPU card 35, and the processor 92 of the GPU card 35 Thus, the process B is executed, and the process result is returned to the main memory 32.

From time t51 to t52, as indicated by “DMA OUTPUT # 0” in the figure, the DMA controller 51 outputs the image data # 0 subjected to the processes A and B stored in the main memory 32. .

In parallel with the processing of “DMA OUTPUT # 0”, the image data # 1 stored in the main memory 32 by the DMA controller 51 as shown by “processing A # 1” in the figure at time t35 to t36. Is supplied to the GPU card 35, and the process A is executed by the processor 92 of the GPU card 35.

Further, in parallel with the processing of “Process A # 1”, the image data # 2 supplied as an input signal by the DMA controller 51 as shown by “DMA INPUT # 2” in the figure at time t24 to t25. Store in the main memory 32.

At time t37 to t38, as indicated by “processing B # 1” in the figure, the DMA controller 51 supplies the image data # 1 stored in the main memory 32 to the GPU card 35, and the GPU card 35 Processing B is executed by the processor 92, and the processing result is returned to the main memory 32.

From time t53 to t54, as indicated by “DMA OUTPUT # 1” in the figure, the DMA controller 51 outputs the image data # 1 subjected to the processes A and B stored in the main memory 32. .

In parallel with the processing of “DMA OUTPUT # 1”, the image data # 2 stored in the main memory 32 by the DMA controller 51 as shown by “processing A # 2” in the figure at time t39 to t40. Is supplied to the GPU card 35, and the process A is executed by the processor 92 of the GPU card 35.

At time t41 to t42, as indicated by “Processing B B # 2” in the figure, the DMA controller 51 supplies the image data # 2 stored in the main memory 32 to the GPU card 35, and the GPU card 35 Processing B is executed by the processor 92, and the processing result is returned to the main memory 32.

At time t55 to t56, as indicated by “DMA OUTPUT # 2” in the figure, the DMA controller 51 outputs the image data # 2 subjected to the processes A and B stored in the main memory 32. .

By such processing, image data # 0 to # 2 are time-division processed, and “DMA INPUT”, “Process A”, “Process B”, and “DMA OUTPUT” are processed in parallel as necessary. Thus, the latency can be reduced as a whole. In addition, since the image data # 0 to # 2 subjected to the processes A and B are partially displayed when the process is completed, the display is also expeditious and the latency is reduced. Is possible.

<Horizontal division>
As described above, the image is divided in the vertical direction to reduce the latency, and the image processing apparatus 11 in FIG. 1 is provided with a plurality of GPU cards 35. To do. That is, when the image data P1 shown in the left part of FIG. 4 is input, it is divided in the horizontal direction as shown in the upper right part of FIG. 4, and as described with reference to FIG. Time-sharing process.

In the upper right part of FIG. 4, an area Z1 shown on the left side of the image data P1 is a processing range by “GPU # 0” corresponding to the GPU card 35-1, and is shown on the right side of the image data P1. The area Z2 to be processed is the processing range by “GPU # 1” corresponding to the GPU card 35-2. In the lower right part of FIG. 4, an example of the vertical division method in the conventional parallel processing is shown, and the processing by “GPU # 0” corresponding to the GPU card 35-1 indicated by the area Z11 in the upper stage. The lower part is the processing range by “GPU # 1” corresponding to the GPU card 35-2 indicated by the area Z12. That is, in the lower right part of FIG. 4, an example in which the GPU card 35 is divided into two in the vertical direction is shown.

<Vertical division>
Further, as shown in FIG. 5, the image processing apparatus 11 of FIG. 1 has a GPU card 35-1 when the areas Z1 and Z2 of the image P1 are each divided into ranges C1 to C4 in the vertical direction from the top. In the region Z1, time division processing is performed in the order of the ranges C1 to C4 (in order from the top to the bottom). Similarly, the image processing apparatus 11 in FIG. 1 controls the GPU card 35-2 to perform time division processing in the order of the ranges C1 to C4 in the region Z2.

In this way, a plurality of (two in FIG. 5) GPU cards 35 are processed in parallel in the horizontal direction, and further, image processing is performed by time-sharing processing in each GPU card 35 in the vertical direction. It is possible to increase the speed and to reduce the latency.

<Overhead>
The process executed on the image by the processor 92 of the GPU card 35 is generally a filter process. For example, consider a case where a Gaussian filter as shown in FIG. 6 needs to be processed three times for each pixel. In FIG. 6, a Gaussian filter of 3 pixels × 3 pixels is used. The pixel of interest is 4/16, the pixel of interest is 4 pixels above, below, left and right, and 2/16 is diagonally above and below the pixel of interest. 1/16 is a filter that sets each weighting factor as a weighting coefficient and calculates these sums as pixels.

In this case, as shown in the upper left part of FIG. 7, the first filtering process is performed on a range of 5 pixels × 5 pixels around the target pixel P to be processed. In this case, in order to filter the pixels at the upper, lower, left, and right end portions of the range of 5 pixels × 5 pixels, the pixels adjacent to the opposite side toward the target pixel by one pixel at the end portions are also required. In addition, for the corner pixels in the range of 5 pixels × 5 pixels, one pixel adjacent to the opposite side to the target pixel in the diagonal direction is further required. That is, in order to perform the first filtering process on the range of 5 pixels × 5 pixels indicated by the square marked with “1”, a total of 7 pixels × 7 pixels centered on the pixel of interest P in the figure. Is required.

Next, as shown in the upper center portion of FIG. 7, the second filtering process is performed on a range of 3 pixels × 3 pixels centering on the target pixel P. Also in this case, in order to filter the pixels at the top, bottom, left, and right ends of the range of 3 pixels × 3 pixels, the pixels adjacent to the opposite side toward the target pixel by one pixel at the ends are also required. In addition, for the corner pixel in the range of 3 pixels × 3 pixels, one pixel adjacent to the opposite side to the target pixel is also required in the diagonal direction. That is, in the second filtering process, a total of 9 pixels in the range of 3 pixels × 3 pixels indicated by the squares with “2” in the figure are required.

Then, as shown in the upper right part of FIG. 7, 8 pixels that have been processed up to the second time obtained in this way are used, and the target pixel P is assigned with “3”. Three filtering processes are performed.

As a result, as shown in the lower part of FIG. 7, in order to perform the filtering process three times on the target pixel P, pixels in the range of 7 pixels × 7 pixels indicated by the hatched portion around the target pixel P are used. By doing this, it is possible to perform the filter process three times on the target pixel P. That is, when the target pixel is subjected to the filter process three times, the pixel in the region of 3 pixels × 3 pixels centered on the target pixel in the third process becomes the reference pixel. In addition, for each pixel of 3 pixels × 3 pixels that becomes the third reference pixel in the second process, the reference pixels necessary for the second filter process are required for nine pixels centering on each of the nine pixels. Therefore, a range of 5 pixels × 5 pixels is a reference pixel. Furthermore, each pixel of 5 pixels × 5 pixels is set as a reference pixel for each of the second reference pixels in the first process, and as a result, a range of 7 pixels × 7 pixels is required as a reference pixel.

Here, a pixel to be processed, that is, a reference pixel other than the target pixel, which is required when processing the target pixel, or the number of reference pixels is referred to as overhead, and an area in which the reference pixel exists Is referred to as an overhead region. Therefore, in the case of FIG. 7, as shown in the lower left part of FIG. 7, an attention width of 1 pixel is generated as an overhead area in the overhead width Dp = 4 pixels from the attention pixel up and down and left and right. Become. That is, in the case of FIG. 7, the overhead pixels are generated for 48 pixels excluding the target pixel. However, even a reference pixel is not an overhead pixel for a pixel that can be a pixel of interest separately. A pixel that is not an object to be processed but is required only as a reference pixel is referred to as overhead.

In the following, the overhead width Dp is adopted as a method for expressing the amount of overhead generated for the target pixel. The overhead width Dp is the number of pixels up to either the top, bottom, left, or right end of the pixel as an overhead viewed from the target pixel P, as shown in the lower left part of FIG. Therefore, if the overhead width Dp = 4, the number of pixels that become overhead is 48 pixels.

In the area where the pixel to be processed exists, there are a plurality of pixels. As described above, an image for one frame is divided into two in the horizontal direction, and GPU cards 35-1 and 35 are respectively provided. When the time-division processing is sequentially performed for each range divided into four in the vertical direction by the respective processors 92-1 and 92-2 of -2, for example, overhead as shown in FIG. 8 occurs.

That is, in FIG. 8, an overhead region OHZ1C2 occurs in the region Z1C2 specified by the range C2 in the second stage from the top in the region Z1 on the left side of the image P1.

Therefore, as shown in FIG. 8, when each of the regions Z1 and Z2 is divided into a total of 8 regions of the ranges C1 to C4 divided into 4 in the vertical direction, the overhead is estimated to be 8 times that of the overhead region OHZ1C2. Will do.

In addition, for each overhead, in the case of the above-described filter of 3 pixels × 3 pixels, an example in which an overhead (48 pixels) with an overhead width Dp = 4 pixels is generated for one pixel has been described. The above overhead occurs.

For example, as shown in FIG. 9, in each process related to defect correction processing, RAWNR processing, demosaicing processing, image quality enhancement processing, and enlargement processing, overhead width Dp = 2, 6, 8, 40, 8 pixels. Assuming that overhead occurs, an overhead of a total overhead width Dp = 64 pixels occurs. That is, overhead pixels having the number of pixels excluding the target pixel are generated in the range of 129 pixels × 129 pixels. Further, for example, when dividing into two in the horizontal direction and dividing into four in the vertical direction, the overhead may increase by about 30% as compared to the case where no division processing is performed.

<Overhead reduction method>
As described above, when the overhead increases, the amount of calculation processing becomes enormous, and the processing time increases, so that a processor with higher calculation performance is required to achieve real time. Therefore, in the image processing apparatus 11 of FIG. 1, the overhead is reduced by the following calculation. That is, when the areas Z1 and Z2 of the image P1 are assigned to the GPU cards 35-1 and 35-2, the previous filter processing results are buffered in the memories 93-1 and 93-2. Thus, it can be used in subsequent filter processing.

That is, when n-stage filter processing is necessary as a whole, the processing region in the range C1 by the first filter processing (filter # 1) as shown by the upward-sloping hatched portion in the upper left part of FIG. The number of lines is wider than 1/4 of the total number of lines in the vertical direction for one frame including the number of lines of all reference pixels required in the above process. After that, the ranges C2 and C3 are set so that the range is 1/4 of the total number of lines, and the remaining range is set for the last range C4. In FIG. 10, when the first filter processing (filter # 1), the second filter processing (filter # 2), and the nth filter processing (filter #n) are performed from the left, the ranges C1 to C4 are sequentially applied from the top. Shows the processing range to be performed on the entire image P1 when is sequentially processed.

As a result, the processing result of the range C1 is buffered in the memory 93, and as a result of the processing of the range C2, the area where the necessary reference pixel exists is preliminarily set as indicated by the hatched portion at the lower right. Since the processing is completed in C1, it is only necessary to refer to this, so no overhead occurs. Further, since the range C1 is a range having a line number wider than 1/4 of the total number of lines, the position of the range C2 is 1 of the total number of lines at a position closer to the range C3 than the original range C2. / 4 range. As a result, the region where the reference pixel exists in the range C3 is buffered as the processing result of the range C2, so that it is not necessary to perform filtering again, thereby suppressing the occurrence of overhead.

Similarly, the position of the range C3 is set to ¼ of the total number of lines closer to the range C4 than the original position of the range C3. Since the region where the reference pixel exists in C4 is buffered as the processing result of the range C3, it is not necessary to perform the filtering process again, thereby suppressing the occurrence of overhead.

Further, as indicated by the upward-sloping hatched portion shown in the upper center portion of FIG. 10, the processing area in the range C1 by the second filter processing (filter # 2) is 1 of the total number of lines including the reference pixels thereafter. This is a range wider than / 4, which is narrower than the number of lines of the first filter processing (filter # 1) indicated by the upward-sloping diagonal line at the upper left of FIG. After that, the ranges C2 and C3 are set so that the range is 1/4, and the remaining range is set for the last range C4.

That is, as for the number of lines in the range C1 of the second filter process (filter # 2), the area where the reference pixels exist becomes narrower by the smaller number of subsequent filters than the first filter process (filter # 1). As shown by the upward-sloping hatched portion at the upper center of FIG. 10, the range is wider than 1/4 of the total number of lines, but is narrower than the range C1 of the first filter processing (filter # 1). Become.

As a result, the ranges C2 and C3 are also set so as to be shifted to positions close to ¼ of the original total number of lines. The range C4 is set to be more than the range C1 in the first filter processing (filter # 1). The line width is increased by the reduced number of lines in the range C1.

Thereafter, as the number of remaining filters decreases, the number of lines in the range C1 approaches ¼ of the total number of lines, and the positions of the ranges C2 and C3 become ¼ of the original total number of lines. Approaching. In the last n-th filter processing (filter #n), it is not necessary to consider the reference pixel of the subsequent filter, and as shown in the right part of FIG. The position is one-fourth the number of lines.

As described above, for the line in the range where the reference pixels necessary for the subsequent filtering process exist, the filtering process is first performed in the preceding filtering process, the processing result is buffered, and used in the subsequent filtering process. Thus, it is possible to suppress the occurrence of overhead.

<Low latency display processing>
Next, low latency display processing by the image processing apparatus 11 in FIG. 1 will be described with reference to the flowchart in FIG.

In step S11, the camera IF 71 of the IF card 34 accepts input of image data captured by a camera (not shown) and supplies it to the CPU 51 via the PCIe bridge 73 and the bus 33. The CPU 51 stores the image data input in response to this supply in the main memory 32.

In step S12, the DMA controller 51 divides the image in the horizontal direction according to the number of the GPU cards 35 based on the image data stored in the main memory 32, and further divides each divided area in the vertical direction. The amount of processing is calculated based on the number of ranges when dividing into the number of divisions for division processing, the number of filters related to the processing, and information on the area where the reference pixel exists.

That is, the processing amount is roughly divided into two types, that is, the processing amount related to the vertical processing and the processing amount related to the horizontal processing, and the DMA controller 51 calculates and adds them up.

<Vertical processing amount>
That is, with respect to the vertical direction, after being finally stored in the main memory 32, the output buffer size output from the DMA is used as a reference, and the first filter process # 1 (filter # 1) to the nth filter process (filter It is obtained in the order reverse to the processing order by the number of reference pixels and the processing unit block in each filter processing up to the processing of #n).

That is, as shown in FIG. 12, in the normal filter processing sequentially performed in the vertical direction, the processing result obtained by the first filter processing (filter # 1) is processed by the second filter (filter # 2). Further, processing such as processing by the third filter processing (filter # 3) is repeated, and finally the n-th filter processing is performed, and the DMA transfer is performed and output (output DMA at the upper right part in the figure). .

Therefore, the calculation of the amount of processing in the vertical direction is obtained from the number of lines of the output DMA and sequentially from the number of reference pixels and the processing unit block in each filter processing in the reverse direction. That is, for example, if the number of lines to be the output buffer size is PY (DMA), the number of lines obtained by the nth filter process (filter #n) is determined in advance by the number of pixels constituting the image. For example, the number of lines PY (n) = PY (DMA).

In this case, the number of lines PY (n−1) of the (n−1) th filter processing (filter # (n−1)) is obtained by the following equation (1).

PY (n−1) = PY (n) + BY (n−1) × z
... (1)

Here, PY (n−1) is the number of lines of the (n−1) th filter process (filter # (n−1)), and PY (n) is the line of the nth filter process (filter #n). The number BY (n−1) indicates the number of lines indicating the size of the processing unit block in the (n−1) th filter processing (filter # (n−1)).

Z is a value such that BY (n−1) × A is larger than the number of reference pixels and A is minimum.

That is, as shown in the lower right part of FIG. 12, the (n−1) th filter processing (filter # (n) is performed with respect to the number of lines (number of processing lines) output by the n th filter processing (filter #n). Let us consider a case where the number of lines constituting the reference pixel in -1)) is a range painted in a grid pattern.

Here, the number of processing lines in the n-th filter processing (filter #n) is equivalent to four blocks of processing unit blocks each having a predetermined number of lines indicated by a downward slanting portion at the lower right in FIG. In addition, the reference pixels in the (n−1) th filter processing (filter # (n−1)), as shown as a grid-like range in the lower right part of FIG. There is no range for several lines.

By the way, each filter processing can be performed only for each processing unit block having a predetermined number of lines. Therefore, in the case of the lower right part of FIG. 12, the part having the number of lines less than one block is regarded as one block. Thereby, in the lower right part of FIG. 12, z shown by Formula (1) will be calculated | required as 3. FIG.

Therefore, in the case of the lower right part of FIG. 12, the number of processing lines for the (n-1) th filter processing (filter # (n-1)) is substantially obtained as the number of lines for 7 blocks. .

Thereafter, the number of processing unit blocks up to the first filter processing (filter # 1) is calculated, the processing amount corresponding to the processing unit block number is sequentially calculated, and the total is calculated as the vertical processing amount. .

In this case as well, the number of lines required in each filter processing is the number of lines including reference pixels required in the subsequent stage of each filter so as to reduce overhead as described with reference to FIG. Is set.

<Horizontal processing amount>
The amount of processing in the horizontal direction is also stored in the main memory 32, and the first filter processing (filter # 1) to the n-th filter processing (filter #n) with reference to the output buffer size output by DMA. The number of reference pixels and the processing unit block in each of the filter processes up to are sequentially obtained in the reverse order of the processing order.

That is, as shown in FIG. 13, in the normal horizontal filtering process, the processing result obtained by the first filter process (filter # 1) is processed by the second filter process (filter # 2). Further, the process such as the third filter process (filter # 3) is repeated, and finally the n-th filter process (filter #n) is performed, and the DMA transfer is performed (output DMA in the figure). ).

Therefore, the calculation of the amount of processing in the horizontal direction is obtained from the number of reference pixels and the processing unit block in each filter processing sequentially in the reverse direction from the width defined by the multiple of the processing unit block in the horizontal direction of the output DMA. Go. However, in the horizontal processing, the processing for reducing overhead in the vertical processing is not performed, and the width in the horizontal direction in each filter processing is equal to the number of processing unit blocks corresponding to the number of reference pixels in each filter processing. Is a processing amount corresponding to the horizontal width simply added.

That is, for example, the horizontal width Xk required for calculating the processing amount in the k-th filter processing #k is expressed by the following equation (2).

Xk = w + zk × xk
... (2)

Here, Xk is a width required for calculating the processing amount in the kth filter processing #k, and w is a horizontal direction set by a multiple of the processing unit block in the nth filter processing #n. This is the width, and zx is the width of the processing unit block.

Zk is the sum of the reference pixel numbers in the previous filter process (r1 + r2 +... + R (k−1) + rk), where ri is the reference pixel number in the i-th filter process (filter #i). And zk × xk is a minimum value.

That is, the horizontal direction output by the nth filter process (filter #n (n = 1 to 6: n = 6 is the final stage)) as indicated by the lowermost grid pattern in the lower right part of FIG. The reference pixel in the fifth filter process (filter # 5) with respect to the width of the sixth filter process (filter # 6) corresponding to the nth filter process (filter #n) as the output buffer size Assume that the number is two.

Then, as shown by the grid pattern in the second row from the lower right of FIG. 13, in the fourth filter process (filter # 4), if the number of reference pixels is 1, in this case, In the fifth filter process (filter # 5), 2 which is the number of reference pixels is added to be 3 as indicated by the diagonally shaded area in the lower right.

Similarly, if the number of reference pixels is 3 in the third filter processing (filter # 3) as shown by the grid pattern in the third row from the lower right in FIG. In this case, 3 as the reference pixel number up to the fourth filter process (filter # 4) is added to 6 as indicated by the diagonally shaded area.

Further, as shown in the fourth row from the lower right in FIG. 13, if the number of reference pixels is 1 in the second filter processing (filter # 2), in this case, the diagonally downward slanting portion As indicated by the squares, 6 which is the number of reference pixels up to the third filter process (filter # 3) is added to be 7.

Then, as shown in the uppermost row at the lower right in FIG. 13, if the number of reference pixels is 1 in the first filter processing (filter # 1), in this case, As shown, 7 which is the number of reference pixels up to the second filter process (filter # 2) is added to obtain 8.

That is, in this case, for example, when the processing unit block is composed of one pixel as shown in the third row from the lower right in FIG. 13, the above-described equation in the third filter processing (filter # 3) Zk (= z3) of (2) is 2.

By the above-described method, the processing amount corresponding to the combined result of the widths that are multiples of the processing unit blocks to be processed in the horizontal filters is sequentially obtained.

The DMA controller 51 calculates the vertical processing amount and the horizontal processing amount described above according to the number of horizontal divisions and the number of vertical divisions of the image, and adds both of them to be necessary for processing. Calculate the amount of processing to be done.

In step 13, the DMA controller 51 calculates processing times for various filter processes according to the processing capacity of the processors 92 mounted on the GPU card 35 and the processing amount obtained by the above calculation. Further, various timings such as image data reading or transfer timing are calculated from the obtained processing time. With this process, a timing chart indicating which image data is transferred to which GPU card 35 at which timing in the subsequent processes is constructed.

In step S14, the DMA controller 51 starts processing from a predetermined timing based on this timing chart, determines whether or not the current processing timing is now, and until the next processing timing is reached. Repeat the same process.

In step S14, for example, if it is determined that it is time to start the next process, the process proceeds to step S15.

In step S15, the DMA controller 51 reads out the image data set as the next processing from the main memory 32 based on the timing chart, and transfers it to the GPU card 35 set as the transfer destination. The processor 92 of the card 35 is caused to execute processing. Alternatively, when the processing by the processor 92 of the GPU card 35 is executed and the processing result is transmitted, the DMA controller 51 receives this and stores it in the main memory 32.

In step S16, the DMA controller 51 refers to the timing chart to determine whether or not the next process exists. For example, if there is a next process, the process returns to step S14, and the subsequent processes are performed. Repeated.

That is, the processes in steps S14 to S16 are repeated until it is determined in step S16 that there is no next process. Then, when the processes of steps S14 to S16 are repeated and all the processes set in the timing chart are completed, it is considered that there is no next process in step S16, and the process proceeds to step S17.

In step S <b> 17, the DMA controller 51 displays the image data stored in the main memory 32 and subjected to processing such as high image quality from the display IF 72 via the bus 33 and the PCIe bridge 73 of the IF card 34. To output.

In step S18, the DMA controller 51 determines whether or not the next image has been supplied. If the next image exists, the process returns to step S11, and the subsequent processing is repeated.

If it is determined in step S18 that the next image is not supplied, the process ends.

That is, as described above, the images are divided in the horizontal direction by the processors 92 of the plurality of GPU cards 35 and are shared by the processors 92 for processing. In addition, each processor 92 is divided into a predetermined number of ranges in the vertical direction, and the divided ranges are subjected to time division processing. In this time division processing, the range in which the reference pixel exists in the subsequent filter processing is executed in the previous filter processing and is buffered in the memory 93.

Thus, it becomes possible to execute processing in parallel by the processors 92 in the plurality of GPU cards 35. For example, parallel processing as shown in the lower part of FIG. 3 is possible. In addition, the processing efficiency of each processor 92 is improved so as to reduce the overhead of recalculating reference pixels when performing time division processing in the vertical direction.

As a result, the speed until image data is displayed with high image quality is improved, and low latency can be realized.

<Uniform processing time>
With the above processing, various processing times may change due to a trade-off with overhead reduction in vertical filter processing.

That is, the left part of FIG. 14 shows the reference pixels in the subsequent filtering process in the preceding filtering process so as to reduce overhead in various processes such as defect correction, RAWNR, demosaicing, high image quality, enlargement, and output DMA. An example of the number of lines set in the ranges C1 to C4 when buffering is performed is shown.

In the left part of FIG. 14, the number of lines in the ranges C1 to C4 in the defect correction process is 604, 540, 540, and 476 lines, and the number of lines in the ranges C1 to C4 in the RAWNR process is 596,540, 540 and 484 lines, and the number of lines in the ranges C1 to C4 in the demosaic process is 588, 540, 540, and 492 lines. In addition, the number of lines in the ranges C1 to C4 in the image quality improvement processing is 548, 540, 540, and 532 lines, and the number of lines in the expansion processing ranges C1 to C4 is 540, 540, 540, and 540 lines. It is shown that the number of lines in the output DMA processing range C1 to C4 is 540, 540, 540, and 540 lines.

In this case, the processing time in the ranges C1 to C4 is as shown in the right part of FIG. 14, and the maximum difference Δ in the total processing time in the ranges C1 to C4 is, for example, the processing time in the ranges C1 and C4. This is a difference, and is about 5% of the processing time in the range C1. This is caused by the change in the number of processing lines for the purpose of reducing overhead in the filtering process in the vertical direction, and is caused by changing various processing times. In the right part of FIG. 14, the total processing time and the breakdown of each processing time in the ranges C1 to C4 from the left are shown.

As a countermeasure against such non-uniform processing time, for example, in the ranges C1 to C4, the number of lines to be finally output may be adjusted to smooth the processing time.

That is, for example, as shown in the lower left part of FIG. 15, the processing time non-uniformity is eliminated such that the number of lines in the output DMA processing is 520 lines for the range C1 and 560 lines for the range C4. To be non-uniform.

In this way, as shown in the lower right part of FIG. 15, the processing time difference Δ in the ranges C1 and C4 is almost eliminated, and the processing time can be smoothed and made uniform as a whole. It becomes. In addition, the upper left part and upper right part of FIG. 15 are the same as the left part and right part of FIG. 14, respectively.

Further, for example, among the ranges C1 to C4, for a range where the processing time is early, processing that does not need to adjust the processing speed in real time may be shared. For example, as shown in FIG. It may be assigned to detection processing or the like in the time zone indicated by the uppermost black range in C2 to C4 so that the processing time becomes uniform as a whole.

With the above processing, the image is divided in the horizontal direction, assigned to a plurality of processors, the horizontally divided areas are time-divided in the vertical direction, and the ranges divided in the vertical direction are The range including the reference pixels required for the subsequent processing is set in the range. Then, in the processing of the top range, the buffer processing is performed after performing the filtering processing including the processing of the reference pixel in advance, and in the subsequent filtering processing, the processing is executed with reference to the buffered one. did. As a result, the display processing of the captured image can be reduced in latency, and the captured image can be displayed at high speed at a timing closer to the actual time when the captured image is captured.

For this reason, the image processing apparatus 11 in FIG. 1 uses, for example, an endoscope that is used for endoscopic surgery as an imaging device that images a patient's surgical part, a microscope that is used for neurosurgery, and the like. The present invention can be applied to an image processing apparatus that processes an image obtained by imaging an operation part, and further to a surgical system including an endoscope or a microscope as an imaging apparatus. In addition, when using the processor 92 in the GPU card 35, it is possible to reduce consideration of a time lag or the like in displaying an image, so that it is possible to improve programmability. Furthermore, when displaying an image received via a broadcast wave or the like, it is possible to reduce the latency, so that it is possible to display the image while suppressing the time lag.

Further, when processing an image, the DMA controller 51 calculates the processing amount in advance according to the number of reference pixels corresponding to the filter used for processing and the processing unit block, and then reads out the image data and writes the image data. Since the processing is executed after optimizing, it is possible to reduce the latency in an optimum state regardless of the processing content.

Incidentally, the above-described series of processing can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

FIG. 17 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

The input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, programs, and various types. A storage unit 1008 including a hard disk drive for storing data, a LAN (Local Area Network) adapter, and the like are connected to a communication unit 1009 that executes communication processing via a network represented by the Internet. Also, magnetic disks (including flexible disks), optical disks (including CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc)), magneto-optical disks (including MD (Mini Disc)), or semiconductors A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

In the computer configured as described above, the CPU 1001 loads the program stored in the storage unit 1008 to the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program, for example. Is performed.

The program executed by the computer (CPU 1001) can be provided by being recorded on the removable medium 1011 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by attaching the removable medium 1011 to the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

In this specification, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Accordingly, a plurality of devices housed in separate housings and connected via a network and a single device housing a plurality of modules in one housing are all systems. .

Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared by a plurality of devices via a network and is jointly processed.

Further, each step described in the above flowchart can be executed by one device or can be shared by a plurality of devices.

Further, when a plurality of processes are included in one step, the plurality of processes included in the one step can be executed by being shared by a plurality of apparatuses in addition to being executed by one apparatus.

In addition, this technique can also take the following structures.
(1) including a plurality of arithmetic processing units that perform processing in a time-sharing manner for each range in which an image obtained by imaging a patient's surgical part is vertically divided;
The image processing device, wherein the arithmetic processing unit performs processing by time-dividing the image divided in the horizontal direction by the number of the arithmetic processing units in the vertical direction.
(2) The plurality of arithmetic processing units are configured by a plurality of GPUs (Graphical Processing Units),
The image processing device according to (1), wherein the arithmetic processing unit performs processing on the image divided in the horizontal direction by the number of the GPUs.
(3) The image processing apparatus according to (1) or (2), wherein the process performed on the image is a process of applying an n-stage filter.
(4) The n stages of filters perform processing in a time-division manner in order from a range obtained by dividing the image in the vertical direction from the uppermost stage in the vertical direction downward. Image processing device.
(5) Based on the amount of processing performed on the image calculated based on the number of horizontal divisions and the number of vertical divisions of the image, and the processing speed of the arithmetic processing unit, the arithmetic processing unit The image processing apparatus according to any one of (1) to (4), further including a timing control unit that controls timing of the calculation.
(6) Of the range in which the image is time-divided in the vertical direction, the processing range in the first period includes reference pixels required for processing in the second period after the first period. The image processing device according to any one of (1) to (5).
(7) The arithmetic processing unit includes a memory for buffering a processing result,
In the process of the second period, an arithmetic process is executed using a process result corresponding to the reference pixel from the process result of the first period buffered in the memory. Processing equipment.
(8) The arithmetic processing unit includes a memory for buffering a processing result,
Of the range obtained by dividing the image in the vertical direction, the uppermost processing range in the vertical direction by the filter at each stage is required for the processing of the filter in the processing range of the second stage or less in the vertical direction. A range that includes the number of reference pixel lines,
When the arithmetic processing unit executes arithmetic processing for processing by the filter, the processing using the reference pixel is performed on the reference pixel from the processing result of the filtering processing up to the previous stage buffered in the memory. The image processing apparatus according to (3), wherein arithmetic processing is executed using a corresponding processing result.
(9) The image processing device according to any one of (1) to (8), wherein the arithmetic processing unit performs at least an enlargement process on an image obtained by imaging the surgical site of the patient.
(10) The image processing device according to any one of (1) to (9), wherein the image obtained by imaging the surgical site of the patient is an image taken by an endoscope.
(11) The image processing device according to any one of (1) to (9), wherein the image obtained by imaging the surgical site of the patient is an image taken by a microscope.
(12) In an image processing method of an image processing apparatus including a plurality of arithmetic processing units that perform time-division processing on an image obtained by imaging a patient's surgical part for each range divided in the vertical direction.
An image processing method in which the arithmetic processing unit performs processing by time-dividing the image divided in the horizontal direction in the vertical direction by the number of the arithmetic processing units.
(13) The image processing method according to (12), wherein the image is an image captured by an endoscope.
(14) The image processing method according to (12), wherein the image is an image captured by a microscope.
(15) an imaging device for imaging the surgical site of a patient;
A plurality of arithmetic processing units that perform processing in a time-sharing manner for each range divided in the vertical direction of the image captured by the imaging device;
The operation system includes: an image processing device that performs processing by time-dividing the image divided in the horizontal direction in the vertical direction by the number of the arithmetic processing units.

11 Information processing section, 31 CPU, 32 main memory, 33 bus, 34 IF card, 35, 35-1, 35-2 GPU card, 51 DMA controller, 71 camera IF, 72 display IF, 73, 91, 91-1 , 91-2 PCIe bridge, 92, 92-1, 92-2 processor, 93, 93-1, 93-2 memory

Claims

A plurality of arithmetic processing units that perform processing in a time-sharing manner for each range divided vertically in an image obtained by imaging a patient's surgical part,
The image processing device, wherein the arithmetic processing unit performs processing by time-dividing the image divided in the horizontal direction by the number of the arithmetic processing units in the vertical direction.
The plurality of arithmetic processing units are configured by a plurality of GPUs (Graphical Processing Units),
The image processing apparatus according to claim 1, wherein the arithmetic processing unit performs processing on the image divided in the horizontal direction by the number of GPUs.
The image processing apparatus according to claim 1, wherein the process applied to the image is a process of applying an n-stage filter.
The image processing apparatus according to claim 3, wherein each of the n stages of filters sequentially processes a range obtained by dividing the image in the vertical direction in a time-division manner from the uppermost range in the vertical direction downward. .
Based on the amount of processing performed on the image calculated based on the number of divisions in the horizontal direction and the number of divisions in the vertical direction of the image, and the processing speed of the arithmetic processing unit, The image processing apparatus according to claim 1, further comprising a timing control unit that controls timing.
The reference pixel that is required for processing in a second period after the first period is included in the processing range in the first period among the range in which the image is time-divided in the vertical direction. The image processing apparatus according to 1.
The arithmetic processing unit includes a memory for buffering a processing result,
7. The image according to claim 6, wherein in the processing of the second period, an arithmetic process is executed using a processing result corresponding to the reference pixel from a processing result of the first period buffered in the memory. Processing equipment.
The arithmetic processing unit includes a memory for buffering a processing result,
Of the range obtained by dividing the image in the vertical direction, the uppermost processing range in the vertical direction by the filter at each stage is required for the processing of the filter in the processing range of the second stage or less in the vertical direction. A range that includes the number of reference pixel lines,
When the arithmetic processing unit performs arithmetic processing for the processing by the filter, the processing using the reference pixel is performed on the reference pixel from the processing result of the filtering processing up to the previous stage buffered in the memory. The image processing apparatus according to claim 3, wherein arithmetic processing is executed using a corresponding processing result.
The image processing apparatus according to claim 1, wherein the arithmetic processing unit performs at least enlargement processing on an image obtained by imaging the surgical site of the patient.
The image processing apparatus according to claim 1, wherein the image obtained by imaging the surgical site of the patient is an image obtained by an endoscope.
The image processing apparatus according to claim 1, wherein the image obtained by imaging the surgical site of the patient is an image taken by a microscope.
In an image processing method of an image processing apparatus including a plurality of arithmetic processing units that perform time-division processing for each range divided in the vertical direction from an image obtained by imaging a patient's surgery unit,
An image processing method in which the arithmetic processing unit performs processing by time-dividing the image divided in the horizontal direction in the vertical direction by the number of the arithmetic processing units.
The image processing method according to claim 12, wherein the image is an image captured by an endoscope.
The image processing method according to claim 12, wherein the image is an image captured by a microscope.
An imaging device for imaging a patient's surgical site;
A plurality of arithmetic processing units that perform processing in a time-sharing manner for each range divided in the vertical direction of the image captured by the imaging device;
The operation system includes: an image processing device that performs processing by time-dividing the image divided in the horizontal direction in the vertical direction by the number of the arithmetic processing units.