JP2023160275A - System, method, and program for three-dimensionally displaying two-dimensional moving images - Google Patents

Abstract

To three-dimensionally display two-dimensional moving images captured by a monocular imaging apparatus.SOLUTION: A system for three-dimensionally displaying two-dimensional moving images of the present invention comprises: receiving means that receives the two-dimensional moving images; depth information acquisition means that acquires depth information from each of the frames of the two-dimensional moving images; image pair creation means that creates an image pair having parallax for each of the frames of the two-dimensional moving images by using the depth information acquired from each of the frames of the two-dimensional moving images; and display means that displays the image pair for each of the frames of the two-dimensional moving images.SELECTED DRAWING: Figure 1

Description

The present invention relates to a system, method, and program for displaying two-dimensional moving images three-dimensionally.

BACKGROUND ART Compound eye imaging devices capable of capturing three-dimensional images are known (for example, Patent Document 1).

JP2009-48181A

An object of the present invention is to make it possible to display three-dimensionally even a two-dimensional moving image that does not have depth information.

The present invention provides a system for displaying a two-dimensional moving image three-dimensionally, which enables, for example, a two-dimensional moving image that does not have depth information to be displayed three-dimensionally.

In one embodiment, the present invention provides the following items.
(Item 1)
A system for displaying 2D videos in 3D,
a receiving means for receiving a two-dimensional video;
Depth information acquisition means for acquiring depth information from each frame of the two-dimensional video;
image pair creation means for creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
and display means for displaying the pair of images of each frame of the two-dimensional video.
(Item 2)
The image pair creation means, for each frame of the two-dimensional video,
determining a displacement amount of at least one pixel of the frame based on the depth information;
creating a parallax image in which parallax is added to the frame by shifting the at least one pixel of the frame by the determined displacement amount; 2. The system of item 1, wherein the system generates an image pair.
(Item 3)
The system according to item 2, wherein the image pair creation means determines the displacement amount such that the smaller the depth, the larger the displacement amount.
(Item 4)
The image pair creation means determines pixels to be processed and pixels not to be processed for each frame of the two-dimensional video based on the depth information,
4. The system of item 2 or item 3, wherein the at least one pixel is at least one of the pixels to be processed.
(Item 5)
The image pair creation means, for each frame of the two-dimensional video,
determining an enlargement or reduction ratio of at least one region of the frame based on the depth information;
In any one of items 2 to 4, the at least one region of the frame and the corresponding at least one region of the parallax image are enlarged/reduced at the determined enlargement rate or reduction rate. The system described.
(Item 6)
The image pair creation means determines the magnification rate and/or the reduction rate such that the magnification rate is greater in an area with a smaller depth, and/or the reduction rate is greater in an area with a greater depth. The system described in item 5.
(Item 7)
7. The system according to any one of items 1 to 6, wherein the display means includes a lenticular display.
(Item 8)
A method for displaying a two-dimensional video three-dimensionally, the method comprising:
receiving a two-dimensional video;
obtaining depth information from each frame of the two-dimensional video;
creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
displaying the image pair of each frame of the two-dimensional video.
(Item 9)
A program for displaying a two-dimensional moving image three-dimensionally, the program being executed in a computer including a processor section and a display section, the program comprising:
receiving a two-dimensional video;
obtaining depth information from each frame of the two-dimensional video;
creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
Displaying the pair of images of each frame of the two-dimensional moving image on the display unit.

According to the present invention, even a two-dimensional image without depth information can be displayed three-dimensionally.

A diagram schematically showing an example of a flow for displaying a two-dimensional video three-dimensionally using the system of the present invention. A diagram showing an example of the configuration of a system 100 for displaying a two-dimensional video three-dimensionally. A diagram schematically showing an example of processing by the image pair creation means 130 A diagram showing an example of the result of performing enlargement/reduction processing on the image pair shown in FIG. 3A at an enlargement/reduction ratio according to the depth. Flowchart of an example of processing (processing 400) in the system 100 for displaying a two-dimensional video three-dimensionally Flowchart showing an example of processing performed by the processor unit in step S403 Diagram showing image pairs created in the example Diagram showing image pairs created in the example Diagram showing image pairs created in the example Diagram showing image pairs created in the example

Embodiments of the present invention will be described below with reference to the drawings.

(1. System for displaying monocular endoscopic videos in three dimensions)
The inventors of the present invention recognized problems in microsurgery or endoscopic surgery. One of the problems is that a two-dimensional moving image taken with a microscope or an endoscope cannot be seen three-dimensionally. An experienced surgeon may be able to imagine a three-dimensional structure from a two-dimensional video, but an inexperienced surgeon may not be able to do so by viewing videos obtained with a microscope or endoscope. It is extremely dangerous for inexperienced surgeons to perform surgery.

One way to solve this problem is to use a compound endoscope. A compound eye endoscope can acquire two-dimensional moving images with depth information. By using a compound eye endoscope, the resulting moving image can be displayed three-dimensionally, so even an inexperienced surgeon can recognize the three-dimensional structure and perform surgery. However, the inventor of the present invention considered that there are drawbacks to using a compound endoscope. Specifically, compound-eye endoscopes have the disadvantage that they are less sensitive to high magnification than monocular endoscopes, and the images become blurred due to lack of focus at high magnifications. In addition, in order to display two-dimensional videos obtained with a compound eye endoscope three-dimensionally, it is necessary to wear 3D glasses, etc., and watching videos for a long time through 3D glasses can cause so-called 3D motion sickness. It also had the disadvantage of being prone to Furthermore, the necessity of wearing 3D glasses or the like makes it impossible to share the video with multiple people, which poses difficulties, for example, in terms of medical education. In addition, compound endoscopes are much more expensive than monocular endoscopes, and they also have the drawback of being extremely expensive to introduce.

As a result of intensive research, the inventor of the present invention has developed a system for three-dimensionally displaying two-dimensional moving images obtained from a monocular endoscope without relying on a compound-eye endoscope. Note that in this specification, displaying a video three-dimensionally means displaying a video so that it is recognized as three-dimensional by the viewer, in other words, displaying a video so that stereoscopic viewing is possible. This means that the displayed video does not necessarily have to be a 3D video.

FIG. 1 schematically shows an example of a flow for displaying a two-dimensional video three-dimensionally using the system of the present invention.

First, using the monocular endoscope 10, a body part S within the field of view of the monocular endoscope 10 is photographed. Body part S includes structure A and structure B. From the video taken by the monocular endoscope 10, it is not possible to recognize what kind of three-dimensional structures structure A and structure B have. This is because the moving image acquired by the monocular endoscope 10 is two-dimensional and lacks depth information.

In step S1, a two-dimensional video captured by the monocular endoscope 10 is input to the system 100 of the present invention. Here, a moving image is a plurality of consecutive still images, and in this specification, each of the still images making up the moving image is referred to as a "frame." When a two-dimensional video is input to the system 100 of the present invention, the system 100 of the present invention creates an image pair for each frame of the two-dimensional video. The image pairs have parallax with each other, one of the image pairs becomes a frame for the right eye, and the other image pair becomes a frame for the left eye.

In step S2, the system 100 of the present invention sequentially displays image pairs created from the video. By viewing the frame for the right eye with the right eye and viewing the frame for the left eye with the left eye, the viewer can three-dimensionally recognize the subject reflected in the pair of images having parallax. For example, a video 20 will be displayed to the viewer. In the video 20, a body part S photographed by the monocular endoscope 10 is displayed three-dimensionally. This allows the viewer to recognize the three-dimensional structures of Structure A and Structure B. In the example shown in FIG. 1, it can be seen from the video 20 that structure A is a depressed structure with respect to its surroundings, and structure B is a structure that is raised with respect to its surroundings.

The image pair can be displayed in any three-dimensional display format. For example, the pair of images may be displayed so that the viewer can view the video 20 by wearing a 3D display device (for example, 3D glasses), or the viewer may The image pairs may be displayed so that the video 20 can be viewed without the need to wear a mounting device. Preferably, the image pair may be displayed so that a viewer can view the video 20 without having to wear specialized equipment. This can be achieved, for example, by displaying the image pair on a special display device, such as a lenticular display.

In this way, even a two-dimensional moving image taken by the monocular endoscope 10 can be displayed three-dimensionally according to the system 100 of the present invention. surgery can be performed by recognizing the structure. In addition, images taken with the monocular endoscope 10 are more resistant to high magnification than those of compound eye endoscopes, and are in focus even at high magnifications, making it possible to recognize even fine three-dimensional structures. can.

Furthermore, by displaying the image pair using a lenticular display or the like so that a dedicated mounting device is not required, the viewer is less likely to suffer from so-called 3D motion sickness. Furthermore, the video 20 can be shared and viewed by multiple people, which is useful for medical education. Furthermore, since a dedicated mounting device requires contact for attachment and removal, the system 100 of the present invention becomes a non-contact tool and will be useful even in the post-corona era. Become.

Another advantage is that there is no need to introduce an expensive compound endoscope, and safe surgery can be performed at low cost.

In addition, the system 100 of the present invention can perform operations in substantially real time (e.g., with a delay of about 20 to 50 milliseconds or less relative to the time the two-dimensional video is input, preferably with a delay of about 30 milliseconds or less). The image pair can be output (typically with a delay of about 33 milliseconds or less). The human visual delay discrimination threshold is said to be 20 to 30 milliseconds, and humans can perceive that it is real time if the delay is about 50 milliseconds or less, preferably about 30 milliseconds or less. . The ability of the system of the present invention to output image pairs in near real time is particularly advantageous in surgical applications. This is because, in surgical applications, the difference between the time when the video is taken and the time when the video is displayed makes it impossible for the surgeon to perform accurate surgery.

In the above example, a two-dimensional video captured by the monocular endoscope 10 was explained, but the system 100 of the present invention can display three-dimensional images using the monocular endoscope 10. The present invention is not limited to two-dimensional videos shot. The system 100 of the present invention can display any other two-dimensional video three-dimensionally, as long as the two-dimensional video does not have depth information. The two-dimensional video may be, for example, an existing animation, movie, TV program, drama, etc. As a result, existing animations, movies, TV programs, dramas, etc. will be displayed three-dimensionally. This may create a new user experience.

The system 100 of the present invention described above can have the configuration described below.

(2. Configuration of system for displaying 2D video in 3D)
FIG. 2 shows an example of the configuration of a system 100 for displaying a two-dimensional moving image three-dimensionally.

The system 100 includes a receiving means 110, a depth information acquiring means 120, an image pair creating means 130, and a display means 140.

The receiving means 110 is configured to receive a two-dimensional moving image.

The two-dimensional video may be any two-dimensional video. The two-dimensional video may preferably be a video captured by a monocular imaging device (for example, a monocular endoscope). The two-dimensional video may be, for example, an existing animation, movie, TV program, drama, etc.

The two-dimensional moving image received by the reception means 110 is passed to the depth information acquisition means 120 and the image pair creation means 130.

The depth information acquisition means 120 is configured to acquire depth information from each frame of the two-dimensional video received by the reception means 110. Depth information is information representing the distance (depth) from a predetermined reference point (eg, camera position) to a subject. For example, the depth information acquisition unit 120 may acquire depth information for each pixel, or may acquire depth information for an area including a plurality of pixels. By the depth information acquisition unit 120 acquiring depth information for a region including a plurality of pixels, the load of processing for acquiring depth information can be reduced and the processing speed can be increased. Depth information may be expressed, for example, as a depth map, or in a format such as a matrix that describes each pixel and its depth.

The depth information acquisition means 120 can acquire the depth information of each frame using, for example, a machine learning model. A machine learning model is a model that has learned the relationship between an image and its depth information. Such a machine learning model can be constructed by learning about each of a plurality of images, using an image as input training data and depth information of the image as output training data. Alternatively, such a machine learning model can be constructed by learning about each of a plurality of images using an image as input training data and a depth map of the image as output training data. When you input an image to such a machine learning model, it will output depth information or a depth map for that image.

The images used for learning may preferably be images taken by a compound eye imaging device. This is because if the image is captured by a compound eye imaging device, depth information can be easily acquired or a depth map can be easily created. A method of acquiring depth information from an image photographed by a compound-eye imaging device or a method of creating a depth map may be a method known in the art.

The depth information of each pixel of each frame of the two-dimensional video created by the depth information acquisition means 120 is passed to the image pair creation means 130.

The image pair creation means 130 is configured to create an image pair having parallax for each frame of a two-dimensional video using depth information. The image pair may be a pair of a frame and an image created by the image pair creation means 130, or a pair of a first image and a second image created by the image pair creation means 130. It's okay. Preferably, the image pair may be a pair of a frame and an image created by the image pair creation means 130. This is because by using a frame as one of the image pairs, the processing of the image pair creation means 130 can be reduced, which can lead to faster processing.

The frame and the created image have a mutual parallax, and the first image and the second image also have a mutual parallax. Here, two images having parallax means that the positions of the same subject in the two images are shifted from each other by an amount of displacement corresponding to the difference in viewpoint (for example, camera position) of the two images. means. For example, two images of the same subject taken from two different viewpoints will have parallax. In the image pair created by the image pair creation means 130, the position of the subject in the frame and the position of the same subject in the created image differ from each other by an amount of displacement corresponding to the distance between the eyes of the person. Or, the position of the subject in the first image created and the position of the same subject in the second image created are only a displacement amount corresponding to the distance between the person's eyes. They are out of alignment with each other. For example, the closer the object is (smaller the depth), the larger the positional shift between the two images becomes, and the farther the object is (larger the depth), the larger the difference between the two images becomes. It is possible to create a pair of images in which the positional deviation between the images is small.

The image pair creation means 130 determines, for each frame of the two-dimensional video, the displacement amount of at least one pixel of the frame based on the depth information, and the determined displacement amount of at least one pixel of the frame. By shifting the frame by the amount, an image with parallax is created in which parallax is added to the frame, and an image pair consisting of the frame and the image with parallax can be created. At this time, the image pair creation means 130 can determine the amount of displacement such that the smaller the depth, the larger the amount of displacement. For example, depth and displacement may have a linear relationship. For example, if a first pixel in a frame has a first depth and a second pixel has a depth twice the first depth, the amount of displacement of the first pixel is equal to the amount of displacement of the second pixel. This can be twice the amount of displacement of the pixel.

By shifting the pixels by a displacement amount corresponding to the depth in this manner, the created image pair is displayed three-dimensionally with a more natural appearance. This is because it is in line with human physiological binocular recognition to shift a subject with a small depth (near) by a larger amount and shift a subject with a larger depth (farther) by a smaller amount.

FIG. 3A schematically shows an example of processing by the image pair creation means 130. FIG. 3A(a) represents a frame of a moving image, and FIG. 3A(b) represents a created image with parallax. For ease of explanation, each image will be described as having 6x7 pixels. It is assumed that a pixel represented in light gray has a depth of 0.5, a pixel represented in dark gray has a depth of 1.0, and a pixel represented in white has a depth of 10. The smaller the depth, the closer to the viewpoint of the image (for example, the camera position).

When the frame of the moving image shown in FIG. 3A(a) is input to the image pair creation means 130, the image pair creation means 130 determines, for example, that the amount of displacement of the pixel represented in light gray is 2 pixels, and The amount of displacement of the pixel represented in gray is determined to be 1 pixel, and the amount of displacement of the pixel represented in white is determined to be 0 pixel. Then, the image pair creation means 130 creates the parallax image shown in FIG. 3A(b) by shifting each pixel by the determined displacement amount. For example, by shifting the light gray pixel at (3,1) in FIG. 3A(a) by two pixels to the right in the paper, the corresponding pixel becomes the pixel at (3,3) in FIG. 3(b). For example, by shifting the dark gray pixel (2, 2) in FIG. 3A(a) by one pixel to the right in the paper, the corresponding pixel becomes the pixel (2, 3) in FIG. 3(b). In this way, an image pair consisting of the moving image frame shown in FIG. 3A(a) and the parallax image shown in FIG. 3A(b) is created.

In one embodiment, the image pair creation means 130 determines pixels to be processed and pixels not to be processed based on the depth information, and processes only the pixels to be processed (specifically Specifically, it may be possible to perform processing to determine the amount of displacement and shift by the amount of displacement), and do nothing to pixels that should not be processed. This has the advantage that the resources required for processing can be reduced, thereby reducing the load on the computer and leading to faster processing. This embodiment is preferred for near real-time image pair display.

For example, the image pair creation means 130 determines that a pixel whose depth is less than a predetermined threshold is a pixel that should be processed, and a pixel whose depth is greater than or equal to a predetermined threshold is a pixel that should not be processed. can be determined. The predetermined threshold value may be a fixed value or a variable value. When the predetermined threshold value is a variable value, the predetermined threshold value may be, for example, the average value, median value, first quartile, or third quartile of the depth of all pixels; For example, the depth may be 90%, 80%, 70%, 60%, 40%, or 30% of the maximum depth.

For example, in the example described above with reference to FIG. 3A, the image pair creation means 130 determines the pixels represented in white and the pixels represented in dark gray as pixels that should not be processed, based on the depth information. However, no processing is performed on pixels expressed in white and pixels expressed in dark gray. On the other hand, the image pair creation means 130 determines the pixel represented in light gray as the pixel to be processed based on the depth information, determines the amount of displacement of the pixel represented in light gray, and determines the amount of displacement of the pixel represented in light gray. Shifts the represented pixel by the amount of displacement. This eliminates the need to process pixels represented in white and pixels represented in dark gray, making it possible to reduce the resources required for processing and speed up processing. can.

In addition to the process of determining the amount of displacement described above, the image pair creation means 130 determines, for each frame of the two-dimensional video, the enlargement rate or reduction rate of at least one region of the frame based on the depth information. At least one region of the frame and at least one corresponding region of the parallax image may be enlarged/reduced at the determined enlargement rate or reduction rate. The at least one region may be, for example, a region containing pixels with the same or similar depth. As used herein, similar depth refers to a depth of ±10%. The image pair creation means 130 can determine the enlargement rate and/or the reduction rate such that the smaller the depth is, the larger the enlargement rate is, and/or the larger the depth is, the larger the reduction rate is. . For example, the depth and the enlargement rate and reduction rate may have a linear relationship. For example, if a first region in a frame has a first depth and a second region has a depth twice the first depth, then the magnification factor of the first region is the same as that of the second region. It can be twice the area.

By enlarging/reducing the image at an enlargement/reduction ratio according to the depth in this manner, the created image pair is displayed in a three-dimensional manner with a more natural appearance. Making objects with a small depth (closer) larger and objects with a larger depth (farther) smaller is in line with human physiological binocular recognition and increases the effect of sensory stereopsis. This is because it is possible. Furthermore, such enlargement/reduction changes the focal point of the image for the viewer, so that the edges of the subject appear to be emphasized. This may cause, for example, objects that are close to each other to appear sharp, have continuous contours, and/or good resolution, while objects that are far away appear blurred, have discontinuous contours, and/or have poor resolution. become. This can also enhance the effect of sensory stereopsis.

FIG. 3B shows an example of the result of performing enlargement/reduction processing on the image pair shown in FIG. 3A at an enlargement/reduction ratio depending on the depth. FIG. 3B(a) represents a frame of a moving image, and FIG. 3B(b) represents a created image with parallax. As described above with reference to FIG. 3A, pixels represented by light gray have a depth of 0.5, pixels represented by dark gray have a depth of 1.0, and are represented by white. Assume that a pixel has a depth of 10.

For the image pair shown in FIG. 3A, the image pair creation means 130 determines, for example, that the enlargement ratio of the area represented in light gray is 160%, and the enlargement ratio of the area represented in dark gray and white is determined to be 160%. The magnification rate of the area is determined to be 100% (that is, no expansion/reduction). Then, the image pair creation means 130 creates the image pair shown in FIG. 3B by enlarging each region at the determined magnification rate. That is, it is composed of an image in which at least one region of a frame of a moving image shown in FIG. 3B(a) is enlarged, and an image with parallax in which at least one corresponding region is enlarged, shown in FIG. 3B(b). A pair of images is created.

In this example as well, the image pair creation means 130 determines pixels that should be processed and pixels that should not be processed based on the depth information, and processes only the pixels that should be processed (specifically In particular, it may be possible to perform processing for enlarging/reducing the image by determining an enlargement/reduction ratio), and do nothing to pixels that should not be processed. This has the advantage that a pair of images with a high sensory stereoscopic effect can be created at high speed while reducing the load on the computer.

The created image pair is passed to display means 140.

The display means 140 is configured to display image pairs created for each frame of the two-dimensional video. The display means 140 can display the image pair in any three-dimensional display format. The display means 140 is configured to display image pairs so that a viewer can view a three-dimensional moving image based on the image pairs by wearing a 3D display device (for example, 3D glasses). Alternatively, the image pairs may be displayed so that the viewer can view the three-dimensional animation of the image pairs without having to wear a special mounting device. Preferably, the display means 140 may display the image pairs so that a viewer can view a three-dimensional animation of the image pairs without having to wear specialized equipment. As a result, viewers are less likely to suffer from so-called 3D motion sickness, and furthermore, 3D videos can be shared and viewed by multiple people. Furthermore, since a dedicated mounting device requires contact for attachment and removal, the system 100 of the present invention becomes a non-contact tool and will be useful even in the post-corona era. Become.

The display means 140 may specifically be a lenticular display. A lenticular display is a display in which lenticular lenses are arranged on a display surface, and makes it possible to view three-dimensional images without wearing a mounting device for 3D display.

In this manner, the system 100 of the present invention creates image pairs from each frame of the input video and displays the image pairs, thereby making the video appear three-dimensional. Since the system 100 of the present invention does not require processing such as combining image pairs to generate a three-dimensional video, it is possible to reduce the load and speed up the processing. This can lead to near real-time display. Furthermore, pixels to be shifted by a large amount of displacement may be selectively processed, regions to be enlarged/reduced by large enlargement/reduction ratios may be selectively processed, and/or pixels/regions to be processed may be selectively processed. Selecting and processing the information can also lead to near real-time display.

The system 100 described above can be implemented as a computer device, for example.

The computer device includes an input section, a memory section, a processor section, and a display section.

The input unit is configured to allow input from outside the computer device. A two-dimensional moving image can be input to the input unit. The manner in which the two-dimensional video is input to the input unit does not matter. For example, a two-dimensional video may be input directly from an imaging device, may be input from an imaging device via a network (e.g. LAN, Internet), or may be input via a storage medium. It may also be configured to be input. Preferably, the two-dimensional moving image can be directly input from an imaging device. This is because the imaging device can take an image and the computer device can display a moving image almost simultaneously.
For example, the receiving means 110 of the system 100 of the invention can be implemented by an input unit,
The memory unit stores programs required for execution of processing by the computer device, data required for execution of the programs, and the like. For example, part or all of a program for causing a computer device to perform processing for displaying a two-dimensional video three-dimensionally (for example, a program for realizing the processing shown in FIGS. 4A and 4B described later) is stored. has been done. Here, it does not matter how the program is stored in the memory section. For example, the program may be preinstalled in the memory unit. Alternatively, the program may be installed in the memory unit by being downloaded via a network. The program may be stored on a computer readable tangible storage medium. The memory section may be implemented by any storage means.

The processor section controls the operation of the entire computer device. The processor section reads a program stored in the memory section and executes the program. This allows the computer device to function as a device that executes desired steps. The processor section may be implemented by a single processor or by multiple processors.

For example, at least a part of the receiving means 110, the depth information acquisition means 120, the image pair creation means 130, and the display means 140 of the system 100 of the present invention may be implemented by a processor section.

The display section is configured to display images. The display unit can display the created image pair. The display unit may be any display device, preferably a lenticular display.

For example, the display means 140 of the system 100 of the invention may be implemented by a display.

Each component of the computer device may be provided within the computer device or may be provided outside the computer device. For example, if the input section, memory section, processor section, and display section are each composed of separate hardware components, each of the hardware components may be connected via an arbitrary network. At this time, the type of network does not matter. Each hardware component may be connected via a LAN, wirelessly, or wired, for example. Computer devices are not limited to any particular hardware configuration. For example, it is also within the scope of the present invention to configure the processor section with an analog circuit rather than a digital circuit. The configuration of the computer device is not limited to that described above as long as its functions can be realized.

The computer device can be used, for example, as a surgical support device. In this case, a two-dimensional video of the surgical site captured by a surgical imaging device (for example, a monocular endoscope) is input to the input unit. A two-dimensional moving image of the surgical site will be displayed three-dimensionally from the display unit.

The computer device can be used, for example, as a video playback device. In this case, an existing two-dimensional moving image (for example, animation, movie, TV program, drama) is input to the input unit. The existing two-dimensional moving image will be displayed three-dimensionally from the display unit.

(3. Processing in a system for displaying 2D video in 3D)
FIG. 4A shows a flowchart of an example of a process (process 400) in the system 100 for displaying a two-dimensional video three-dimensionally. In this example, an example in which the system 100 is implemented by a computer device will be described. Process 400 will be executed in a processor section of a computer device.

In step S401, the processor unit receives a two-dimensional video input from the input unit. The two-dimensional video may be any two-dimensional video. The two-dimensional video may preferably be a video captured by a monocular imaging device (for example, a monocular endoscope). The two-dimensional video may be, for example, an existing animation.

In step S402, the processor unit obtains depth information from each frame of the two-dimensional video received in step S401.

The processor unit can obtain depth information for each frame using, for example, a machine learning model. A machine learning model is a model that has learned the relationship between an image and its depth information. When each frame of the two-dimensional video received in step S401 is input to such a machine learning model, depth information of each frame is output.

In step S403, the processor unit uses the depth information acquired in step S402 for each frame of the two-dimensional video received in step S401 to determine whether each frame of the two-dimensional video received in step S401 is , create a pair of images with parallax. The image pair may be a pair of a frame and an image created by the image pair creation means 130, or a pair of a first image and a second image created by the image pair creation means 130. It's okay. For example, the closer the object is (smaller the depth), the larger the positional shift between the two images becomes, and the farther the object is (larger the depth), the larger the difference between the two images becomes. It is possible to create a pair of images in which the positional deviation between the images is small.

In step S403, the processor unit determines, for each frame of the two-dimensional video, the displacement amount of at least one pixel of the frame based on the depth information, and determines the amount of displacement of at least one pixel of the frame. By shifting the frame by the amount of displacement and creating a parallax image in which parallax is added to the frame, an image pair consisting of the frame and the parallax image can be created. At this time, the processor unit can determine the amount of displacement such that the smaller the depth, the larger the amount of displacement. For example, depth and displacement may have a linear relationship. For example, if a first pixel in a frame has a first depth and a second pixel has a depth twice the first depth, the amount of displacement of the first pixel is equal to the amount of displacement of the second pixel. This can be twice the amount of displacement of the pixel.

By shifting the pixels by a displacement amount corresponding to the depth in this manner, the created image pair is displayed three-dimensionally with a more natural appearance. This is because it is in line with human physiological binocular recognition to shift a subject with a small depth (near) by a larger amount and shift a subject with a larger depth (farther) by a smaller amount.

In step S403, the processor unit determines pixels that should be processed and pixels that should not be processed based on the depth information, and processes only the pixels that should be processed (specifically, It is also possible to perform processing to determine the amount of displacement and shift by the amount of displacement), and do nothing to pixels that should not be processed. This has the advantage that the resources required for processing can be reduced, thereby reducing the load on the computer and leading to faster processing.

In step S403, in addition to the above-described processing, the processor unit determines, for each frame of the two-dimensional video, an enlargement rate or reduction rate of at least one region of the frame based on the depth information; At least one area of the frame and at least one corresponding area of the parallax image may be enlarged/reduced at a determined enlargement rate or reduction rate. At this time, the processor unit can determine the enlargement rate and/or the reduction rate so that the smaller the depth is, the larger the enlargement rate is, and/or the larger the depth is, the larger the reduction rate is. . The at least one region may be, for example, a region containing pixels with the same or similar depth.

By enlarging/reducing the image at an enlargement/reduction ratio according to the depth in this manner, the created image pair is displayed in a three-dimensional manner with a more natural appearance. Making objects with a small depth (closer) larger and objects with a larger depth (farther) smaller is in line with human physiological binocular recognition and increases the effect of sensory stereopsis. This is because it is possible. Furthermore, such enlargement/reduction changes the focal point of the image for the viewer, so that the edges of the subject appear to be emphasized. This may cause, for example, objects that are close to each other to appear sharp, have continuous contours, and/or good resolution, while objects that are far away appear blurred, have discontinuous contours, and/or have poor resolution. become. This can also enhance the effect of sensory stereopsis.

In step S404, the processor unit displays the image pair created in step S403 for each frame of the two-dimensional video on the display unit. The processor section can display the image pair on the display section in any three-dimensional display format. The processor unit is configured to display the image pair on the display unit, for example, so that a viewer can view a three-dimensional moving image based on the image pair by wearing a wearable device for 3D display (e.g., 3D glasses). Alternatively, the image pair may be displayed on a display unit so that the viewer can view a three-dimensional moving image of the image pair without having to wear a special attachment device. Preferably, the processor unit is capable of displaying the image pair on the display unit so that a viewer can view a three-dimensional animation of the image pair without having to wear special equipment. As a result, viewers are less likely to suffer from so-called 3D motion sickness, and furthermore, 3D videos can be shared and viewed by multiple people.

FIG. 4B is a flowchart illustrating an example of processing performed by the processor unit in step S403. Step S403 is similarly performed for each frame of the two-dimensional video. Here, we will explain what is done for one frame of a two-dimensional video as an example.

In step S4031, the processor unit determines pixels to be processed and pixels not to be processed based on the depth information. The processor unit may, for example, determine that a pixel whose depth is less than a predetermined threshold is a pixel that should be processed, and a pixel whose depth is greater than or equal to a predetermined threshold as a pixel that should not be processed. . The predetermined threshold value may be a fixed value or a variable value. When the predetermined threshold value is a variable value, the predetermined threshold value may be, for example, the average value, median value, first quartile, or third quartile of the depth of all pixels; For example, the maximum value may be 90%, 80%, 70%, 60%, 40%, or 30% of the maximum value.

In step S4032, i=1 is defined.

In steps S4033 and S4034, the pixels determined to be processed in step S4031 are sequentially processed. Pixels determined as not to be processed are not processed at all. This makes it possible to reduce the resources required for processing, thereby reducing the load on the computer and leading to faster processing.

The order of pixels processed in steps S4033 and S4034 may preferably be from pixels with greater depth to pixels with smaller depth. This is because errors such as a farther object overlapping a closer object are less likely to occur in the created parallax image.

In step S4033, the processor unit determines the amount of displacement for the i-th pixel among the pixels determined as pixels to be processed, based on the depth information. The processor unit can determine the amount of displacement of the i-th pixel, for example, according to a predetermined relationship between depth information and the amount of displacement. The relationship between the predetermined depth information and the amount of displacement may be, for example, a linear relationship, and may be expressed by a relationship such as (amount of displacement)=α×(depth)+β (α and β are constants).

In step S4034, the processor section shifts the i-th pixel in the frame to the right or left by the amount of displacement determined in step S4033. The direction of shift is determined depending on whether the created parallax image is an image for the left eye or an image for the right eye.

In step S4035, i is incremented (ie, i=i+1).

In step S4036, it is determined whether all the pixels determined to be processed in step S4031 have been processed. If it is determined that all pixels have been processed, the process advances to step S4037; if it is determined that all pixels have not been processed yet, the process returns to step S4033, and the process continues from step S4033 to step S4033 until it is determined that all pixels have been processed. Step S4036 is repeated.

In step S4037, a parallax image with pixels shifted based on the depth information is created, and an image pair consisting of the frame and the created parallax image is created.

Note that before step S4037, based on the depth information, the enlargement rate or reduction rate of at least one area of the frame is determined, and the at least one area of the frame and the corresponding at least one area of the image with parallax are Enlargement/reduction may be performed at a determined enlargement rate or reduction rate. By enlarging/reducing at an enlargement/reduction ratio according to the depth, the image pair created in step S4037 can be displayed three-dimensionally with a more natural appearance. Making objects with a small depth (closer) larger and objects with a larger depth (farther) smaller is in line with human physiological binocular recognition and increases the effect of sensory stereopsis. This is because it is possible. Furthermore, such enlargement/reduction changes the focal point of the image for the viewer, so that the edges of the subject appear to be emphasized. This may cause, for example, objects that are close to each other to appear sharp, have continuous contours, and/or good resolution, while objects that are far away appear blurred, have discontinuous contours, and/or have poor resolution. become. This can also enhance the effect of sensory stereopsis.

In this manner, the process 400 creates image pairs from the input two-dimensional video and displays the image pairs, thereby making the two-dimensional video appear three-dimensional. In the process 400, there is no need to perform processes such as combining image pairs to generate a three-dimensional moving image, and the load can be reduced and the process can be speeded up. This can lead to near real-time display. Furthermore, pixels to be shifted by a large amount of displacement may be selectively processed, regions to be enlarged/reduced by large enlargement/reduction ratios may be selectively processed, and/or pixels/regions to be processed may be selectively processed. Selecting and processing the information can also lead to near real-time display.

In the example described above, it has been explained that each step is performed in a specific order, but the order of each step is not limited to that described. Each step can be processed in any logically possible order. For example, step S4034 is not performed after step S4033, and after it is determined in step 4036 that all pixels to be processed have been processed, in step S4034, the target pixels are collectively shifted by the determined displacement amount. You may also do so.

In the example described above, it has been explained that the processing of each step shown in FIGS. 4A and 4B is realized by the processor unit and the program stored in the memory unit, but the present invention is not limited to this. At least one of the processes in each step shown in FIGS. 4A and 4B may be realized by a hardware configuration such as a control circuit.

The present invention is not limited to the embodiments described above. It is understood that the invention is to be construed in scope only by the claims. It will be understood that those skilled in the art will be able to implement the present invention to an equivalent extent based on the description of the present invention and common general technical knowledge from the description of the specific preferred embodiments of the present invention.

Image pairs were created by the system of the present invention using two-dimensional videos taken during surgery.

The microscope used for the surgery was equipped with the following camera.
・NIR camera x 1 (for ICG)
・White light camera x 2 (for left eye, for right eye)
(Center size: 1/1.2 inch CMOS)

The resolution was 1080p (Full HD).

Of the cameras on board, the white light camera for the right eye was used to take pictures. A white camera for the left eye and a NIR camera were not used.

A two-dimensional video captured by a white light camera for the right eye was processed by the system of the present invention. The system of the present invention was implemented using a NVIDIA (GPU) computer using the Python programming language.

5A-5D show image pairs created in this example.

In FIGS. 5A to 5D, (a) on the left is a frame of a two-dimensional video captured by a white light camera, and (b) on the right is a frame created by the system of the present invention.

In FIGS. 5A to 5D, the right eye views the frame (a) on the left (the actual image), and the left eye views the frame (b) on the right (the image created by the system of the present invention). , it can be seen that by superimposing these frames, the image appears three-dimensional. Note that in this example, the actual video is placed on the left and the video created by the system of the present invention is placed on the right, but the placement is not limited to this. For example, the actual image may be placed on the right and the image created by the system of the invention may be placed on the left; again, the superimposed images will appear stereoscopic.

In this way, according to the system of the present invention, image pairs could be created from frames of a two-dimensional video so as to enable stereoscopic viewing.

Furthermore, the system of the present invention was able to process each frame of a two-dimensional video at 30 FPS (30 frames/second) to create an image pair. Thus, the system of the present invention was able to output image pairs in near real time. Furthermore, when the created image pair was displayed on a display device, the moving image was displayed without any perceived delay.

INDUSTRIAL APPLICATION This invention is useful as a system etc. which can display three-dimensionally even a two-dimensional image which does not have depth information.

10 Monocular endoscope 20 Video 100 System 110 Receiving means 120 Depth information acquisition means 130 Image pair creation means 140 Display means

Claims (10)

A system for displaying 2D videos in 3D,
a receiving means for receiving a two-dimensional video;
Depth information acquisition means for acquiring depth information from each frame of the two-dimensional video;
image pair creation means for creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
and display means for displaying the pair of images of each frame of the two-dimensional video. The image pair creation means, for each frame of the two-dimensional video,
determining a displacement amount of at least one pixel of the frame based on the depth information;
creating a parallax image in which parallax is added to the frame by shifting the at least one pixel of the frame by the determined displacement amount; 2. The system of claim 1, wherein the system creates pairs of images. The system according to claim 2, wherein the image pair creation means determines the displacement amount such that the smaller the depth, the larger the displacement amount. The image pair creation means determines pixels to be processed and pixels not to be processed for each frame of the two-dimensional video based on the depth information,
3. The system of claim 2, wherein the at least one pixel is at least one of the to-be-processed pixels. The image pair creation means determines pixels to be processed and pixels not to be processed for each frame of the two-dimensional video based on the depth information,
4. The system of claim 3, wherein the at least one pixel is at least one of the to-be-processed pixels. The image pair creation means, for each frame of the two-dimensional video,
determining an enlargement or reduction ratio of at least one region of the frame based on the depth information;
6. Enlarging/reducing the at least one region of the frame and the corresponding at least one region of the parallax image at the determined enlargement rate or reduction rate. system described in. The image pair creation means determines the magnification rate and/or the reduction rate such that the magnification rate is greater in an area with a smaller depth, and/or the reduction rate is greater in an area with a greater depth. 7. The system of claim 6. 2. The system of claim 1, wherein the display means includes a lenticular display. A method for displaying a two-dimensional video three-dimensionally, the method comprising:
receiving a two-dimensional video;
obtaining depth information from each frame of the two-dimensional video;
creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
displaying the image pair of each frame of the two-dimensional video. A program for displaying a two-dimensional moving image three-dimensionally, the program being executed in a computer including a processor section and a display section, the program comprising:
receiving a two-dimensional video;
obtaining depth information from each frame of the two-dimensional video;
creating an image pair having parallax for each frame of the two-dimensional video using the depth information of each frame of the two-dimensional video;
Displaying the pair of images of each frame of the two-dimensional moving image on the display unit.