WO2022202700A1

WO2022202700A1 - Method, program, and system for displaying image three-dimensionally

Info

Publication number: WO2022202700A1
Application number: PCT/JP2022/012787
Authority: WO
Inventors: ホースーンカン
Original assignee: 株式会社オルツ; ホースーンカン
Priority date: 2021-03-22
Filing date: 2022-03-18
Publication date: 2022-09-29

Abstract

The purpose of the present invention is to provide a method or the like capable of creating a pseudo three-dimensional image in order to display an image three-dimensionally. The present invention provides a method for displaying an image three-dimensionally, the method comprising: receiving an image including a target image; creating a pseudo three-dimensional image with a pseudo three-dimensional effect by adding, in the image, three-dimensional representation of elements other than the target image, by processing the image; and displaying the pseudo three-dimensional image.

Description

Method, program and system for displaying images three-dimensionally

The present invention relates to a method, program and system for displaying images three-dimensionally.

When an image is displayed on a general display device, the image is displayed two-dimensionally. This is because the display surface of the display device is flat.

A special device has been developed for displaying images in three dimensions (for example, Patent Document 1)

Japanese translation of PCT publication No. 2010-513970

An object of the present invention is to provide a method or the like capable of creating a pseudo-three-dimensional image in order to display an image three-dimensionally.

The present invention provides the following items.
(Item 1)
A method for displaying an image three-dimensionally, comprising:
receiving an image containing an image of interest;
processing the image to create a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
and displaying the simulated three-dimensional image.
(Item 2)
Item 1, wherein creating the pseudo-three-dimensional image includes creating a pseudo-three-dimensional animation as the pseudo-three-dimensional image by rotating the three-dimensional representation of the element around the image of the object. described method.
(Item 3)
a portion of the three-dimensional representation of the element is superimposed on the image of the object such that the portion of the image of the object is hidden by the three-dimensional representation of the element; is superimposed under the image of interest and the other portion of the three-dimensional representation of the element is obscured by the image of interest. 2. The method of item 1 or item 2.
(Item 4)
the element includes a plurality of horizontal scan lines;
4. Any one of items 1-3, wherein adding a three-dimensional representation of the element within the image includes adding a three-dimensional representation of the plurality of horizontal scan lines onto the image of the object. The method described in section.
(Item 5)
Creating the pseudo three-dimensional image includes generating a plurality of images with different viewpoints from the image, and temporally successively combining the plurality of images with different viewpoints to form the pseudo three-dimensional image. 5. The method according to any one of items 1 to 4, comprising creating a simulated three-dimensional animation.
(Item 6)
The pseudo three-dimensional image is a pseudo three-dimensional video,
The method includes:
synchronizing sound with the simulated three-dimensional image;
6. The method of any one of items 1-5, further comprising: playing said synchronized sound while displaying said simulated three-dimensional image.
(Item 7)
7. Method according to item 6, wherein the sound changes based on movement in the image.
(Item 8)
8. The method of item 6 or item 7, wherein the sound changes in response to the image of the object exceeding a boundary defined around the image of the object.
(Item 9)
9. The method of item 8, wherein the sound changes in response to movement of the object outside the boundary.
(Item 10)
10. Method according to item 8 or item 9, wherein the boundary is defined by a three-dimensional representation of the elements arranged around the image of the object.
(Item 11)
11. The method of any one of items 7-10, wherein the change in sound includes a change in at least one of loudness, pitch and timbre of the sound.
(Item 12)
displaying the simulated three-dimensional image includes displaying the simulated three-dimensional image on a rotating display in which at least one member rotates about a first axis to form a planar display surface; The method according to any one of items 1-11.
(Item 13)
The rotary display is configured such that the orientation of the display surface can be changed,
The method includes:
detecting a user's position relative to the rotating display;
13. The method of item 12, comprising: reorienting the display surface based on the detected position.
(Item 14)
Displaying the pseudo three-dimensional image includes changing the orientation of the object in the pseudo three-dimensional image based on the orientation of the display surface and displaying the pseudo three-dimensional image on the display surface. 8. The method of item 7, comprising
(Item 15)
Displaying the pseudo-three-dimensional image includes rotating at least one member about a first axis and about a second axis substantially perpendicular to the first axis to form a substantially spherical display surface. 12. The method of any one of items 1-11, comprising displaying the simulated three-dimensional image on a rotating display.
(Item 16)
16. A method according to any one of items 12 to 15, wherein the rotatable display comprises a plurality of members each rotating to form a plurality of display surfaces.
(Item 17)
A program for displaying an image three-dimensionally, said program being executed in a computer system comprising a processor and a display unit, said program comprising:
receiving an image containing an image of interest;
processing the image to create a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
A program causing the processor to perform processing including: displaying the pseudo three-dimensional image on the display unit.
(Item 17A)
18. Program according to item 17, including features according to one or more of the above items.
(Item 18)
A system for three-dimensionally displaying an image, comprising:
receiving means for receiving an image including an image of interest;
creating means for creating, by processing the image, a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
and display means for displaying the pseudo three-dimensional image.
(Item 18A)
19. The system of item 18, including the features of one or more of the above items.
(Item 18B)
A storage medium storing a program for three-dimensionally displaying an image, the program being executed in a computer system comprising a processor and a display unit, the program comprising:
receiving an image containing an image of interest;
processing the image to create a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
A storage medium causing the processor to perform processing including: displaying the pseudo three-dimensional image on the display unit.
(Item 18C)
A storage medium according to item 18B, comprising features according to one or more of the above items.
(Item 19)
A method for displaying an image three-dimensionally, comprising:
receiving an image;
synchronizing sound with the image, wherein the sound changes in response to movement in the image; displaying the image;
and playing said synchronized sound while displaying said image.
(Item 19A)
20. Method according to item 19, including features according to one or more of the above items.
(Item 20)
A program for three-dimensionally displaying an image, said program being executed in a computer system comprising a processor, a display unit, and a sound output unit, said program comprising:
receiving an image;
synchronizing sound with the image, wherein the sound changes in response to movement in the image; displaying the image on the display;
and reproducing the synchronized sound from the sound output unit when the image is displayed.
(Item 20A)
21. Program according to item 20, including features according to one or more of the above items.
(Item 21)
A system for three-dimensionally displaying an image, comprising:
a receiving means for receiving an image;
synchronization means for synchronizing sound with said image, said sound varying in response to movement in said image; display means for displaying said image;
reproduction means for reproducing said synchronized sound when displaying said image.
(Item 21A)
22. The system of item 21, including the features of one or more of the above items.
(Item 22)
A method of displaying an image on a display, comprising:
detecting the position of a user's viewpoint with respect to the display;
Determining the portion of the image to be displayed on the display by processing the image, comprising:
setting a virtual sphere centered at the user's viewpoint and having a radius equal to the distance between the user's viewpoint and the display;
pasting the image onto the inner surface of the virtual sphere;
identifying a portion of the image pasted on a portion of the inner surface of the phantom sphere corresponding to the display surface of the display;
including
and displaying the determined portion of the image on the display surface of the display.
(Item 23)
23. The method of item 22, wherein the image is represented in an equirectangular projection.
(Item 24)
24. The method of item 22 or item 23, wherein the display is a stationary display.
(Item 25)
25. The method of item 24, wherein the display is a rotating display in which at least one member rotates to form a viewing surface.

According to the invention, it is possible to create a pseudo three-dimensional image from the image to be displayed, so that, for example, even an image for which no three-dimensional model exists can be represented three-dimensionally. become.

FIG. 10 is a diagram showing how an image 10 is displayed; FIG. 1B shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of one embodiment of the present invention. FIG. 1B is a diagram showing an example in which the target image 11 is three-dimensionally expressed by further enhancing the perspective of the target image 11 in the pseudo three-dimensional image 20 shown in FIG. 1B; FIG. 1B is a diagram showing an example in which the target image 11 is three-dimensionally expressed by further enhancing the perspective of the target image 11 in the pseudo three-dimensional image 20 shown in FIG. 1B; A diagram showing a process in which the target image 11 moves A diagram showing a process in which the target image 11 moves The figure which shows a mode that the image 11 of object moves. A diagram showing the process in which the cube 12 moves A diagram showing the process in which the cube 12 moves The figure which shows a mode that the cube 12 moves. FIG. 1B shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of another embodiment of the invention. FIG. 2B is a diagram showing an example in which the target image 11 is three-dimensionally expressed by further enhancing the perspective of the target image 11 in the pseudo three-dimensional image 30 shown in FIG. 2A; A diagram showing a pseudo three-dimensional image 20' A diagram showing a pseudo three-dimensional image 20 ″ FIG. 1B shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of another embodiment of the invention. FIG. 1B shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of another embodiment of the invention. FIG. 1 shows an example of an image display device for displaying a pseudo three-dimensional image; 4B is a diagram showing an example of an image displayed on the display surface 23 of the rotary display 20 in the orientation of the display surface shown in FIG. 4A; FIG. A diagram showing a state in which the user U has moved around the rotary display 20 in the direction of the arrow. FIG. 4D is a diagram showing an example of an image displayed on the display surface 23 facing the user position shown in FIG. 4C. FIG. 4 is a diagram showing how an image is displayed on the display surface 23 of the rotary display 20; FIG. 2 shows a rotating display 25 in one embodiment of the present invention; FIG. 3 shows a rotating display 27 in another embodiment of the invention; FIG. 2 schematically illustrates an example flow of a technique in one embodiment of the present invention; FIG. 4 is a diagram when the distance between the viewpoint of the user U and the display 20 is small; FIG. 4 is a diagram showing a case where the distance between the viewpoint of the user U and the display 20 is large; A diagram showing an example of the configuration of a system 100 for displaying an image three-dimensionally. A diagram showing an example of the configuration of a system 100' for three-dimensionally displaying an image in another embodiment. A diagram showing an example of the configuration of the user device 200 FIG. 7 shows an example of a process 700 by the system 100 for displaying an image three-dimensionally. FIG. 7 illustrates an example process 710 by system 100' for displaying an image in three dimensions; FIG. 8 shows an example process 800 for displaying an image on a display.

The present invention will be described below. It should be understood that the terms used herein have the meanings commonly used in the art unless otherwise specified. Thus, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In case of conflict, the present specification (including definitions) will control.

In this specification, "image" refers to an image that can be displayed on a two-dimensional plane. The image includes not only a "two-dimensional image" containing two-dimensional information (length x width) but also a "three-dimensional image" containing three-dimensional information (length x width x depth). A "three-dimensional image" can be acquired, for example, using an RGB-D camera. A "three-dimensional image" can be obtained, for example, by performing a process of estimating depth information on a two-dimensional image and adding the depth information to the two-dimensional image. Images include still images and moving images. A moving image is considered to be a plurality of temporally consecutive still images.

In this specification, "three-dimensional" refers to a state that is not three-dimensional, but appears to be three-dimensional.

In this specification, "three-dimensionally displaying" means displaying something that is not three-dimensional (for example, something that is in a two-dimensional plane) as if it were three-dimensional.

In this specification, "three-dimensional representation" means representation of something that is not three-dimensional (for example, something in a two-dimensional plane) as if it were three-dimensional. For example, the three-dimensional representation includes a three-dimensional representation by adding shading, a three-dimensional representation by adding light and shade, and a three-dimensional representation by adding parallax. One that is expressed as if it exists, one that is expressed as if it were three-dimensional by adding a difference in size, or one that is expressed as if it is three-dimensional by adding perspective At least one.

In this specification, the "pseudo three-dimensional effect" refers to the effect of appearing three-dimensionally due to visual illusion (optical illusion). Depending on the viewer's perception, there can be varying degrees of pseudo three-dimensional effect. A "pseudo three-dimensional image" refers to an image that produces a "pseudo three-dimensional effect".

In this specification, "object" refers to any object appearing in an image. A subject may be, for example, animate or inanimate. A subject may be, for example, a human, an animal, or a plant.

As used herein, "about" means ±10% of the following numerical value.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1.1 Method for Three-Dimensionally Representing Images The inventors of the present invention have developed a method for three-dimensionally representing an image (two-dimensional image) displayed on a flat display. When this method is used, the image displayed on the flat display is displayed three-dimensionally, and the person viewing the flat display sees the image displayed on the flat display as if it were displayed in a three-dimensional space. can be misunderstood. A pseudo three-dimensional image is an image that produces such an illusory effect. With this method, an image can be represented three-dimensionally even if a three-dimensional model of the image does not exist.

FIG. 1A shows how the image 10 is displayed. The image 10 includes an image 11 of an object (a person in this example).

Since the image 10 is displayed on a flat display, the target image 11 appears two-dimensional or two-dimensional.

FIG. 1B shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of one embodiment of the present invention.

In the pseudo three-dimensional image 20 shown in FIG. 1B, compared to the original image 10, an element (cube 12 in this example) different from the target image 11 is added. A cube 12 has been added to the image in a three-dimensional representation. For example, as shown in FIG. 1B, the cube 12 is represented as thicker on the side closer to the viewer and thinner on the side farther from the viewer. As a result, the cube 12 has a sense of perspective. In the pseudo three-dimensional image 20, the perspective of the cube 12 also causes the object image 11 to have a perspective, so that the object image 11 can appear three-dimensional.

Further, the cube 12 is positioned around the target image 11 , the cube 12 overlaps the target image 11 such that a portion of the cube 12 obscures a portion of the target image 11 and a portion of the cube 12 . It overlaps the target image 11 so that the part is hidden by a part of the target image 11 . This may enhance the perspective of the cube 12 and, in turn, the perspective of the image 11 of the object.

In this way, in the pseudo three-dimensional image 20, although the target image 11 itself is a two-dimensional image, the existence of the cube 12 makes it easy to misunderstand that the target image 11 is represented three-dimensionally.

1C and 1D show examples in which the perspective of the target image 11 is further enhanced in the pseudo three-dimensional image 20 shown in FIG. 1B to express the target image 11 three-dimensionally. In the examples shown in FIGS. 1C and 1D, the pseudo three-dimensional image 20 is a moving image, and the images shown in FIGS. 1C and 1D can be considered one frame of the moving image.

1C and 1D, the cube 12 is rotated around its axis. The axis is, for example, the central axis passing through the top and bottom surfaces of the cube 12 . As the cube 12 is rotated, the edges near and far from the viewer transition, as shown in FIGS. 1C and 1D. Along with this, the edge represented thick and the edge represented thin transition. This further enhances the perspective of the rotated cube 12 . In image 10, the enhanced perspective of cube 12 also enhances the perspective of image of object 11, which may make image of object 11 appear more three-dimensional.

Furthermore, the cube 12 is rotated around the image 11 of the object. As the cube 12 is rotated, the portion of the cube 12 obscuring the portion of the image 11 of interest and the portion of the cube 12 obscuring the portion of the image 11 of the object transition. This may further enhance the perspective of the cube 12 and, in turn, the perspective of the image 11 of the object.

In this way, in the pseudo three-dimensional image 20, although the target image 11 itself is a two-dimensional image, the presence of the rotated cube 12 makes the target image 11 appear three-dimensional. easy to be

It should be noted that the axis can be any axis. The axis is preferably the axis whose rotation enhances the perspective of the cube 12 . The axis may be, for example, a central axis passing through a side surface of the cube 12, an axis (a central axis or an off-center axis) passing through at least one surface of the cube 12, or a 12 out of the axis.

In the above example, it was explained that the image is three-dimensionally represented by a visual effect by elements other than the target image 11, but in one embodiment of the present invention, in addition to the visual effect, or Instead of visual effects, auditory effects can be used to represent images in three dimensions. It is assumed that the flat display in this embodiment has a speaker or is connected to a speaker.

For example, assume that the image 10 is a moving image that expresses how the target moves. Image 10 will include image 11 of a moving object. Assume that the target image 11 moves from the state shown in FIG. 1E to the state shown in FIG. 1F in the pseudo three-dimensional image 20 generated from such an image 10 . This movement causes the image of the subject's arm to appear to be outside the cube 12 . In the process of changing the image of the subject from the state shown in FIG. 1E to the state shown in FIG. The sound can change as it extends farther. Sound may be played from the speaker. The sound played at the moment the subject's arm image touches the cube 12 creates the illusion that the cube 12 is in three-dimensional space, and the distance between the cube 12 and the arm image. By changing the sound according to , it is possible to create the illusion of a three-dimensional spread of space. The change in sound can be, for example, at least one of loudness, pitch of sound, and timbre of sound. For example, the sound can increase as the subject's arm image extends farther from the cube 12 . Alternatively, for example, the sound may decrease as the subject's arm image extends further from the cube 12, or, for example, the sound may increase or decrease as the subject's arm image extends further from the cube 12. or, for example, the subject's arm image may approach a different tone as it extends farther from the cube 12 . Such an auditory effect emphasizes the presence of the cube 12, which in turn further emphasizes the perspective of the image 11 of interest.

Further, the sound may vary in response to other motions of the subject's image outside the cube 12 in addition to or alternatively to movement of the subject's arm image away from the cube 12 . . For example, as shown in FIG. 1G, when the subject's arm image moves up and down in an arc outside the cube 12 in the direction of the arrow, the sound can be varied to match the movement. For example, the sound can be louder (or softer) as the subject's arm image moves down in an arc outside the cube 12 in the direction of the arrow, and softer (or louder) as it moves up. Or, for example, make the sound higher (or lower) as the image of the subject's arm moves downward in an arc outside the cube 12 in the direction of the arrow, and lower (or higher) as it moves upward. Alternatively, for example, as the subject's arm image moves downward in an arc outside the cube 12 in the direction of the arrow, the sound approaches a first timbre, and as it moves upward, the sound shifts to a second timbre. You can bring it closer. Such an auditory effect also emphasizes the presence of the cube 12, which in turn further emphasizes the perspective of the image 11 of interest.

Thus, the presence of the cube 12 can be emphasized by playing and varying the sound according to the relationship between the cube 12 and at least a portion of the image of interest 11, and the image of interest 11 is: If it is represented three-dimensionally, it becomes easier to be illusioned. The relationship between the cube 12 and at least a portion of the target image 11 may be, for example, the distance between the cube 12 and the portion of the target image 11 (i.e., static relationship), as described above, It may be the movement of a portion of the image of interest 11 relative to the cube 12 (ie dynamic relationship) or other relationship.

In the above example, it was explained that the relationship with the cube 12 changes as the target image 11 moves, but the present invention is not limited to this. For example, as described above with reference to FIGS. 1C and 1D, as cube 12 moves, the relationship between cube 12 and at least a portion of object image 11 also changes, and cube 12 and object image 11 change accordingly. The sound may be reproduced and changed according to the relationship with at least part of 11 .

For example, in FIGS. 1H-1J, the cube 12 is rotated about its axis, similar to FIGS. 1C and 1D. In FIG. 1H, the image of the object 11 is contained within the cube 12 . Therefore, no sound is reproduced from the speaker.

When the cube 12 is rotated about its axis, the subject's arm image appears to be outside the cube 12, as shown in FIG. 1I. In the process of changing the cube 12 from the state shown in FIG. 1H to the state shown in FIG. You can change the sound as you go. Sound may be played from the speaker. The sound played at the moment the cube 12 touches the subject's arm image creates the illusion that the cube 12 is in a three-dimensional space, and the distance between the cube 12 and the arm image. By varying the sound according to distance, the illusion of three-dimensional expansion of space can be created. The change in sound can be, for example, at least one of loudness, pitch of sound, and timbre of sound. For example, the sound can be louder as the cube 12 is farther from the subject's arm image. Alternatively, for example, the sound may decrease as the cube 12 moves away from the target's arm image, or the sound may increase or decrease as the cube 12 moves away from the target's arm image. However, the further away the cube 12 is from the target arm image, the closer to a different tone it may be. Such an auditory effect emphasizes the presence of the cube 12, which in turn further emphasizes the perspective of the image 11 of interest.

Furthermore, in addition to the movement of the cube 12 moving away from the target arm image, the sound may be changed according to the movement of the target image outside the cube 12 . For example, as shown in FIG. 1J, when the image of the subject's arm moves up and down in an arc outside the cube 12 in the direction of the arrow and/or when the image of the subject's arm moves outside the cube 12 to the direction of the arrow. When moving left and right in the direction of , the sound can be changed according to the movement. For example, the sound can be louder (or softer) as the subject's arm image moves down in an arc outside the cube 12 in the direction of the arrow, and softer (or louder) as it moves up. Or, for example, make the sound higher (or lower) as the image of the subject's arm moves downward in an arc outside the cube 12 in the direction of the arrow, and lower (or higher) as it moves upward. Alternatively, for example, as the subject's arm image moves downward in an arc outside the cube 12 in the direction of the arrow, the sound approaches a first timbre, and as it moves upward, the sound shifts to a second timbre. You can bring it closer. For example, the sound may be louder or softer as the subject's arm image extends farther from the cube 12, e.g. or, for example, the subject's arm image may approach a different tone as it extends farther from the cube 12 . Such an auditory effect also emphasizes the presence of the cube 12, which in turn further emphasizes the perspective of the image 11 of interest.

In the example above, with the cube 12 as the boundary, changing the sound as a portion of the image of interest extends farther from the boundary or changing the sound in response to a portion of the image of interest moving outside the boundary. , but the boundary is not limited to this. The boundary can be any boundary as long as it is defined near, eg, around the image of interest. For example, the boundary may be visible, such as cube 12, or invisible. If the boundaries are not visible, auditory effects will render the image three-dimensionally without relying on visual effects.

The boundary can have any shape. For example, it may be a shape that surrounds the target image (e.g., spherical, elliptical, cylindrical, prismatic, etc.), or a shape that does not surround the target image (e.g., planar, curved, hemispherical). shape, etc.). The boundary may change over time or may not change over time. For example, as in the example above where the boundary is represented by cube 12, the boundary may rotate about an axis over time.

Any sound can be played and changed. The sound may be, for example, a sound that is directly related to the image, a sound that is somewhat related to the image, or a sound that is unrelated to the image. Preferably, the sound may be sound unrelated to the image, more preferably sound directly related to the image. Sound that is directly related to the image may, for example, have been synchronized to the original image 10 (eg, if the original image 10 was a still image with sound or a moving image with sound). sound. Sounds that are somewhat related to images are, for example, sounds that are associated with images (e.g., bird wing sounds or chirping for bird images, car running sounds or horn sounds for car images, etc.). can be Even if the sound was synchronized to the original image 10 , the sound can be a sound other than the sound that was synchronized to the original image 10 .

In the above examples, visual effects and/or auditory effects are used to represent images in three dimensions. It is also possible to represent an image three-dimensionally.

FIG. 2A shows an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of another embodiment of the invention.

In the pseudo three-dimensional image 30 shown in FIG. 2A, compared to the original image 10, an element (horizontal scanning line 13 in this example) different from the target image 11 is added and displayed. A horizontal scan line 13 has been added over the image 11 of the object. Horizontal scan lines 13 are then added to represent the contour shape of the object, based on the three-dimensional information contained in or derived from the image 10 . For example, as shown in FIG. 2A, the horizontal scan line 13 is represented as curving along the curved surface of the subject's face and curving along the contours of the subject's nose. In the pseudo-three-dimensional image 30, the horizontal scanning lines 13 express the contour shape of the object, so that the object image 11 has a sense of perspective, so that the object image 11 can be seen three-dimensionally.

Thus, in the pseudo three-dimensional image 30, although the target image 11 itself is a two-dimensional image, the presence of the horizontal scanning lines 13 gives the illusion that the target image 11 is represented three-dimensionally. Cheap.

FIG. 2B shows an example in which the target image 11 is three-dimensionally expressed by further enhancing the perspective of the target image 11 in the pseudo three-dimensional image 30 shown in FIG. 2A.

In FIG. 2B, the cube 12 described above with reference to FIG. 1B has been added around the image 11 of interest. As described above, the perspective of the cube 12 also causes the image of the object 11 to have a perspective, so that the image of the object 11 can appear three-dimensional.

The cube 12 can be rotated about its axis as described above with reference to Figures 1C and 1D. As a result, the pseudo three-dimensional image 30 becomes a moving image. By rotating the cube 12 about its axis, the perspective of the image 11 of the object can be further enhanced.

Further, as described above with reference to FIGS. 1E-1J, the presence of the cube 12 can be detected by playing and varying the sound according to the relationship between the cube 12 and at least a portion of the image 11 of interest. It can also be enhanced, further enhancing the perspective of the image 11 of the object.

Thus, in the pseudo-three-dimensional image 30, although the image of interest 11 itself is a two-dimensional image, the presence of the horizontal scan lines 13 as well as the presence of the cube 12 or the rotated cube 12, and also the presence of the cube 12 and the cube 12 to be rotated. Due to the presence of sounds that reproduce and change according to their relationship with a portion of the image 11 of the object, the image 11 of the object is more likely to give the illusion of being represented three-dimensionally.

In the above example, the element added to the image (the cube 12 in the above example) overlaps the target image 11, but the element added to the image does not necessarily overlap the target image 11. No need. Elements can be added anywhere in the image as long as it produces a pseudo three-dimensional effect. For example, the added elements can be placed adjacent to the image of interest 11, as shown in the pseudo-three-dimensional image 20' of FIG. 2C. The three-dimensional representation of the added elements introduces some perspective in the image and may also introduce some perspective in the image 11 of interest. This allows the image 11 of the object to appear three-dimensional.

Also, in the above example, an example in which one element (the cube 12 in the above example) is added to the image 10 has been described, but the number of added elements is not limited to this. For example, multiple elements 12 can be added to the image 10, as shown in the pseudo-three-dimensional image 20'' of FIG. 2D. The added elements may, for example, be rotated about their respective axes, and at least some of the elements may be rotated, as shown in the pseudo-three-dimensional image 20'' of FIG. 2D. may be rotated about an axis common to . The presence of the added element or the rotation of the added element creates perspective in the image and may also create perspective in the image 11 of interest. This allows the image 11 of the object to appear three-dimensional.

By using the above-described method to add elements that originally do not exist in the image 10 to the pseudo-three-dimensional image, the pseudo-three-dimensional image gives a strong impression that it is a virtual image.

3A-3B show an example three-dimensional representation of the image 11 of the object in the image 10 shown in FIG. 1A, according to the technique of another embodiment of the present invention. In the examples shown in FIGS. 1C and 1D, the pseudo three-dimensional image is a moving image, and the still

images

41, 42 shown in FIGS. 3A and 3D can be considered one frame of the moving image.

In FIG. 3A, a still image 41 created from the image 10 shown in FIG. 1A and viewed from the first line-of-sight direction is displayed. A technique for creating images with different line-of-sight directions from a given image may be a technique known in the art. For example, based on the three-dimensional information contained in the image or the three-dimensional information derived from the image, images with different line-of-sight directions can be created from a given image. For example, machine learning techniques can be used to create images with different viewing directions from an image. For example, if image 10 is a moving image, a still image can be generated for each frame of the moving image.

The first line-of-sight direction is the line-of-sight direction when the object is viewed from a more left direction than the line-of-sight direction of the image 10 shown in FIG. 1A.

In FIG. 3B, a still image 42 created from the image 10 shown in FIG. 1A is displayed, looking at the object from the second line-of-sight direction. As described above, the method of creating images with different line-of-sight directions from a certain image may be a method known in the art.

The second line-of-sight direction is the line-of-sight direction when the object is viewed from a more right direction than the line-of-sight direction of the image 10 shown in FIG. 1A.

In the example shown in FIGS. 3A and 3B, the created still

images

41 and 42 are temporally continuously combined to generate and display a pseudo three-dimensional moving image. A pseudo three-dimensional moving image having two frames of still images from different viewpoints gives the target image 11 a sense of perspective due to the parallax generated from the different viewpoints. This allows the image 11 of the object to appear three-dimensional.

In the generated pseudo-three-dimensional video, for example, still

images

41 and 42 may appear alternately and repeatedly. This can generate animations of arbitrary length. Alternatively, when the image 10 is a moving image, the still

images

41 and 42 created from each frame may appear continuously in frame order in the generated moving image. At this time, the frame rate can be set to any value. For example, maintaining the frame rate of the image 10 may generate a pseudo-3D animation that is twice the length of the image 10 . For example, by doubling the frame rate, a pseudo-three-dimensional video of the same length as the image 10 can be generated.

The above-described elements (eg, element 12, horizontal scanning line 13) may be added to the generated pseudo-three-dimensional video, and sound may be played along with the image. Thereby, the perspective of the target image 11 can be enhanced. If the element 12 is added, the element 12 can be rotated around the axis. This can further enhance the perspective of the target image 11 .

With the above-described method, for example, even if there is no 3D model of the target and only a 2D image of the target, the target image can be represented in 3D. Of course, even if there is a three-dimensional model of the object, it is possible to apply the technique described above.

FIG. 4A shows an example of an image display device for displaying a pseudo three-dimensional image.

In this example, the image display device is a rotary display 20 (also called a "hologram display") in which at least one member 21 rotates to form a display surface. At least one member 21 is rotatable around a rotation axis C1. By rotating at least one linear member 21, it is possible to form a planar display surface. A light source (for example, an LED) is arranged on at least one member 21 . Light emission from the light source on at least one member 21 is controlled according to the rotation angle of at least one member 21, so that an image can be projected onto the display surface by the afterimage effect. In the rotatable display 20, the background can be seen through the rotation of at least one member 21, so there is an effect that the image appears as if it is floating in the air.

Also, the frame rate of the image displayed on the rotary display 20 depends on the rotation speed of at least one member 21 . In general, the frame rate of images displayed on rotating display 20 is significantly lower than the frame rate of images displayed on typical display devices. The frame rate of rotating display 20 can be, for example, from about 20 fps to about 40 fps, such as about 30 fps. As a result, the image displayed on the rotary display 20 can be rougher than the image displayed on a typical display device. By displaying a rough image, the impression that the image displayed on the rotary display 20 is a virtual image is enhanced.

The rotary display 20 has a main body 22. The main body 22 is configured to be rotatable around the rotation axis C2. By rotating the main body 22 around the rotation axis C2, the orientation of the display surface formed by the at least one member 21 can be changed. For example, the rotatable display 20 detects the position of the user viewing the rotatable display 20 by a detection means (not shown), and rotates the main body 22 around the rotation axis C2 so as to face the detected position of the user U. can be rotated to

FIG. 4B shows an example of an image displayed on the display surface 23 of the rotary display 20 in the orientation of the display surface shown in FIG. 4A.

In the example shown in FIG. 4B, the target image 11 is displayed on the display surface 23, similar to the example shown in FIG. 1A. In the image 11 of the object, the object faces the front. As a result, the user U viewing the display surface sees the front side of the object.

The display surface 23 can display the pseudo three-dimensional image described above with reference to FIGS. 1B to 3B. By displaying the pseudo three-dimensional image on the display surface 23 through which the background of the rotary display 20 is visible, the pseudo three-dimensional image appears as if it is floating in the air, and the pseudo three-dimensional image is displayed. Three-dimensional feeling can be emphasized. In addition, by displaying the pseudo three-dimensional image on the display surface 23 of the rotary display 20 having a significantly low frame rate, the impression that the pseudo three-dimensional image is a virtual image is enhanced.

For example, as shown in FIG. 4C, when the user U moves around the rotatable display 20 in the direction of the arrow and moves to the left of the rotatable display 21, the rotatable display 20 detects the detection means (not shown). ) to detect the position of the user U and rotate the main body 22 clockwise around the rotation axis C2 so as to change the orientation of the display surface. As a result, the display surface of the rotary display 20 faces the user U. As shown in FIG. The user U can see the display surface of the rotary display 20 even after moving.

FIG. 4D shows an example of an image displayed on the display surface 23 facing the user position shown in FIG. 4C.

Since the user position shown in FIG. 4C has moved to the left with respect to the user position shown in FIG. 4A, the user U is facing forward in the image displayed on the display surface shown in FIG. 4A. can be viewed from the left side. Therefore, on the display surface 23 facing the user position shown in FIG. 4C, it is possible to display an image 11' of a front facing object viewed from the left side. This may give the user U the illusion that the object in the object images 11, 11' is a three-dimensional object. Such an illusion can be further enhanced by having the display surface of rotating display 21 directed toward user U at both the user position shown in FIG. 4A and the user position shown in FIG. 4C. This is because the display surface always faces the user U, so the user U does not easily perceive that the display surface is planar.

By displaying the pseudo-three-dimensional image described above with reference to FIGS. 1B to 3B on such a display surface 23, the pseudo-three-dimensional effect can be further emphasized.

For example, according to the rotatable display 20 described above, the user U can face the display surface 23 from any angular position with respect to the rotatable display 20, and can visually recognize an image without distortion. For example, even if the user U1 views the rotary display 20 from any angular position with respect to the rotary display 20, the image of the horse displayed on the display surface 23 is not distorted, as shown in FIG. 4E(a). , is presented to user U1. On the other hand, when user U1 is viewing the display, if another user U2 tries to view rotary display 20 from a different angular position than user U1, display surface 23 of rotary display 20 will not be visible to user U2. As a result, user U2 sees a distorted image. For example, the horse image displayed on the display surface 23 is distorted and presented to the user U2 as shown in FIG. 4E(b).

In one embodiment of the present invention, the rotating display 25 rotates at least one member 26 about a first rotation axis C1 and rotates at least one member 26 about a second rotation axis C2. Thus, a three-dimensional display surface can be formed. The second axis of rotation C2 may be substantially perpendicular to the first axis of rotation C1. In FIG. 4F(a), the direction of rotation about the first rotation axis _C1 is indicated by RC1, and the direction of rotation about the second rotation axis _C2 is indicated by RC2. By rotating at least one linear member 26 about a first rotation axis C1 and a second rotation axis C2, a substantially spherical display surface can be formed. A light source (eg, an LED) is disposed on at least one member 26 . Light emission from the light source on at least one member 26 is controlled according to the rotation angle of the at least one member 26, so that an image can be projected onto the substantially spherical display surface by the afterimage effect. For example, the rotary display 25 can form a substantially spherical display surface, as shown in FIG. 4F(b). Except for the configuration described above, the rotary display 25 may have the same configuration as the rotary display 20 described above.

With such a substantially spherical display surface, even if another user U2 views the rotary display 25 from a different angular position from the user U1 while the user U1 is viewing the display, User U2 can visually recognize the undistorted image in the same way as user U1. For example, even when a plurality of other users view the rotatable display 25 from different angular positions than the user U1, each of the plurality of users can visually recognize the undistorted image similarly to the user U1. can.

The pseudo-three-dimensional image described above with reference to FIGS. 1B to 3B can be displayed on the substantially spherical display surface of the rotary display 25 . By displaying the pseudo-three-dimensional image on the display surface of the rotary display 25 through which the background can be seen, the pseudo-three-dimensional image appears as if it is floating in the air. A sense of dimension can be emphasized. Moreover, by displaying the pseudo three-dimensional image on the display surface of the rotary display 25 having a significantly low frame rate, the impression that the pseudo three-dimensional image is a virtual image is enhanced. Furthermore, since the undistorted pseudo three-dimensional image can be viewed from any angular position with respect to the rotary display 25, the three-dimensional feel of the pseudo three-dimensional image can be emphasized.

It has been explained that the rotary displays 20, 25 described above rotate at least one member 21, 26 around a common axis to form one planar display surface and one substantially spherical display surface. The invention is not limited to this. For example, it is within the scope of the invention to rotate multiple members about their respective axes to form multiple display surfaces.

FIG. 4G shows an example of the rotating display 27 in one embodiment. The rotary display 27 has a first display surface 28 formed by rotating at least one first member and a second display surface 29 formed by rotating at least one second member. is configured to form a The first member and the second member can be rotated about two axes to each form a substantially spherical viewing surface. In the example shown in FIG. 4G, the first member and the second member are rotated about a common axis (body axis). Except for the configuration described above, rotary display 27 may have a configuration similar to

rotary display

20 or 25 described above.

By forming multiple display surfaces in this way, the display area can be expanded. For example, a separate image may be displayed on each display surface, or one image may be displayed over a plurality of display surfaces. Multiple display surfaces expand the range of video expression.

The pseudo three-dimensional image described above with reference to FIGS. 1B to 3B can be displayed on the substantially spherical display surface of the rotary display 27. FIG. By displaying the pseudo-three-dimensional image on the display surface of the rotary display 27 through which the background is visible, the pseudo-three-dimensional image appears as if it is floating in the air. A sense of dimension can be emphasized. Further, by displaying the pseudo three-dimensional image on the display surface of the rotary display 27 having a significantly low frame rate, the impression that the pseudo three-dimensional image is a virtual image is enhanced. Furthermore, since the undistorted pseudo three-dimensional image can be visually recognized from any angular position with respect to the rotary display 27, the three-dimensional feeling of the pseudo three-dimensional image can be conspicuous. Furthermore, it is possible to express a variety of pseudo three-dimensional images using a plurality of display surfaces.

In the above example, the pseudo three-dimensional image is displayed on the special

rotary displays

21, 25, and 27, but the image display device that displays the pseudo three-dimensional image is not limited to this. The pseudo three-dimensional image can be displayed on any other image display device. Preferably, the image display device can be a transparent display through which the background can be seen. This is because the pseudo-three-dimensional effect of the pseudo-three-dimensional image displayed is enhanced. Additionally or alternatively, the image display device may be a display with a significantly lower frame rate. As a result, the pseudo three-dimensional image is displayed at a low frame rate, the impression that the pseudo three dimensional image is a virtual image is enhanced, and the pseudo three dimensional effect can be enhanced. is.

1.2 Method for Providing Virtual Reality Images Without Relying on Dedicated Display Device developed a method to By using this technique, the user can enjoy natural virtual reality images without discomfort without wearing a dedicated display device. For example, using this technique, it becomes possible to display a natural virtual reality image on a stationary display or on a rotating display such as 20, 25, 27 described above.

FIG. 5A schematically illustrates an example flow of a technique in one embodiment of the invention.

First, an image 51 that is the basis of an image displayed as a virtual reality image is acquired. The image 51 is preferably represented in a specific drawing method. A particular projection may be, for example, an equirectangular projection (also called an equirectangular projection). In the equirectangular projection, the lines of latitude and longitude are at right angles and the lines of latitude and longitude intersect at regular intervals. As a result, the distance between the two points is represented correctly. The equirectangular projection is a projection that is often used when displaying virtual reality images. As shown in FIG. 5A, an image 51 represented by the equitangler projection appears to have distortion in the image 51 .

Next, the image 51 is pasted on the inner surface of the virtual sphere 52 . For example, if an image represented by an equirectangular projection is pasted on the inner surface of a sphere, a natural image without distortion can be generated.

The virtual sphere 52 is a virtual sphere whose center is the viewpoint of the user U and whose radius is the distance between the viewpoint of the user U and the display 20 that displays the virtual reality image. For example, when the distance between the user U's viewpoint and the display 20 is small, as shown in FIG. When the distance from the display 20 is large, the diameter of the phantom sphere 52 is large.

The distance between the viewpoint of the user U and the display 20 can be measured, for example, by sensing means (not shown) that the display 20 may have. The detection means can detect the position of the user's U eyes and measure the distance between the user's U viewpoint and the display 20 using techniques known in the field of distance measurement.

Next, the portion of the image 51 pasted on the portion of the inner surface of the phantom sphere 52 corresponding to the display surface of the display 20 is specified, and the image 53 of the specified portion is displayed on the display surface of the display 20. User U can see image 53 .

For example, as shown in FIG. 5B, when the distance between the user U's viewpoint and the display 20 is small, the diameter of the virtual sphere 52 is small, so the inner surface of the virtual sphere 52 corresponding to the display surface of the display 20 is The part of the image pasted on the part becomes relatively large. On the other hand, as shown in FIG. 5C, when the distance between the viewpoint of the user U and the display 20 is large, the diameter of the virtual sphere 52 is large. The portion of the image pasted on the inner surface of the is relatively small. That is, when the user U approaches the display, a larger area within the image 51 will be displayed so as to represent the perspective between the user U and the display, and when the user U moves away from the display, the user A smaller area within the image 51 will be displayed to represent the perspective between U and the display.

For example, focus on the car 54 in the image 53 in FIGS. 5B and 5C. In the image 53 shown in FIG. 5B, the car 54 is displayed small on the display 20, but since the user U is close to the display 20, the user U perceives the display 20 itself to be large, and thus the car 54 is also correspondingly large. perceive it as size. When the user U moves away from the display 20, the car 54 is displayed large on the display 20 as in the image 53 shown in FIG. 5C. perceive it as size. As a result, the user U perceives the car 54 to be substantially the same size when approaching the display 20 and when moving away from the display 20 .

Through the image displayed on the display 20, an object in the foreground in the image is perceived as close to the user U, regardless of whether the user U approaches the display or moves away from the display, and an object in the background in the image is perceived. is perceived as far from the user U, whether the user U is closer to the display or farther from the display. Since the perspective of the image displayed on the display 20 is maintained in this manner, the user U can view the image through the display 20 with a feeling of the real world. In other words, the user U can experience virtual reality images through the display 20 .

In the above example, the image is displayed on the rotary display 20, but the present invention is not limited to this. The image can be displayed on any display as long as the distance between the user U and the display can be measured.

The technique of three-dimensionally representing an image and the technique of providing a virtual reality image described above can be implemented, for example, by the system 100 for three-dimensionally displaying an image, which will be described later.

2. Configuration of System for Three-Dimensional Display of Images FIG. 6A shows an example of the configuration of a system 100 for three-dimensional display of images.

The system 100 comprises receiving means 110 , creating means 120 and displaying means 130 .
The receiving means 110 are arranged to receive images. The receiving means 110 can receive images in any manner. The received image contains the image of the object. The receiving means 110 may receive an image from outside the system 100, or may receive an image from inside the system 100 (for example, from a storage means that the system may have). When receiving an image, the receiving means 110 may, for example, receive the image from a storage medium connected to the system 100, or may receive the image via a network connected to the system 100. Here, the type of network does not matter, and any network such as the Internet or LAN can be used.

The received image can be in any data format. The received image may be a two-dimensional image containing two-dimensional information (length x width) or a three-dimensional image containing three-dimensional information (length x width x depth).

The received image is passed to the creation means 120.

The creating means 120 is configured to create a pseudo three-dimensional image by processing the image. The generating means 120 may generate a pseudo-three-dimensional image by, for example, processing the image to add in the image a three-dimensional representation of elements separate from the image of interest (see, for example, FIGS. 1B-2D). can be created. Here, the three-dimensional representation of the elements is at least one of, for example, shading the elements, lighting the elements, giving the elements different sizes, or giving perspective to the elements. including one. The processing by the creating means 120 may be image processing known in the art.

The creating means 120 can create a pseudo-three-dimensional animation, for example, by rotating the three-dimensional representation of the elements added in the image around the target image. Such a pseudo three-dimensional moving image is preferable in that it enhances the pseudo three dimensional effect of the target image.

For example, the creation means 120 superimposes a part of the three-dimensional representation of the element on the image of the object, and hides a part of the image of the object by the three-dimensional representation of the element. A three-dimensional representation of an element can be added such that another part is superimposed under the image of interest and said other part of the three-dimensional representation of the element is obscured by the image of interest. This can further enhance the pseudo three-dimensional effect of the image of the object.

Here, the element can be any object, and can have any shape, size, color, etc.

The creation means 120 can add, for example, a three-dimensional representation of a plurality of horizontal scanning lines onto the target image. The three-dimensional representation of the plurality of horizontal scanlines can be scanlines drawn along the three-dimensional outline of the object. The three-dimensional contour shape of the object can be determined, for example, based on three-dimensional information contained in the image or derived from the image. The process of deriving three-dimensional information from images can be performed, for example, by techniques known in the art. The process of deriving 3D information from images can be performed using an AI model capable of estimating depth information from images.

The generating means 120 generates, for example, a plurality of images with different viewpoints from the images (see, for example, FIGS. 3A and 3B), and temporally successively combines the plurality of images with different viewpoints to create a pseudo three-dimensional image. You can create videos. A plurality of images from different viewpoints can be created, for example, based on three-dimensional information contained in the images or three-dimensional information derived from the images. For example, images with different viewpoints can be created by setting a virtual viewpoint and estimating how it looks from the virtual viewpoint based on three-dimensional information. A plurality of images from different viewpoints can be created using techniques known in the art, for example. For example, multiple images from different viewpoints can be performed using an AI model capable of creating image pairs with parallax.

The pseudo three-dimensional image created by creating means 120 is passed to display means 130 .

The display means 130 is configured to display a pseudo three-dimensional image. The display means 130 can be any display means as long as it can display an image. The display means 130 is, for example, a liquid crystal display, an LED display, or the like, but is not limited to these. In one embodiment, display means 130 may be a rotating display in which at least one member rotates to form a display surface. The rotating display can be, for example, rotating

display

20, 25, 27, etc., described above.

In one embodiment, the display means 130 can be configured so that the orientation of the display surface can be changed. The display means 130 may be able to change the orientation of the display surface using any mechanism.

The system 100 may further comprise detection means configured to detect the position of the user positioned in front of the display means 130 . The detection means can be any sensor. The detection means can be, for example, a camera. In this example, the system 100 can change the orientation of the display surface of the display means 130 so that the display surface of the display means 130 faces the position of the user detected by the detection means. Thereby, the user can always see the display surface of the display means 130 from the front.

FIG. 6B shows an example configuration of a system 100' for three-dimensionally displaying an image in another embodiment. The system 100' has the same configuration as the system 100, except that it includes means for synchronously reproducing sound for enhancing the pseudo three-dimensional effect of the pseudo three dimensional image. Here, the same reference numerals are given to the same configurations as those described above with reference to FIG. 6A, and detailed description thereof will be omitted.

The system 100 ′ comprises receiving means 110 , creating means 120 , displaying means 130 , synchronizing means 140 and reproducing means 150 .

The receiving means 110 is configured to receive an image. The received image is passed to the creating means 120 .

The creating means 120 is configured to create a pseudo three-dimensional image by processing the image. The pseudo three-dimensional image created by creating means 120 is passed to display means 130 and synchronization means 140 .

The display means 130 is configured to display a pseudo three-dimensional image.

The synchronizing means 140 is configured to synchronize the sound with the image. The image may be a pseudo three-dimensional image created by creating means 120 or an image received by receiving means 110 . Synchronizer 140 can synchronize the sound with the image using any technique known in the art of motion picture creation.

The sound can be any sound. The sound may be, for example, a sound that is directly related to the image, a sound that is somewhat related to the image, or a sound that is unrelated to the image. Preferably, the sound may be sound unrelated to the image, more preferably sound directly related to the image. Sound directly related to the image may be synchronized, for example, if the sound was already synchronized to the image received by the receiving means 110 (eg, if the image was a still image with sound or a moving image with sound). It could be the sound that was being played. Sounds that are somewhat related to images are, for example, sounds that are associated with images (e.g., bird wing sounds or chirping for bird images, car running sounds or horn sounds for car images, etc.). can be Even if the sound was synchronized to the image, the sound can be a sound other than the sound that was synchronized to the image.

Synchronization means 140 can synchronize sounds such that when the synchronized sounds are played, they appear to change based on motion in the image. For example, the synchronization means 140 may be arranged such that the sound is changing according to the relationship between the three-dimensional representation of the elements in the pseudo-three-dimensional image and the image of interest, as described above with reference to FIGS. 1E-1J. Sounds can be synchronized so that they can be heard. The change in sound can be, for example, at least one of loudness, pitch of sound, and timbre of sound. For example, the sound can be synchronized such that the sound is played when at least a portion of the image of interest touches the three-dimensional representation of the element. For example, such that at least a portion of the image of the object extends farther from the three-dimensional representation of the element, the sound becomes louder or quieter, or, for example, at least a portion of the image of the object moves farther from the three-dimensional representation of the element. Sounds can be synchronized such that they become higher or lower as they extend into the 3D representation, for example, approaching a different timbre as at least a portion of the image of interest extends further from the three-dimensional representation of the element. For example, the sound can be synchronized such that when at least a portion of the image of interest moves outside the three-dimensional representation of the element, the sound appears to change with the movement.

For example, the synchronizing means 140 can synchronize the sounds so that the sounds change according to the relationship between the boundary set in the pseudo three-dimensional image and the target image. This corresponds to the example above where the 3D representation of the elements in the pseudo 3D image is invisible. For example, the sound can be synchronized such that the sound is played in response to at least a portion of the image of interest crossing a boundary. For example, to make the sound louder or quieter as at least a portion of the image of interest extends farther from the boundary, or to make the sound louder or lower as at least a portion of the image of interest extends farther from the boundary, for example. In addition, for example, the sounds can be synchronized such that at least a portion of the image of interest extends farther from the boundary and approaches a different timbre. For example, the sound can be synchronized such that when at least a portion of the image of interest moves outside the boundary, the sound appears to change with the movement.

The boundary can have any shape. For example, it may be a shape that surrounds the target image (e.g., spherical, elliptical, cylindrical, prismatic, etc.), or a shape that does not surround the target image (e.g., planar, curved, hemispherical). shape, etc.). The boundary may change over time or may not change over time.

The sound synchronized with the image by the synchronization means 140 is passed to the reproduction means 150.

The reproduction means 150 is configured to reproduce sound synchronized with the image while the display means 130 is displaying the image. The reproduction means 150 can be any reproduction means as long as it can reproduce sound in time with the image being displayed. The reproduction means 150 is, for example, a speaker. For example, the speaker may be built in the display means 130 or may be externally attached to the display means 130 .

In the above example, it has been explained that sound is reproduced in accordance with the pseudo-three-dimensional image in order to emphasize the three-dimensional feeling by means of visual and auditory effects, but the present invention is not limited to this. For example, a configuration for emphasizing the three-dimensional feeling only by auditory effects is within the scope of the present invention. In this case, for example, the creating means 120 and the displaying means 130 may be omitted in the configuration shown in FIG. 6B.

The systems 100, 100' described above may be implemented in the user equipment 200, for example.

6C shows an example of the configuration of the user device 200. FIG.

The user device 200 can be any terminal device such as smart phones, tablet computers, smart glasses, smart watches, laptop computers, desktop computers, and the like.

The user device 200 includes a communication interface section 210 , an input section 220 , a display section 230 , a memory section 240 and a processor section 250 .

The communication interface unit 210 controls communication of the user device 200 with the outside. The processor unit 250 of the user device 200 can receive information from outside the user device 200 via the communication interface unit 210 and can transmit information to the outside of the user device 200 . For example, the processor portion 250 of the user device 200 can receive images via the communication interface portion 210 . It is possible to transmit the pseudo three-dimensional image to the outside of the user device 200 . Communication interface unit 210 may control communications in any manner.

For example, the receiving means 110 of the system 100 can be implemented by the communication interface section 210.

The input unit 220 allows the user to input information into the user device 200 . It does not matter in what manner the input unit 220 allows the user to input information into the user device 200 . For example, if the input unit 220 is a touch panel, the user may input information by touching the touch panel. Alternatively, if the input unit 220 is a mouse, the user may input information by operating the mouse. Alternatively, if the input unit 220 is a keyboard, the user may input information by pressing keys on the keyboard. Alternatively, if the input unit 220 is a microphone, the user may input information by voice.

The display unit 230 can be any display for displaying information.

For example, the display means 130 of the system 100 may be implemented by the display unit 230.

The memory unit 240 stores programs for executing processes in the user device 200 and data required for executing the programs. The memory unit 240 stores, for example, part or all of a program for three-dimensionally displaying an image (for example, a program for realizing processing shown in FIGS. 7A and 7B described later). The memory unit 240 may store, for example, part or all of a program for displaying an image on the display (for example, a program for realizing processing shown in FIG. 8, which will be described later). The memory unit 240 may store applications that implement arbitrary functions. Here, it does not matter how the program is stored in the memory unit 240 . For example, the program may be pre-installed in memory unit 240 . Alternatively, the program may be installed in memory unit 240 by being downloaded via network 500 . The program may be stored on a computer-readable tangible storage medium. Memory unit 240 may be implemented by any storage means.

The processor unit 250 controls the operation of the user device 200 as a whole. The processor unit 250 reads a program stored in the memory unit 240 and executes the program. This allows the user device 200 to function as a device that executes desired steps. The processor unit 250 may be implemented by a single processor or may be implemented by multiple processors.

For example, the creating means 120 of the system 100 may be implemented by the processor unit 250. For example, the synchronization means 140 of system 100 may be implemented by processor portion 250 .

The user device 200 can include, for example, a detector configured to detect the position of the user positioned in front of the display 230 . The detector can be any sensor. The detector can be, for example, a camera. The system 100 detection means may be implemented by a detection unit.

The user device 200 may include, for example, a reproduction unit (not shown) for reproducing sound. The reproduction unit can be any speaker for reproducing sound. For example, the reproducing means 150 of the system 100 may be implemented by a reproducing section.

Although each component of the user device 200 is provided in the user device 200 in the example shown in FIG. 6C, the present invention is not limited to this. Any of the components of user device 200 may be provided external to user device 200 . For example, the display unit 230 can be provided outside the user device 200 (that is, the display unit 230 is an external display). For example, when the input unit 220, the display unit 230, the memory unit 240, and the processor unit 250 are each configured with separate hardware components, each hardware component may be connected via an arbitrary network. . At this time, the type of network does not matter. Each hardware component may be connected via a LAN, wirelessly, or wired, for example. User device 200 is not limited to a particular hardware configuration. For example, it is within the scope of the present invention to configure the processor section 250 with analog circuits instead of digital circuits. The configuration of user device 200 is not limited to that described above as long as its functions can be realized.

For example, the components of the system 100 may be provided on the user device 200 side as described above, or distributed to both the user device 200 and the server device. If the components of system 100 are distributed in both user device 200 and server device, user device 200 comprises display means 130 (and playback means 150) and server device comprises receiving means 110 and creating means 120 (and synchronization means). 140).

3. Processing by System for Three-Dimensional Display of Images FIG. 7A shows an example of processing 700 by system 100 for three-dimensional display of images. In the example shown in FIG. 7A, the case where the system 100 is implemented by the user device 200 and the processing is executed by the processor unit 250 of the user device 200 will be described as an example. As mentioned above, processor unit 250 may implement creating means 120 .

At step S701, the processor unit 250 receives an image. The image contains the image of the object. The processor unit 250 can receive images received from outside the system 100 via the communication interface unit 210, for example.

In step S702, the processor unit 250 creates a pseudo three-dimensional image by processing the image received in step S701. The processor unit 250 processes the image to create a pseudo-three-dimensional image by adding in the image three-dimensional representations of elements separate from the image of interest (see, for example, FIGS. 1B-2D). can do. Here, the three-dimensional representation of the elements is at least one of, for example, shading the elements, lighting the elements, giving the elements different sizes, or giving perspective to the elements. including one. The processing by the processor unit 250 may be image processing known in the art.

The processor unit 250 can create a pseudo-three-dimensional animation, for example, by rotating the three-dimensional representation of the elements added in the image around the image of interest. Such a pseudo three-dimensional moving image is preferable in that it enhances the pseudo three dimensional effect of the target image.

Additionally or alternatively, the processor unit 250 may, for example, superimpose a portion of the three-dimensional representation of the element on the image of the object so that a portion of the image of the object is a three-dimensional representation of the element. and such that another part of the three-dimensional representation of the element is superimposed under the target image such that said other part of the three-dimensional representation of the element is hidden by the target image. A three-dimensional representation can be added. This can further enhance the pseudo three-dimensional effect of the image of the object.

Additionally or alternatively, the processor unit 250 can add, for example, a three-dimensional representation of multiple horizontal scan lines onto the image of interest. The three-dimensional representation of the plurality of horizontal scanlines can be scanlines drawn along the three-dimensional outline of the object. The processor unit 250 can determine the three-dimensional contour shape of the object based on the three-dimensional information contained in the image or derived from the image. The process of deriving three-dimensional information from images can be performed, for example, by techniques known in the art. The process of deriving 3D information from images can be performed using an AI model capable of estimating depth information from images.

Additionally or alternatively, the processor unit 250 may, for example, generate a plurality of images with different viewpoints from the image (see, for example, FIGS. 3A and 3B), and generate a plurality of images with different viewpoints over time. A pseudo three-dimensional moving image can be created by combining the images continuously. The processor unit 250 can create a plurality of images with different viewpoints based on, for example, three-dimensional information contained in the images or three-dimensional information derived from the images. A plurality of images from different viewpoints can be created using techniques known in the art, for example. For example, multiple images from different viewpoints can be performed using an AI model capable of creating image pairs with parallax.

In step S703, the processor unit 250 displays the pseudo three-dimensional image created in step S702 on the display unit 230.

The processor unit 250 can display the pseudo three-dimensional image created in step 702 as it is on the display unit 230, for example. Alternatively, the processor unit 250 may display the pseudo three-dimensional image on the display unit 230 by changing the orientation of the object in the pseudo three-dimensional image according to the orientation of the display surface of the display unit 230 . As a result, for example, an image corresponding to the orientation of the user with respect to the display unit 230 is displayed on the display unit 230 .

It has been described that the pseudo three-dimensional image is displayed on the display unit 230 by changing the orientation of the object in the pseudo three-dimensional image according to the orientation of the display surface of the display unit 230. It is not limited to the pseudo three-dimensional image created at 702 . For example, if the image received in step S701 is a three-dimensional image, the orientation of the object in the image received in step S701 can be changed and the image can be displayed on display unit 230 . Even in this case, the object in the image appears to be a three-dimensional object, so that the displayed image can result in a pseudo-three-dimensional image.

For example, the process 700 may be distributed to both the user device 200 and the server device. At this time, for example, steps S701 and S702 can be performed by the server device, and step S703 can be performed by the user device 200 .

FIG. 7B shows an example of processing 710 by system 100' for displaying an image three-dimensionally. In the example shown in FIG. 7B , the system 100 ′ is implemented by the user device 200 and the processing is executed by the processor unit 250 of the user device 200 as an example. As noted above, processor portion 250 may implement creating means 120 and synchronizing means 140 .

At step S711, the processor unit 250 receives the image. Step S711 is the same as step S701.

In step S702, the processor unit 250 creates a pseudo three-dimensional image by processing the image received in step S701. Step S712 is similar to step S702.

At step S703, the processor unit 250 synchronizes the sound with the pseudo three-dimensional image created at step S702. Processor unit 250 can synchronize sound with images using any technique known in the art of motion picture production.

The processor unit 250 can synchronize the sounds such that when the synchronized sounds are played, they appear to change based on motion in the image. The processor unit 250 can synchronize the sounds so that the sounds change according to the relationship between the boundary set in the pseudo three-dimensional image and the target image. The boundary may or may not be a three-dimensional representation of the sculpture in the pseudo-three-dimensional image.

In step S714, the processor unit 250 displays the pseudo three-dimensional image created in step S712 on the display unit 230. Step S714 is similar to step S703.

In step S715, the processor unit 250 reproduces the sound synchronized in step S713 from the reproduction unit while the pseudo three-dimensional image is being displayed in step S714.

In addition to the visual pseudo-three-dimensional effect of the pseudo-three-dimensional image, the processing 710 adds the auditory pseudo-three-dimensional effect of the sound reproduced in time with the motion in the pseudo-three-dimensional image, thereby enhancing the three-dimensional appearance of the image. feeling will be emphasized.

For example, the processing 710 may be distributed to both the user device 200 and the server device. At this time, for example, steps S711 to S713 can be performed by the server device, and steps S714 and S715 can be performed by the user device 200. FIG.

Although the processing 710 has been described as enhancing the three-dimensional feeling of the image by the auditory pseudo three-dimensional effect in addition to the visual pseudo three dimensional effect, only the auditory pseudo three dimensional effect You may make it emphasize the three-dimensional feeling of an image. In this case, step S712 is omitted, sound is synchronized with the image received in step S711 in step S713, and the image received in step S711 is displayed in step S714.

FIG. 8 shows an example of a process 800 for displaying an image on a display. Process 800 can provide a virtual reality image to a user without using a dedicated display device (eg, VR goggles, head-mounted display, etc.). Process 800 is performed by processor unit 250 of user device 200, for example.

In step S801, the detection unit of the user device 200 detects the position of the user's viewpoint with respect to the display. The detection unit can detect the position of the user's viewpoint by any detection means. For example, the detection unit can detect the position of the user's viewpoint based on the image captured by the camera. The position of the user's viewpoint may be, for example, the position of the user's eyes (more specifically, for example, the midpoint between the user's eyes).

At step S802, the processor unit 250 of the user device 200 receives an image to be displayed as a virtual reality image and processes the image to determine the portion of the image to be displayed on the display. Determining the portion of the image to be displayed on the display can be performed, for example, by steps S8021-S8023 below.

In step S8021, the processor unit 250 sets a virtual sphere. A virtual sphere is a virtual sphere whose center is the user's viewpoint and whose radius is the distance between the user's viewpoint and the display. For example, when the distance between the user's viewpoint and the display is small, as shown in FIG. 5B, the diameter of the phantom sphere becomes small, while the distance between the user's viewpoint and the display is small, as shown in FIG. 5C. If the distance of is large, the diameter of the phantom sphere will be large.

In step S8022, the processor unit 250 pastes the image to be displayed as a virtual reality image on the inner surface of the virtual sphere set in step S8021. The processor unit 250 can apply an image to the inner surface of the sphere by any processing known in the field of image processing. Here, the image is preferably represented by the equirectangular projection method. This is because an image represented by the equirectangular projection can be pasted on the inner surface of the sphere without distortion.

In step S8023, the processor unit 250 identifies the portion of the image pasted on the inner surface portion of the virtual sphere corresponding to the display surface of the display. The portion of the inner surface of the virtual sphere that corresponds to the display surface of the display is the portion that overlaps the display surface of the display when the virtual sphere is virtually arranged around the user's viewpoint. The processor unit 250 can, for example, derive the inner surface portion of the virtual sphere corresponding to the display surface of the display from the relative positional relationship between the user and the display. Then, the processor unit 250 can specify the portion of the image from the relationship between the derived portion and the image pasted on the virtual sphere.

When the portion of the image to be displayed on the display is thus determined, the process proceeds to step S803.

In step S803, the portion of the image determined in step S802 is displayed on the display surface of the display.

For example, by repeating steps S801 to S803 each time the user changes the position with respect to the display, an image according to the user's position can be displayed.

The image displayed by process 800 will maintain the perspective perceived by the user. That is, objects that are far away in the virtual reality image will still appear far away whether the user moves closer to the display or farther away from the display, and objects that are closer in the virtual reality image will appear farther away when the user moves away from the display. Even when the user moves away from the display, it still appears to be close. In this way, the user U can see the image with a sense of the real world through the display. This display may be a dedicated display device (e.g., VR goggles, head-mounted display, etc.), but may be a general stationary display, or the above-described

rotary display

20, 25, 27, etc. good. That is, the user can view a natural virtual reality image without wearing a dedicated display device.

In the example described above with reference to FIGS. 7A and 7B, the processing of each step shown in FIGS. 7A and 7B and part of the processing shown in FIG. Although it has been described that the present invention is realized by and, the present invention is not limited to this. At least one of the processing of each step shown in FIGS. 7A and 7B and part of the processing shown in FIG. 8 may be realized by a hardware configuration such as a control circuit.

In addition, in the examples described above with reference to FIGS. 7A, 7B, and 8, it has been described that the processing of each step is performed in a specific order, but the order of processing of each step is not limited to that described. . The steps can be processed in any order that is logically possible. Also, any other steps could be added and/or at least one of the steps shown could be omitted.

The present invention is not limited to the embodiments described above. It is understood that the invention is to be construed in scope only by the claims. It is understood that a person skilled in the art can implement an equivalent range from the description of specific preferred embodiments of the present invention based on the description of the present invention and common technical knowledge.

The present invention is useful in that it can provide a method and the like capable of creating a pseudo-three-dimensional image in order to three-dimensionally display an image.

100 System 110 Receiving Means 120 Creating Means 130 Displaying Means

Claims

A method for displaying an image three-dimensionally, comprising:
receiving an image containing an image of interest;
processing the image to create a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
and displaying the simulated three-dimensional image.
2. The pseudo-three-dimensional image of claim 1, wherein creating the pseudo-three-dimensional image comprises creating a pseudo-three-dimensional animation as the pseudo-three-dimensional image by rotating the three-dimensional representation of the element around the image of the object. The method described in .
a portion of the three-dimensional representation of the element is superimposed on the image of the object such that the portion of the image of the object is hidden by the three-dimensional representation of the element; is superimposed under the image of interest and the other portion of the three-dimensional representation of the element is obscured by the image of interest. 3. A method according to claim 1 or claim 2.
the element includes a plurality of horizontal scan lines;
4. Any of claims 1-3, wherein adding a three-dimensional representation of the element within the image comprises adding a three-dimensional representation of the plurality of horizontal scan lines onto the image of the object. The method according to item 1.
Creating the pseudo three-dimensional image includes generating a plurality of images with different viewpoints from the image, and temporally successively combining the plurality of images with different viewpoints to form the pseudo three-dimensional image. A method according to any one of claims 1 to 4, comprising creating a simulated 3D animation.
The pseudo three-dimensional image is a pseudo three-dimensional video,
The method includes:
synchronizing sound with the simulated three-dimensional image;
The method of any one of claims 1-5, further comprising: playing said synchronized sound while displaying said simulated 3D image.
The method of claim 6, wherein the sound changes based on movement in the image.
A method according to claim 6 or claim 7, wherein the sound changes in response to the image of the object exceeding a boundary defined around the image of the object.
9. The method of claim 8, wherein the sound changes in response to movement of the object outside the boundary.
10. A method according to claim 8 or claim 9, wherein the boundary is defined by a three-dimensional representation of the elements arranged around the image of the object.
The method according to any one of claims 7 to 10, wherein said change in sound includes change in at least one of loudness, pitch and timbre of said sound.
displaying the simulated three-dimensional image includes displaying the simulated three-dimensional image on a rotating display in which at least one member rotates about a first axis to form a planar display surface; A method according to any one of claims 1-11.
The rotary display is configured such that the orientation of the display surface can be changed,
The method includes:
detecting a user's position relative to the rotating display;
13. The method of claim 12, comprising: reorienting the display surface based on the detected position.
Displaying the pseudo three-dimensional image includes changing the orientation of the object in the pseudo three-dimensional image based on the orientation of the display surface and displaying the pseudo three-dimensional image on the display surface. 8. The method of claim 7, comprising:
Displaying the pseudo-three-dimensional image includes rotating at least one member about a first axis and about a second axis substantially perpendicular to the first axis to form a substantially spherical display surface. A method according to any preceding claim, comprising displaying the simulated three-dimensional image on a rotating display.
The method according to any one of claims 12 to 15, wherein the rotary display has a plurality of members that rotate to form a plurality of display surfaces.
A program for displaying an image three-dimensionally, said program being executed in a computer system comprising a processor and a display unit, said program comprising:
receiving an image containing an image of interest;
processing the image to create a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
A program causing the processor to perform processing including: displaying the pseudo three-dimensional image on the display unit.
A system for three-dimensionally displaying an image, comprising:
receiving means for receiving an image including an image of interest;
creating means for creating, by processing the image, a pseudo-three-dimensional image that produces a pseudo-three-dimensional effect by adding a three-dimensional representation of elements in the image that are separate from the image of interest;
and display means for displaying the pseudo three-dimensional image.
A method for displaying an image three-dimensionally, comprising:
receiving an image;
synchronizing sound with the image, wherein the sound changes in response to movement in the image; displaying the image;
and playing said synchronized sound while displaying said image.
A program for three-dimensionally displaying an image, said program being executed in a computer system comprising a processor, a display unit, and a sound output unit, said program comprising:
receiving an image;
synchronizing sound with the image, wherein the sound changes in response to movement in the image; displaying the image on the display;
and reproducing the synchronized sound from the sound output unit when the image is displayed.
A system for three-dimensionally displaying an image, comprising:
a receiving means for receiving an image;
synchronization means for synchronizing sound with said image, said sound varying in response to movement in said image; display means for displaying said image;
reproduction means for reproducing said synchronized sound when displaying said image.
A method of displaying an image on a display, comprising:
detecting the position of a user's viewpoint with respect to the display;
Determining the portion of the image to be displayed on the display by processing the image, comprising:
setting a virtual sphere centered at the user's viewpoint and having a radius equal to the distance between the user's viewpoint and the display;
pasting the image onto the inner surface of the virtual sphere;
identifying a portion of the image pasted on a portion of the inner surface of the phantom sphere corresponding to the display surface of the display;
including
and displaying the determined portion of the image on the display surface of the display.
The method according to claim 22, wherein said image is represented by an equirectangular projection.
The method according to claim 22 or 23, wherein said display is a stationary display.
25. The method of claim 24, wherein the display is a rotating display in which at least one member rotates to form a display surface.