WO2012132267A1 - Dispositif d'émission d'image stéréoscopique omnidirectionnelle - Google Patents

Dispositif d'émission d'image stéréoscopique omnidirectionnelle Download PDF

Info

Publication number
WO2012132267A1
WO2012132267A1 PCT/JP2012/001676 JP2012001676W WO2012132267A1 WO 2012132267 A1 WO2012132267 A1 WO 2012132267A1 JP 2012001676 W JP2012001676 W JP 2012001676W WO 2012132267 A1 WO2012132267 A1 WO 2012132267A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
stereo image
viewpoint
stereo
change
Prior art date
Application number
PCT/JP2012/001676
Other languages
English (en)
Japanese (ja)
Inventor
智典 中村
朋子 片山
敬一 田中
Original Assignee
パナソニック株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by パナソニック株式会社 filed Critical パナソニック株式会社
Publication of WO2012132267A1 publication Critical patent/WO2012132267A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps

Definitions

  • the present invention relates to a technique for converting a two-dimensional (2D) image into a stereo image (also referred to as a stereoscopic image or a three-dimensional (3D) image), and more particularly to a technique for converting a panoramic image into a stereo image.
  • a stereo image also referred to as a stereoscopic image or a three-dimensional (3D) image
  • One of the services provided through the Internet is street view.
  • “Street view” is a function that, when a point on a map displayed on a web browser is designated, can display a 360 degree view of a landscape photographed from that point (for example, Patent Documents). 1). Usually, a plurality of points that can be designated are set along the road shown on the map.
  • the server manages panoramic images with each point as a viewpoint.
  • a panoramic image is obtained by projecting a landscape in all directions seen from a viewpoint onto a three-dimensional model such as a cylindrical surface or a spherical surface centered on the viewpoint, and further projecting the projected image onto a plane.
  • a panoramic image with that point as a viewpoint is downloaded from the server to the client.
  • the client further converts the region of the viewing direction designated by the user in the panoramic image into an image on the 2D screen and displays it on the browser.
  • Information on the depth of the subject such as a building wall or road surface shown in the panoramic image, is also downloaded from the server to the client.
  • the client uses the depth information to display a graphic element such as a line indicating a route or an arrow indicating the direction of an adjacent point on the landscape image in a three-dimensional manner.
  • the most realistic method is to cause the client to download an existing 2D panoramic image and depth information of the subject from the server, and generate a 3D panoramic image using them.
  • street view using 3D panoramic images can be realized from existing data.
  • the amount of data to be processed at the time of generating a one-frame stereo image increases as compared to a street view using a 2D panoramic image.
  • a street view using a 3D panoramic image requires depth information in addition to a 2D panoramic image.
  • a pair of 2D panoramic images for right eye and left eye must be generated from the 2D panoramic image and depth information.
  • Household appliances such as 3D televisions have a relatively small memory capacity and a relatively low processing capacity of the CPU.
  • the increase in the data amount as described above increases the processing time.
  • the waiting time required for screen switching increases. If the screen before the change continues to be displayed for a long time from when the change of the viewpoint is requested, the user feels uncomfortable because the response to the request cannot be obtained. For the purpose of reducing the discomfort, even if the 2D image is displayed during the waiting time until the screen is switched, the switching between the 3D image and the 2D image gives the user another discomfort.
  • An object of the present invention is to solve the above-described problems, and in particular, to provide an omnidirectional stereo image output apparatus capable of responding in a short time to a request for changing a viewpoint from a user.
  • the image output device is a device that converts a region located in a predetermined viewing direction with respect to the viewpoint of a panoramic image into a stereo image and outputs the stereo image.
  • This apparatus includes an acquisition unit, a parallax panoramic image generation unit, a rendering unit, an output unit, and a control unit.
  • the acquisition unit acquires a panoramic image and depth information representing the depth of an object shown in the panoramic image.
  • the parallax panoramic image generation unit generates a parallax panoramic image by shifting the coordinates of each pixel of the panoramic image according to the depth represented by the depth information.
  • the rendering unit projects the panorama image and the parallax panorama image onto the stereoscopic model, and converts a region located in a predetermined viewing direction with respect to the viewpoint from the projected image into a stereo image.
  • the output unit outputs the stereo image converted by the rendering unit.
  • the control unit causes the output unit to output the alternative stereo image until the panoramic image seen from the changed viewpoint is acquired and converted into a stereo image.
  • the alternative stereo image is a stereo image that approximates at least one of a stereo image that can be seen from the viewpoint before the change and a stereo image that can be seen from the viewpoint after the change.
  • the alternative stereo image generation process is faster than the process of acquiring a panoramic image seen from the changed viewpoint and converting it to a stereo image.
  • the alternative stereo image may be one in which the control unit causes the rendering unit to process the first stereo image converted from the panoramic image seen from the viewpoint before the change.
  • the alternative stereo image approximates the stereo image that can be seen from the viewpoint before the change.
  • the generation of the alternative stereo image does not require acquisition of the panorama image and depth information, generation of the parallax panorama image, and conversion from the panorama image and the parallax panorama image to the stereo image. Therefore, the alternative stereo image generation process is faster than the process of acquiring a panoramic image seen from the changed viewpoint and converting it to a stereo image.
  • the alternative stereo image may be a low-resolution panoramic image converted to a stereo image.
  • the low-resolution panoramic image is a panoramic image that can be seen from the viewpoint after the change, and has a lower resolution than the panoramic image that can be seen from the viewpoint before the change.
  • the alternative stereo image approximates a stereo image that can be seen from the changed viewpoint.
  • the acquisition unit further acquires a low-resolution panoramic image and low-resolution depth information representing the depth of an object shown in the low-resolution panoramic image.
  • the control unit first causes the parallax panorama image generation unit to generate a low resolution parallax panorama image from the low resolution panorama image and the low resolution depth information.
  • the control unit further causes the rendering unit to project the low-resolution panoramic image and the low-resolution parallax panoramic image onto the three-dimensional model, and newly adds a region located in the viewing direction with respect to the changed viewpoint from the projected image.
  • the image is converted into a stereo image, and a substitute stereo image is generated using the new stereo image.
  • the low-resolution panoramic image has a smaller data amount than the original resolution panoramic image. Therefore, the acquisition of the low resolution panorama image and the low resolution depth information, the generation of the low resolution parallax panorama image, and the conversion from the low resolution panorama image and the low resolution parallax panorama image to the new stereo image are all performed at the original resolution. Faster than panoramic image processing. As a result, the generation process of the alternative stereo image is faster than the process of acquiring a panoramic image with an original resolution that can be seen from the changed viewpoint and converting it to a stereo image.
  • the image output device outputs an alternative stereo image when a change in the viewpoint of the panoramic image is requested. Since the substitute stereo image approximates at least one of the stereo image seen from the viewpoint before the change and the stereo image seen from the viewpoint after the change, the discomfort given to the user by the insertion of the substitute stereo image can be suppressed.
  • the alternative stereo image generation process is faster than the process of acquiring a panoramic image that can be seen from the changed viewpoint and converting it to a stereo image. As a result, this apparatus can respond in a short time to a request for changing the viewpoint from the user.
  • FIG. 1 is a schematic diagram showing a part of the map displayed on the screen 131 of the display device 103 shown in FIG. 1 by street view.
  • (B) is a schematic diagram showing a stereo image displayed on the screen 131 of the display device 103 based on the street view information.
  • FIG. 2 is a block diagram showing a hardware configuration of the playback device 102 shown in FIG. 1.
  • FIG. 3 is a block diagram showing functional units of a playback device 102 used for image processing related to street view. It is a table
  • (A) is a schematic diagram which shows an example of the panoramic image decoded from the compression panoramic image.
  • (B) is a schematic diagram showing a depth map representing the depth of the panoramic image shown in (a) of FIG. 6.
  • A) is a schematic diagram which shows an example of the image which the texture for left eyes represents.
  • (B) is a schematic diagram showing a depth map of the image shown in (a).
  • (C) is a schematic diagram showing an image represented by a texture for the right eye generated by DIBR from the texture for the left eye shown in (a) and the depth map shown in (b).
  • (A) is a schematic diagram showing the process which the rendering part 374 shown by FIG. 3 projects a texture on a spherical model.
  • B) is a schematic diagram showing the inside of the spherical model 820 seen from the direction of the North Pole.
  • FIG. 8C is a schematic diagram illustrating an image of an area extracted by the rendering unit 374.
  • FIG. 3 is a first half of a flowchart of image processing relating to street view by the playback apparatus shown in FIG. 1; 6 is a second half of a flowchart of image processing relating to street view by the playback device shown in FIG. 1;
  • FIG. 4 is a schematic diagram illustrating a stereo image scaling process performed by a rendering unit 374 illustrated in FIG. 1.
  • (A) is a top view schematically showing a state when a subject OBJ is photographed by a camera installed at the first viewpoint VP1.
  • (B) is a top view schematically showing a state when the subject OBJ is photographed by the camera installed at the second viewpoint VP2.
  • FIG. 1 It is a flowchart of the production
  • A is a schematic diagram showing the image of the frame for right eyes seen from the viewpoint before a change.
  • B is the schematic diagram showing the image of the frame for right eyes of the alternative stereo image by Embodiment 2 of this invention.
  • It is a flowchart of the production
  • (A)-(d) is a schematic diagram showing a series of alternative stereo images representing a process in which a stereo image seen from a viewpoint before change is naturally changed to a stereo image seen from a viewpoint after change by morphing. is there. It is a flowchart of the production
  • FIG. 1 is a schematic diagram showing a home theater system according to Embodiment 1 of the present invention.
  • This home theater system employs a time separation system (also referred to as a frame sequential system) as a stereo image display system.
  • the stereo image is a pair of a left-eye frame that represents a 3D space scene that appears in the left eye and a right-eye frame that represents the scene that appears in the right eye.
  • the time separation method is a method of alternately displaying a left-eye frame and a right-eye frame on the screen.
  • the home theater system includes a recording medium 101, a playback device 102, a display device 103, shutter glasses 104, and a remote controller 105.
  • the recording medium 101 is a read-only Blu-ray Disc (registered trademark) (BD: Blu-ray Disc), that is, a BD-ROM disc.
  • the recording medium 101 may be another portable recording medium, for example, a semiconductor memory device such as an optical disk, a removable hard disk drive (HDD), an SD memory card, or the like according to another method such as a DVD.
  • the recording medium, that is, the BD-ROM disc 101 stores stereo image contents.
  • This content includes a left-eye video stream and a right-eye video stream.
  • Each video stream represents a frame sequence for a left eye and a right eye of a stereo image.
  • the content may further include a depth map stream.
  • the depth map stream represents a depth map of each frame of the stereo image.
  • the depth map represents the depth of each part of the stereo image for each pixel.
  • a pair of left-eye video stream and right-eye video stream, or a pair of left-eye or right-eye video stream and depth map stream is played back as a stereo image by the playback device. Used to do.
  • the playback device 102 is equipped with a BD-ROM drive 121.
  • the BD-ROM drive 121 is an optical disk drive conforming to the BD-ROM system.
  • the playback apparatus 102 reads content from the BD-ROM disc 101 using the BD-ROM drive 121.
  • the playback device 102 further decodes the content into image data / audio data.
  • the playback device 102 uses a combination of a left-eye video stream and a right-eye video stream, or a combination of either a left-eye video stream or a right-eye video stream and a depth map stream. Decodes a pair with a frame.
  • the playback device 102 is connected to the display device 103 by an HDMI (High-Definition Multimedia Interface) cable 122.
  • the playback device 102 converts the image data, audio data, and control information into an HDMI serial signal, and transmits the converted signal to the display device 103 through the HDMI cable 122.
  • the image data both the left-eye frame and the right-eye frame are multiplexed in a time division manner.
  • the control information includes a horizontal synchronization signal, a vertical synchronization signal, and auxiliary data.
  • the display device 103 is a liquid crystal display.
  • the display device 103 may be a flat panel display or projector of another type such as a plasma display and an organic EL display.
  • the display device 103 displays an image on the screen 131 according to the image data received from the playback device 102 through the HDMI cable 122, and generates sound from a built-in speaker according to the sound data. Accordingly, the left eye frame and the right eye frame are alternately displayed on the screen 131.
  • the display device 103 includes a left / right signal transmission unit 132.
  • the left / right signal transmitting unit 132 transmits the left / right signal LR to the shutter glasses 104 by infrared rays or wirelessly.
  • the left / right signal LR indicates whether the image currently displayed on the screen 131 is for the left eye or for the right eye.
  • the display device 103 first detects frame switching by identifying a left-eye frame and a right-eye frame from a control signal such as a synchronization signal associated with image data and auxiliary data. Next, the display device 103 causes the left / right signal transmission unit 132 to change the left / right signal LR in synchronization with the detected frame switching.
  • the shutter glasses 104 include two liquid crystal display panels 141L and 141R and a left / right signal receiving unit 142.
  • the liquid crystal display panels 141L and 141R constitute left and right lens portions.
  • Each of the liquid crystal display panels 141L and 141R is normally white and transmits light as a whole when it receives no signal from the left and right signal receiving unit 142.
  • the left / right signal receiving unit 142 receives the left / right signal LR and sends signals to the left and right liquid crystal display panels 141L and 141R according to the change.
  • Each of the liquid crystal display panels 141L and 141R transmits or blocks light uniformly according to the signal.
  • the left / right signal LR indicates the display of the left-eye frame
  • the left-eye liquid crystal display panel 141L transmits light
  • the right-eye liquid crystal display panel 141R blocks light.
  • the left / right signal LR indicates the display of the right eye frame.
  • the two liquid crystal display panels 141L and 141R alternately transmit light in synchronization with the change. Therefore, when the viewer views the screen 131 with the shutter glasses 104, the left-eye frame appears only in the viewer's left eye, and the right-eye frame appears only in the right eye. At that time, the viewer perceives the difference between the images seen by each eye as binocular parallax with respect to the same three-dimensional object, so that the images appear three-dimensional.
  • the remote control 105 includes an operation unit and a transmission unit.
  • the operation unit includes a plurality of buttons. Each button is associated with each function of the playback device 102 or the display device 103, such as turning on / off the power or starting or stopping playback of the BD-ROM disc 101.
  • the operation unit detects pressing of each button by the user, and transmits the identification information of the button to the transmission unit by a signal.
  • the transmission unit converts the signal into an infrared or wireless signal IR and sends the signal IR to the playback device 102.
  • the playback device 102 receives the signal IR and identifies the function associated with the button indicated by the signal IR.
  • the playback device 102 further realizes the function when the function is its own, and instructs the display device 103 to realize the function via the HDMI cable 122 when the function is that of the display device 103. .
  • the server 106 is connected to the playback device 102 through a network 107 such as the Internet.
  • the user can request the playback device 102 to receive a street view service from the server 106 by operating the remote controller 105.
  • a street view service first, when a user designates a specific area with the remote controller 105, data regarding the map of the area is downloaded from the server 106 to the playback device 102.
  • the playback device 102 displays the map on the screen 131 of the display device 103 according to the data.
  • FIG. 2 is a schematic diagram showing a part of the map displayed on the screen 131 of the display device 103.
  • a cursor 201 and a specific mark 202 called “avatar” are displayed in an overlapping manner on the map.
  • the user operates the cursor 201 through the remote controller 105 to specify the position and direction of the avatar 202.
  • information indicating the position and direction of the avatar 202 is sent from the playback device 102 to the server 106.
  • street view information is downloaded from the server 106 to the playback device 102.
  • the street view information includes a panoramic image whose viewpoint is the point indicated by the position of the avatar 202, and depth information such as a wall of a building or a road surface reflected in the panoramic image.
  • the playback device 102 converts the panoramic image into a specific stereo image based on the street view information.
  • the stereo image represents a landscape that can be seen when a line of sight is directed toward the avatar 202 from the point indicated by the position of the avatar 202.
  • the playback device 102 further causes the display device 103 to display the stereo image.
  • FIG. 2B is a schematic diagram showing a stereo image displayed on the screen 131 of the display device 103 based on the street view information.
  • the playback device 102 changes the stereo image to a landscape in that direction.
  • the playback device 102 further displays a graphic element such as a line 203 or an arrow 204 on a stereo image representing a landscape.
  • the playback device 102 uses the depth information included in the street view information to display the graphics elements 203 and 204 three-dimensionally.
  • a line 203 indicates the direction of an adjacent point where a panoramic image can be displayed.
  • An arrow 204 is a GUI component for instructing to change the stereo image displayed on the screen 131 to another stereo image.
  • the playback device 102 changes the stereo image on the screen 131 to the display device 103 to a stereo image with the next point existing in the direction indicated by the arrow 204 as a viewpoint.
  • the playback device 102 When a stereo image with one point of view is changed to a stereo image with another point of view, it takes several tens of seconds to acquire the panorama image and depth information, and it is necessary to decode the acquired data. Several tens of seconds are required, and it takes several seconds to convert a panoramic image and depth information into a stereo image. Therefore, during these processes, the playback device 102 generates a substitute stereo image and displays it on the display device 103. Details of the alternative stereo image will be described later.
  • FIG. 3 is a block diagram illustrating a hardware configuration of the playback device 102.
  • the playback device 102 includes an acquisition unit 310, an HDD 320, a user operation (UO) module 330, an internal bus 340, a CPU 350, a memory unit 360, a drawing unit 370, and an output unit 380.
  • UO user operation
  • the acquisition unit 310 is a collection of elements that acquire data from the outside, and includes a tuner 301, a network interface 302, an optical disc drive 303, and a card reader 304.
  • the tuner 301 is connected to the antenna 390 and receives content transmitted by a terrestrial digital broadcast wave or a satellite digital broadcast wave.
  • a network interface 302 is connected to the network 107 and exchanges data with the server 106 and the like.
  • the optical disk drive 303 reads data from an optical disk such as the BD-ROM disk 101.
  • a card reader 304 reads data from a memory card 108 such as an SD card.
  • the HDD 320 is a hard disk drive built in the playback device 102, and receives and stores various data, particularly image data, from the acquisition unit 310 via the internal bus 340.
  • the UO module 330 detects a signal IR by an infrared ray from the remote controller 105 or a press of a button or the like provided on the front panel of the playback device 102.
  • the UO module 330 further decodes an instruction from the user indicated by the detected signal or the like, and outputs a signal indicating the instruction to the internal bus 340.
  • the internal bus 340 is a bus wiring group that transmits data between the elements 310 to 380 in the playback device 102.
  • the CPU 350 controls the other elements 310 to 340 and 360 to 380 in the playback device 102 by executing a program stored in the memory unit 360.
  • the memory unit 360 is the entire semiconductor memory device built in the playback device 102, and includes a ROM and a RAM.
  • the memory unit 360 stores a program group to be executed by the CPU 350 and provides a working memory space for the CPU 350 and the drawing unit 370, such as a frame buffer.
  • the drawing unit 370 is hardware dedicated to image processing, and includes a decoder 371, a depth generation unit 372, a parallax panorama image generation unit 373, and a rendering unit 374. These elements 371 to 374 are integrated on one chip.
  • the drawing unit 370 uses the elements 371 to 374 to process the entire image data acquired by the acquisition unit 310. In particular, in the street view, the drawing unit 370 converts the panoramic image into a stereo image frame sequence. Details of the functions of the elements 371 to 374 in the street view will be described later.
  • the output unit 380 includes an HDMI 1.4 transmitter and an HDMI output connector, and is connected to the display device 103 by an HDMI cable 122.
  • the output unit 380 multiplexes the frame sequence converted by the drawing unit 370 into an HDMI serial signal and outputs the multiplexed signal to the display device 103 through the HDMI cable 122.
  • FIG. 4 is a block diagram showing functional units of the playback device 102 used for image processing related to street view.
  • a memory 410 a left-eye texture memory 411, a right-eye texture memory 412, a left-eye frame buffer 413, and a right-eye frame buffer 414 are included.
  • the file system 401 and the control unit 402 are functional units realized by the CPU 350 executing a predetermined program.
  • the depth map memory 410, the left-eye texture memory 411, the right-eye texture memory 412, the left-eye frame buffer 413, and the right-eye frame buffer 414 are secured in the memory unit 360 shown in FIG. It is a memory area.
  • the file system 401 controls the elements 301 to 304 in the acquisition unit 310 and the HDD 320 to manage the data acquired by the acquisition unit 310 and the data stored in the HDD 320.
  • the file system 401 conforms to UDF (Universal Disc Format).
  • the file system 401 may be compliant with ISO9660.
  • the file system 401 represents data in a directory / file format. That is, these data are acquired by the acquisition unit 310 in directory units or file units, and are read and written by the HDD 320.
  • the file system 401 also controls the establishment / disconnection of the connection between the network interface 302 and the network 107, the loading / unloading of the optical disk 101 in the optical disk drive 303, the loading / unloading of the memory card 108 in the card reader 304, and the like. Notify
  • the control unit 402 controls the drawing unit 370 according to the user operation indicated by the signal from the UO module 330 and the notification from the file system 401.
  • the control unit 402 causes the file system 401 to control the network interface 302 and downloads the street view information from the server 106 to the drawing unit 370 as follows.
  • the signal from the UO module 330 represents that point.
  • the control unit 402 searches for the ID assigned to the point and passes it to the server 106 through the network interface 302.
  • the server 106 searches for street view information including the same ID as the passed ID and provides it to the playback device 102.
  • the control unit 402 further causes the drawing unit 370 to convert the street view information into an appropriate stereo image, and causes the output unit 380 to output the stereo image. Details of the street view information will be described later.
  • the file system 401 When the file system 401 receives the street view information from the acquisition unit 310, the file system 401 separates the compressed panorama image and the compressed panorama depth from the street view information, passes the compressed panorama image to the decoder 371, and converts the compressed panorama depth. It passes to the depth generation unit 372.
  • the “compressed panorama image” is image data compressed in the JPEG format, and represents a panoramic image with a specific point as a viewpoint.
  • “Compressed panorama depth” is image data compressed in PNG format, and represents depth information of the compressed panorama image.
  • the depth information is a two-dimensional array that represents the depth of an object such as a building wall or a road surface that appears in a panoramic image, that is, the distance from the viewpoint to the subject in pixel units.
  • Decoder 371 decodes the panorama image from the compressed panorama image.
  • the panoramic image is a two-dimensional array of pixel data in RGB format or YUV format, that is, a texture, and is developed in the texture memory 411 for the left eye. Since this panoramic image is used as a panoramic image that appears in the left eye of the person positioned at the viewpoint, it is referred to as a “left eye texture”.
  • the depth generation unit 372 decodes the depth information from the compressed panorama depth, generates a depth map based on the depth information, and stores the depth map in the depth map memory 410.
  • the depth map is a two-dimensional array in which all the pixels of the left eye texture are associated with the depth of the image portion represented by the pixels.
  • the luminance of each pixel of the depth map is expressed in RGB format or YUV format, and indicates the depth of the image portion represented by the pixel of the left-eye texture corresponding to the pixel.
  • the depth generation unit 372 also determines an appropriate value using interpolation or the like for the depth of the portion of the image that is not represented in the depth information.
  • the parallax panorama image generation unit 373 reads the left-eye texture from the left-eye texture memory 411 and reads the depth map from the depth map memory 410.
  • the parallax panorama image generation unit 373 further applies DIBR (Depth Image based Renderring) to the read data to generate a right eye texture, and stores the right eye texture in the right eye texture memory 412.
  • DIBR Depth Image based Renderring
  • “Right-eye texture” is a texture that represents a panoramic image that appears in the right eye of the person whose left-eye texture represents a panoramic image that appears in the left eye of the person positioned at the viewpoint.
  • DIBR refers to a process of generating a texture representing a 2D image seen in the other person's eye from a texture representing the 2D image seen in one eye of the person and a depth map for the texture.
  • the position of the subject in the horizontal direction differs between the image shown in one eye and the image shown in the other eye due to the parallax between the eyes.
  • a displacement amount of an image caused by parallax that is, a parallax value is calculated for each pixel from the depth represented by the depth map, and the coordinates of each pixel of the texture are shifted in the horizontal direction by the corresponding parallax value.
  • a new texture is generated from the texture.
  • the image represented by the new texture is referred to as a parallax image with respect to the image represented by the original texture. Therefore, the right-eye texture represents a parallax panoramic image with respect to the panoramic image represented by the left-eye texture.
  • the rendering unit 374 processes the left-eye texture and the right-eye texture using three-dimensional computer graphics such as OpenGL® ES. Accordingly, a region in a predetermined viewing direction is converted into a stereo image among each of the panoramic image represented by the left-eye texture and the parallax panoramic image represented by the right-eye texture. Specifically, the rendering unit 374 first reads the left-eye texture from the left-eye texture memory 411 and reads the right-eye texture from the right-eye texture memory 412. Next, the rendering unit 374 projects each texture onto a cylindrical surface model, and further projects the cylindrical surface model onto a spherical model.
  • three-dimensional computer graphics such as OpenGL® ES. Accordingly, a region in a predetermined viewing direction is converted into a stereo image among each of the panoramic image represented by the left-eye texture and the parallax panoramic image represented by the right-eye texture. Specifically, the rendering unit 374 first reads the left-eye texture from the left-eye texture memory 411 and reads the right-eye texture from
  • the rendering unit 374 installs the virtual camera at the center of the spherical model and directs the virtual camera in a predetermined viewing direction.
  • the viewing direction is designated by the control unit 402.
  • the control unit 402 decodes the direction of the avatar 202 designated by the user, and determines the viewing direction from the direction.
  • the rendering unit 374 further calculates an image when the image projected on the spherical model is captured by the virtual camera. As a result, a left-eye frame is obtained from the left-eye texture, and a right-eye frame is obtained from the right-eye texture.
  • Each frame is a two-dimensional array associated with a pixel group on the screen 131 of the display device 103, and each element of the array represents the luminance of the corresponding pixel.
  • the rendering unit 374 writes the left-eye frame in the left-eye frame buffer 413 and the right-eye frame in the right-eye frame buffer 414.
  • the output unit 380 reads the left eye frame from the left eye frame buffer 413 and reads the right eye frame from the right eye frame buffer 414.
  • the output unit 380 further multiplexes the read pair of frames into an HDMI serial signal and outputs the multiplexed signal.
  • the control unit 402 monitors the signal from the UO module 330 To do.
  • the control unit 402 controls the rendering unit 374 to change the direction of the virtual camera 830 according to the changed viewing direction.
  • another left-eye frame is extracted from the left-eye texture
  • another right-eye frame is extracted from the right-eye texture.
  • the stereo image displayed on the screen 131 changes to that of the changed viewing direction.
  • the control unit 402 searches for the ID assigned to the point where the changed viewpoint is located and passes it to the server 106 through the network interface 302.
  • the server 106 searches for new street view information including the same ID as the passed ID and provides it to the playback device 102. While the drawing unit 370 generates a new stereo image from the new street view information, the control unit 402 causes the rendering unit 374 to generate a substitute stereo image and causes the output unit 380 to output the substitute stereo image. Details of the alternative stereo image will be described later.
  • FIG. 5 is a table showing the data structure of street view information. Street view information is set for each predetermined point on the map. Referring to FIG. 5, the street view information includes ID 501, size 502, compressed panorama image 503, latitude 504, longitude 505, altitude 506, direction 507, slope 508, and compressed panorama depth 509.
  • ID 501 is an 8-digit hexadecimal number and represents an identifier uniquely assigned to a point on the map where the street view information is set.
  • the item “compressed panoramic image” 503 is image data compressed in the JPEG format, and represents a panoramic image with a point assigned with ID 501 as a viewpoint. The panoramic image is expressed as a two-dimensional array by Mercator projection.
  • the item “size” 502 represents the vertical and horizontal sizes of the panoramic image by the number of pixels.
  • the items “latitude” 504, “longitude” 505, and “elevation” 506 represent the latitude, longitude, and altitude of the point to which the ID 501 is assigned, respectively.
  • the items “direction” 507 and “tilt” 508 are two angles representing the tilt of the omnidirectional camera when a panoramic image is captured by the omnidirectional camera.
  • An item “compressed panorama depth” 509 is image data compressed in the PNG format, and represents depth information of the compressed panorama image 503. Depth information is likely to generate noise when compressed by a lossy compression method. Accordingly, the depth information is compressed in a lossless compression format such as the PNG format.
  • the items 501 to 509 of the street view information are described as a single table. However, it does not mean that the street view information is downloaded from the server 106 to the playback device 102 at once. Each item of the street view information may be divided and downloaded. In particular, since the compressed panorama image 503 and the compressed panorama depth 509 have a large data amount, they may be downloaded separately from other items.
  • FIG. 6A is a schematic diagram illustrating an example of a panoramic image decoded from a compressed panoramic image.
  • this panoramic image is a texture that two-dimensionally expresses an image that appears in all directions from a single viewpoint using the Mercator projection.
  • the region closer to both ends in the vertical direction appears to be distorted than the actual scene. If this panoramic image is projected onto a cylindrical surface and further projected onto a spherical surface, the image seen from the center of the sphere matches the landscape actually seen from the original viewpoint.
  • FIG. 6B is a schematic diagram showing a depth map representing the depth of the panoramic image shown in FIG.
  • the luminance of each pixel indicates the depth of the portion of the image represented by that pixel.
  • the higher the luminance of a pixel the closer the portion of the image represented by that pixel is to the viewpoint.
  • the luminance is the lowest value (black in FIG. 6B)
  • the depth is infinite
  • the luminance is the highest value (white in FIG. 6B)
  • the depth matches the viewpoint. To do.
  • FIG. 7A is a schematic diagram illustrating an example of an image represented by the texture for the left eye.
  • a sphere 701 and a pillar 702 are shown in the image.
  • FIG. 7B is a schematic diagram showing a depth map of the image shown in FIG.
  • the sphere 701 has higher luminance than the pillar 702. This means that the sphere 701 is positioned in front of the pillar 702.
  • the parallax panoramic image generation unit 373 calculates a parallax value for each pixel constituting the image from the sphere 701, the pillar 702, and the luminance distribution in the vicinity thereof.
  • the parallax panorama image generation unit 373 further shifts the coordinates of each pixel of the left-eye texture in the horizontal direction according to the calculated parallax value to generate a right-eye texture.
  • FIG. 7C shows a right-eye texture generated by DIBR from the left-eye texture shown in FIG. 7A and the depth map shown in FIG. 7B. It is a schematic diagram which shows an image. Referring to FIG. 7C, the coordinates of the sphere 703 are shifted to the left by the first parallax value HL from the coordinates of the sphere 701 shown in FIG. 7 is shifted to the right by the second parallax value HR from the coordinates of the column 702 shown in FIG. When the image shown in (a) of FIG.
  • the panorama image decoded from the compressed panorama image by the decoder 371 is used as the left eye texture.
  • the panoramic image may be used as the right eye texture.
  • the parallax panorama image generation unit 373 generates a left-eye texture from the right-eye texture and the depth map by DIBR.
  • FIG. 8A is a schematic diagram illustrating processing in which the rendering unit 374 projects a texture onto a spherical model.
  • the rendering unit 374 first projects the left eye texture and the right eye texture onto the cylindrical surface model 810.
  • a straight line representing a pixel row having a constant horizontal coordinate in each texture is projected onto an axial straight line 801 in the cylindrical surface model 810
  • a straight line representing a pixel row having a constant vertical coordinate in each texture is a cylindrical surface model.
  • the image is projected onto a circumferential curve 802.
  • the rendering unit 374 projects the cylindrical surface model 810 onto the spherical model 820.
  • the axial straight line 801 in the cylindrical surface model 810 is projected onto the meridian 803 of the spherical model 820, and the circumferential curve 802 in the cylindrical surface model 810 is projected onto the latitude line 804 of the spherical model 820.
  • FIG. 8B is a schematic diagram showing the inside of the spherical model 820 seen from the direction of the North Pole.
  • a virtual camera 830 is installed at the center of the spherical model 820.
  • the rendering unit 374 first directs the virtual camera 830 in the direction of the angle ⁇ with respect to the reference direction C where the longitude is 0 degree. This angle ⁇ is determined by the control unit 402 as an angle representing the viewing direction from the direction of the avatar 202 specified by the user.
  • the rendering unit 374 extracts a region included in the angle of view ⁇ of the virtual camera 830 from the panoramic image projected on the spherical model 820.
  • FIG. 8C is a schematic diagram illustrating an image of an area extracted by the rendering unit 374.
  • the image coincides with the scenery seen by the person who is located at the center of the spherical model 820 and looks at the viewing direction ⁇ .
  • the vertical end of the image shown in FIG. 8C has no distortion.
  • FIGS. 9 and 10 are flowcharts of image processing related to street view by the playback device. This process is started when the user operates the remote controller 105 to instruct the playback device 102 to start the street view, and the control unit 402 detects the instruction through a signal from the UO module 330.
  • step S ⁇ b> 901 the control unit 402 decodes the point indicated by the position of the avatar 202 designated by the user from the signal from the UO module 330, searches for the ID assigned to the point, and transmits the ID through the network interface 302.
  • the ID is passed to the server 106.
  • the server 106 provides the playback device 102 with street view information including the same ID as the passed ID.
  • the file system 401 downloads the street view information from the server 106 through the network interface 302. Thereafter, the process proceeds to step S902.
  • step S902 the file system 401 separates the compressed panoramic image from the street view information and passes it to the decoder 371.
  • the decoder 371 decodes the left-eye texture from the compressed panoramic image and writes it to the left-eye texture memory 411. Thereafter, the process proceeds to step S903.
  • step S903 the file system 401 separates the compressed panorama depth from the street view information and passes it to the depth generation unit 372.
  • the depth generation unit 372 generates a depth map from the compressed panorama depth and writes it to the depth map memory 410. Thereafter, the process proceeds to step S904.
  • step S904 the parallax panoramic image generation unit 373 first reads the left-eye texture from the left-eye texture memory 411 and reads the depth map from the depth map memory 410. Next, the parallax panoramic image generation unit 373 generates a right eye texture from the left eye texture and the depth map, and writes the right eye texture to the right eye texture memory 412. Thereafter, the process proceeds to step S905.
  • step S905 the rendering unit 374 first reads the left-eye texture from the left-eye texture memory 411 and projects a panoramic image represented by the left-eye texture onto the spherical model 820.
  • the rendering unit 374 receives an angle ⁇ representing the viewing direction designated by the user from the control unit 402.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is left eye Convert to frames.
  • the left eye frame is written into the left eye frame buffer 413. Thereafter, the process proceeds to step S906.
  • step S906 the rendering unit 374 first reads the right-eye texture from the right-eye texture memory 412, and projects the parallax panoramic image represented by the right-eye texture onto the spherical model 820.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the parallax panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is displayed. Convert to right eye frame.
  • the right eye frame is written into the right eye frame buffer 414. Thereafter, the process proceeds to step S907.
  • step S907 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413, and reads the right-eye frame from the right-eye frame buffer 414. Next, the output unit 380 multiplexes the left-eye frame and the right-eye frame into an HDMI serial signal and outputs the serial signal to the display device 103. Thereafter, the process proceeds to step S908.
  • step S908 the control unit 402 checks whether the UO module 330 has received an instruction to end the street view from the user. If the instruction has been received, the process ends. If the instruction has not been received, the process proceeds to step S1001.
  • step S1001 the control unit 402 checks whether or not the UO module 330 has been instructed by the user to change the viewing direction, that is, the direction of the avatar 202. If the change has been instructed, the process proceeds to step S1002. If the change is not instructed, the process proceeds to step S1006.
  • step S1002 the UO module 330 is instructed by the user to change the direction of the avatar 202.
  • the control unit 402 decodes the new direction of the avatar 202 indicated by the signal from the UO module 330, and determines the changed viewing direction from the new direction.
  • the control unit 402 further instructs the rendering unit 374 to change the viewing direction.
  • the rendering unit 374 points the virtual camera 830 in the changed viewing direction. Thereafter, the process proceeds to step S1003.
  • step S1003 the rendering unit 374 converts an area included in the angle of view ⁇ of the virtual camera 830 in the panoramic image projected onto the spherical model 820 into a left-eye frame.
  • the left eye frame is written into the left eye frame buffer 413. Thereafter, the process proceeds to step S1004.
  • step S1004 the rendering unit 374 converts a region included in the angle of view ⁇ of the virtual camera 830 in the parallax panorama image projected onto the spherical model 820 into a right eye frame.
  • the right eye frame is written into the right eye frame buffer 414. Thereafter, the process is repeated from step S908.
  • step S1006 the control unit 402 checks whether or not the UO module 330 has been instructed by the user to change the viewpoint, that is, the position of the avatar 202. If the change has been instructed, the process proceeds to step S1007. If the change is not instructed, the process is repeated from step S908.
  • step S1007 the control unit 402 causes the drawing unit 370 to generate a substitute stereo image, and causes the output unit 380 to output the substitute stereo image to the display device 103. Details of the processing will be described later.
  • step S907 the alternative stereo image is displayed on the screen 131 of the display device 103 until the pair of the left-eye frame and the right-eye frame that can be seen from the changed viewpoint is output.
  • the control unit 402 causes the drawing unit 370 to generate a substitute stereo image and causes the output unit 380 to output the substitute stereo image to the display device 103. Thereby, the substitute stereo image is displayed on the screen 131 during the waiting time.
  • the viewpoint is changed in the direction of the avatar 202. That is, the viewpoint is changed so as to approach the object shown on the screen 131 of the display device 103.
  • the following visual effects are used to generate a substitute stereo image.
  • the control unit 402 causes the rendering unit 374 to scale the stereo image that can be seen from the viewpoint before the change, and uses the stereo image obtained as a substitute stereo image.
  • the size of the subject is larger than the stereo image seen from the viewpoint before the change. As a result, for the person who is viewing the screen 131, it appears that the viewpoint is approaching the subject. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be suppressed.
  • FIG. 11 is a schematic diagram illustrating a stereo image scaling process performed by the rendering unit 374.
  • Each frame buffer 413 and 414 stores a frame pair of a stereo image that can be seen from the viewpoint before the change.
  • the rendering unit 374 scales the frame pair by a predetermined magnification to enlarge the stereo image 1101 that can be seen from the viewpoint before the change.
  • the rendering unit 374 extracts a one-frame size region 1103 from the center of the enlarged stereo image 1102.
  • the position of the area 1103 is adjusted so that the center of the area 1103 coincides with the center of the original stereo image 1101.
  • pixel data belonging to an area outside the area 1103 is discarded.
  • the pixel data in the area 1103 is overwritten in the frame buffers 413 and 414 and used as a substitute stereo image.
  • the object 1105 shown in the alternative stereo image 1103 is larger in size than the object 1104 shown in the original stereo image 1101.
  • the viewpoint appears to approach the object.
  • FIG. 12A is a top view schematically showing a state when the subject OBJ is photographed by the camera installed at the first viewpoint VP1.
  • the Z axis represents the viewing direction of the camera, and the Y axis represents the horizontal direction perpendicular to the Z axis.
  • the first viewpoint VP1 is separated from the subject OBJ by a first distance d1 in the Z-axis direction.
  • An image IM1 of the subject OBJ is projected on the imaging surface SCR of the camera.
  • FIG. 12B is a top view schematically showing a state when the subject OBJ is photographed by the camera installed at the second viewpoint VP2. The second viewpoint VP2 is separated from the subject OBJ by a second distance d2 in the Z-axis direction.
  • An image IM2 of the subject OBJ is projected on the imaging surface SCR of the camera.
  • the scaling factor SR S2 / S1 at that time is expressed by the following equation (1):
  • the control unit 402 determines the scaling factor SR according to Equation (1) from the distance L between the viewpoints before and after the change and the second distance d2.
  • each of the first distance d1 and the second distance d2 cannot be uniquely determined.
  • the first distance d1 and the second distance d2 may be accurately determined by setting a reference subject.
  • the control unit 402 accurately calculates the distance L between the first viewpoint VP1 and the second viewpoint VP2, while fixing the second distance d2 to a predetermined number.
  • the control unit 402 may obtain a number proportional to the depth of each pixel using the depth map at the viewpoint before the change, and use the number as the second distance d2 at the pixel.
  • An alternative stereo image can be generated simply by processing the frames already stored in the frame buffers 413 and 414. Therefore, an alternative stereo image is generated sufficiently earlier than the processing of steps S901 to S907 for the panoramic image seen from the changed viewpoint. Therefore, the alternative stereo image can be displayed on the screen 131 during the waiting time of the processing.
  • FIG. 13 is a flowchart of the alternative stereo image generation process in step S1007 shown in FIG.
  • the substitute stereo image is generated by scaling the frames already stored in the frame buffers 413 and 414.
  • step S1301 the control unit 402 obtains the latitude and longitude of each viewpoint before and after the change. Specifically, the control unit 402 reads the values written in the latitude 504 and the longitude 505 with reference to the street view information regarding each viewpoint before and after the change. Thereafter, processing proceeds to step S1302.
  • step S1302 the control unit 402 obtains the distance between the viewpoint before change and the viewpoint after change using the latitude and longitude obtained in step S1301. Specifically, the shape of the earth is first approximated by a sphere. Next, let x be the angle formed by the line segment l 1 connecting the viewpoint before change and the center of the earth and the line segment l 2 connecting the viewpoint after change and the center of the earth. Then, the angle x is expressed by the following formula (2) by using the spherical trigonometry and using the latitude ⁇ 1 and longitude ⁇ 1 of the viewpoint before the change and the latitude ⁇ 2 and longitude ⁇ 2 of the viewpoint after the change:
  • step S1303 the control unit 402 calculates the scaling factor SR by substituting the distance L and the predetermined number d2 obtained in step S1302 into the equation (1). Thereafter, processing proceeds to step S1304.
  • step S1304 the control unit 402 causes the rendering unit 374 to scale the left eye frame. Specifically, the rendering unit 374 first reads the left-eye frame from the left-eye frame buffer 413, and scales the left-eye frame with the magnification SR obtained in step S1303. Next, the rendering unit 374 extracts an area of one frame size from the center portion of the scaled left eye frame, and overwrites it in the left eye frame buffer 413. Thereafter, processing proceeds to step S1305.
  • step S1305 the control unit 402 causes the rendering unit 374 to scale the right eye frame. Specifically, the rendering unit 374 first reads the right-eye frame from the right-eye frame buffer 414, and scales the right-eye frame with the magnification SR obtained in step S1303. Next, the rendering unit 374 extracts an area of one frame size from the center part of the scaled right eye frame, and overwrites the right eye frame buffer 414. Thereafter, the process proceeds to step S1306.
  • step S1306 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413 and reads the right-eye frame from the right-eye frame buffer 414.
  • the output unit 380 multiplexes the read frame pair into an HDMI serial signal and outputs the serial signal to the display device 103.
  • the central portion of the stereo image that can be seen from the viewpoint before the change is enlarged and displayed on the screen 131 of the display device 103 as an alternative stereo image. While the alternative stereo image is displayed on the screen 131, the process for the panoramic image seen from the changed viewpoint is repeated from step S901 shown in FIG.
  • the viewpoint is changed in the direction of the avatar 202.
  • a reduced stereo image seen from the viewpoint before the change is used as an alternative stereo image. It may be used. In that case, transparent or background pixel data is added to the outside of the reduced image.
  • the stereo image is temporarily changed from the viewpoint before the change to the stereo image when the direction in which the viewpoint is changed is viewed.
  • a scaled stereo image may be used as an alternative stereo image.
  • the playback device 102 when the change of the viewpoint of the panoramic image is requested, the playback device 102 according to the first embodiment of the present invention first generates a substitute stereo image by scaling the stereo image seen from the viewpoint before the change. Next, the playback device 102 outputs a substitute stereo image until a panoramic image seen from the changed viewpoint is acquired and a new stereo image is generated. Since the subject of the alternative stereo image is larger in size than the subject of the stereo image seen from the viewpoint before the change, the viewpoint of the alternative stereo image seems to be closer to the subject than the viewpoint before the change. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be suppressed. The alternative stereo image is generated only by scaling the already generated frame pair.
  • the alternative stereo image generation process is sufficiently faster than the new stereo image generation process. Therefore, when a change of viewpoint is requested, the alternative stereo image is displayed on the screen 131 of the display device 103 in a short time. In this way, the playback device 102 responds to the request for changing the viewpoint in a short time, so that it is possible to reduce discomfort given to the user.
  • the scaled SR obtained by substituting the distance L between the viewpoints before and after the change into the equation (1), and the scaled stereo image seen from the viewpoint before the change is used as the substitute stereo image.
  • the control unit 402 causes the rendering unit 374 to repeatedly generate a substitute stereo image from the stereo image that can be seen from the viewpoint before the change every time the scaling magnification is changed from 1 to the above-described magnification SR at a predetermined interval. May be. In that case, it seems to the viewer who views the screen 131 of the display device 103 that the viewpoint of the stereo image displayed on the screen 131 is gradually moving to the changed viewpoint. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be further suppressed.
  • the scaling factor may be determined as a value obtained by multiplying the distance L between the viewpoints before and after the change by an arbitrary proportional constant in addition to the magnification SR obtained by the equation (1).
  • the magnification can be obtained by a simpler calculation even if it is less accurate than the magnification SR obtained by the equation (1).
  • the scaled stereo image is output as it is as an alternative stereo image.
  • the control unit 402 may correct the parallax value of the alternative stereo image to obtain a more natural sense of depth. Specifically, the control unit 402 first generates a depth map from the frames stored in the frame buffers 413 and 414.
  • control unit 402 specifies the coordinates of the pixel where the common pixel data is located from each of the left-eye frame and the right-eye frame using a known method such as stereo matching, and The difference is set as the pixel data of the depth map.
  • the control unit 402 causes the depth represented by each pixel of the depth map to approach the depth of the screen.
  • the control unit 402 causes the parallax panoramic image generation unit 373 to generate a right eye frame from the depth map and the left eye frame. A pair of the left-eye frame and the right-eye frame obtained in this way is used as an alternative stereo image.
  • control unit 402 may shift the coordinates of the pixels of the left-eye frame uniformly in the left direction and shift the coordinates of the pixels of the right-eye frame uniformly in the right direction. In that case, since the depth of the entire alternative stereo image is changed in a direction away from the viewpoint, the sense of depth of an object that is visible in front of the screen is particularly weakened.
  • the control unit 402 when generating the substitute stereo image, causes the rendering unit 374 to scale the frame pairs stored in the frame buffers 413 and 414, respectively.
  • the control unit 402 may cause the drawing unit 370 to scale the left-eye texture and the depth map, and cause the parallax panoramic image generation unit 373 to generate the right-eye texture from the scaled left-eye texture and depth map.
  • an alternative stereo image is obtained by extracting a region in a predetermined viewing direction from the scaled left-eye texture and right-eye texture. Since the left-eye texture and the depth map are expressed by the Mercator projection, when the scaled texture is projected onto the spherical model, a large distortion occurs in the vicinity of the pole.
  • control unit 402 applies blurring processing to the rendering unit 374 on the upper end portion and the lower end portion of the left eye frame and the right eye frame, respectively. This makes it difficult to see the distortion, thereby reducing the uncomfortable feeling given to the viewer of the alternative stereo image.
  • the playback device 102 is independent of the display device 103.
  • the playback device 102 may be incorporated in the display device 103.
  • only the functional unit used for the image processing related to the street view shown in FIG. 4 in the playback device 102 may be mounted on the display device 103.
  • the elements 371 to 374 of the drawing unit 370 may be separated into a plurality of chips.
  • a decoder and a rendering unit that are used for reproducing 3D video may be combined.
  • step S902 is followed by step S903.
  • step S902 and step S903 may be executed in parallel.
  • Embodiment 2 The playback device according to the second embodiment of the present invention differs from the playback device according to the first embodiment in a method of generating an alternative stereo image. Regarding other features such as configuration, the playback device according to the second embodiment is the same as the playback device according to the first embodiment. For the details of the similar components, the description of the first embodiment is cited.
  • the viewpoint after the change is closer to the subject than the viewpoint before the change. Therefore, in the second embodiment, the parallax value of the stereo image is increased or decreased so that the depth of the object shown in the stereo image seen from the viewpoint before the change approaches the viewpoint. As a result, the obtained stereo image is used as a substitute stereo image.
  • FIG. 14A is a schematic diagram showing an image of the right-eye frame that can be seen from the viewpoint before the change.
  • the image includes a sphere 1401 and a column 1402.
  • the depth of the sphere 1401 is in front of the screen, and the depth of the pillar 1402 is inward of the screen.
  • the positions of the sphere 1403 and the pillar 1404 in the left-eye frame that can be seen from the viewpoint before the change are indicated by broken lines.
  • the sphere 1401 in the right eye frame is positioned to the left by the first parallax value HL1 relative to the sphere 1403 in the left eye frame, and the column 1402 in the right eye frame is the left eye.
  • the second disparity value HR1 is positioned to the right of the pillar 1404 in the work frame.
  • the control unit 402 causes the rendering unit 374 to shift the coordinates of the pixels of the right-eye frame uniformly in the left direction.
  • the obtained right-eye frame is used as a substitute stereo image together with the left-eye frame.
  • FIG. 14B is a schematic diagram showing an image of the right-eye frame of the alternative stereo image.
  • the positions of the sphere 1403 and the pillar 1404 in the left eye frame are indicated by broken lines.
  • the sphere 1405 in the right eye frame is positioned to the left by the third parallax value HL2 from the sphere 1403 in the left eye frame, and the column 1406 in the right eye frame is The fourth parallax value HR2 is positioned to the right of the pillar 1404 in the left eye frame.
  • the third parallax value HL2 is larger than the first parallax value HL1 by a difference ⁇ H
  • the fourth parallax value HR2 is different from the second parallax value HR1 by a difference ⁇ H. Only small.
  • the control unit 402 determines the increase / decrease amount ⁇ H of the parallax value from the distance between the viewpoints before and after the change. Thereby, the sphere and the column in the alternative stereo image appear closer to the viewpoint by the distance between the viewpoints before and after the change than the sphere and the column in the stereo image viewed from the viewpoint before the change. As a result, for the viewer who sees the alternative stereo image, the viewpoint appears to be closer to the sphere and the column than the viewpoint before the change.
  • the alternative stereo image can be generated simply by processing the right eye frame already stored in the right eye frame buffer 414. Therefore, for the panoramic image that can be seen from the changed viewpoint, an alternative stereo image is generated sufficiently earlier than the processing of steps S901 to S907 shown in FIG. 9 is completed. Therefore, the alternative stereo image can be displayed on the screen 131 during the waiting time of the processing.
  • FIG. 15 is a flowchart of the alternative stereo image generation process in step S1007 shown in FIG.
  • the substitute stereo image is generated by shifting the coordinates of the pixels of the right-eye frame already stored in the right-eye frame buffer 414.
  • step S1501 as in step S1301 shown in FIG. 13, the control unit 402 obtains the latitude and longitude of each viewpoint before and after the change. Thereafter, the process proceeds to step S1502.
  • step S1502 similarly to step S1302 shown in FIG. 13, the control unit 402 substitutes the latitude and longitude obtained in step S1501 into equation (2), so that the viewpoint before the change and the post-change viewpoint are changed. Find the distance to the viewpoint. Thereafter, processing proceeds to step S1503.
  • step S1503 the control unit 402 obtains the increase / decrease amount ⁇ H of the parallax value of the alternative stereo image, that is, the number of pixels by which the coordinates of the pixels of the right-eye frame are uniformly shifted from the distance obtained in step S1502. Thereafter, processing proceeds to step S1504.
  • step S1504 the control unit 402 causes the rendering unit 374 to shift the coordinates of the pixel of the right eye frame to the left by the increase / decrease amount ⁇ H of the parallax value. Specifically, the rendering unit 374 first reads the right eye frame from the right eye frame buffer 414. Next, the rendering unit 374 overwrites the right-eye frame in the right-eye frame buffer 414 so that the coordinates of each pixel are shifted leftward by the increase / decrease amount ⁇ H obtained in step S1503. Thereafter, processing proceeds to step S1505.
  • step S1505 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413 and reads the right-eye frame from the right-eye frame buffer 414.
  • the output unit 380 multiplexes the read frame pair into an HDMI serial signal and outputs the serial signal to the display device 103.
  • the parallax value of the stereo image seen from the viewpoint before the change is changed and displayed on the screen 131 of the display device 103 as an alternative stereo image. While the alternative stereo image is displayed on the screen 131, the process for the panoramic image seen from the changed viewpoint is repeated from step S901 shown in FIG.
  • the viewpoint is changed so as to approach the subject. That is, the line of sight is changed in the direction of the avatar 202.
  • the viewpoint is changed in the direction opposite to the direction of the avatar 202, that is, when the viewpoint is changed so as to move away from the object displayed on the screen 131 of the display device 103, the coordinates of the pixels of the right-eye frame are uniformly shifted to the right. Good.
  • the subject appears to have moved to the back rather than the stereo image seen from the viewpoint before the change.
  • the stereo image is temporarily changed from the viewpoint before the change to the stereo image when the direction in which the viewpoint is changed is viewed.
  • a stereo image obtained by increasing or decreasing the parallax value may be used as an alternative stereo image.
  • the playback device 102 when the change of the viewpoint of the panoramic image is requested, the playback device 102 according to the second embodiment of the present invention first generates an alternative stereo image by increasing or decreasing the parallax value of the stereo image seen from the viewpoint before the change. To do. Next, the playback device 102 outputs a substitute stereo image until a panoramic image seen from the changed viewpoint is acquired and a new stereo image is generated. The subject of the alternative stereo image appears to be positioned in front of the subject of the stereo image seen from the viewpoint before the change. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be suppressed. In addition, the alternative stereo image is generated only by shifting the pixel coordinates for one of the already generated frames.
  • the alternative stereo image generation process is sufficiently faster than the new stereo image generation process. Therefore, when a change of viewpoint is requested, the alternative stereo image is displayed on the screen 131 of the display device 103 in a short time. In this way, the playback device 102 responds to the request for changing the viewpoint in a short time, so that it is possible to reduce discomfort given to the user.
  • the pixel coordinates of the right-eye frame shifted to the left by the amount of increase / decrease ⁇ H of the parallax value obtained from the distance between the viewpoints before and after the change are used as the right-eye frame of the alternative stereo image.
  • the control unit 402 every time the control unit 402 changes the increase / decrease amount of the parallax value from 0 to the above increase / decrease amount ⁇ H at a predetermined interval, the control unit 402 causes the rendering unit 374 to substitute an alternative stereo image from the stereo image that can be seen from the viewpoint before the change. It may be generated repeatedly. In that case, it seems to the viewer who views the screen 131 of the display device 103 that the viewpoint of the stereo image displayed on the screen 131 is gradually moving to the changed viewpoint. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be further suppressed.
  • Embodiment 3 The playback device according to the third embodiment of the present invention differs from the playback device according to the first embodiment in a method for generating an alternative stereo image. Regarding other features such as configuration, the playback device according to the third embodiment is the same as the playback device according to the first embodiment. For the details of the similar components, the description of the first embodiment is cited.
  • the control unit 402 causes the rendering unit 374 to move the virtual camera 830 that captures a panoramic image projected on the spherical model 820 from the center of the spherical model 820 in the viewing direction.
  • a stereo image taken by the virtual camera 830 from the moved position is used as a substitute stereo image.
  • FIG. 16 is a schematic diagram showing the inside of the spherical model 820 seen from the north pole direction.
  • the initial position of virtual camera 830 is the center CTR of spherical model 820.
  • the control unit 402 causes the rendering unit 374 to move the virtual camera 830 by a distance ⁇ D from the center CRT toward the viewing direction VD.
  • the distance ⁇ D is set by the control unit 402 to a value proportional to the distance between the viewpoints before and after the change.
  • the region included in the angle of view of the virtual camera 830 in the panoramic image projected onto the spherical model 820 is narrowed. That is, the angle of view of the virtual camera 830 after movement includes only the center portion A2 of the area A1 included in the angle of view of the virtual camera 830 when located at the center CRT.
  • FIG. 17 is a schematic diagram showing a change in the stereo image accompanying the movement of the virtual camera by the rendering unit 374.
  • a panorama image and a parallax panorama image that can be seen from the viewpoint before the change are stored as a left-eye texture and a right-eye texture, respectively.
  • the rendering unit 374 projects each texture onto the spherical model again.
  • the rendering unit 374 directs the virtual camera from the moved position to the viewing direction, and calculates an image when the panoramic image projected on the spherical model is captured by the virtual camera.
  • the range of the panoramic image captured by the virtual camera 830 changes from the area A1 to the center A2.
  • an area corresponding to the central portion A2 indicated by the broken line in FIG. 17 is generated as the stereo image A3 captured by the virtual camera after movement.
  • the stereo image A3 is overwritten in the frame buffers 413 and 414 and used as a substitute stereo image.
  • the object 1701 shown in the alternative stereo image A3 is larger in size than the object 1702 shown in the original stereo image A1.
  • the viewpoint appears to approach the object.
  • An alternative stereo image can be generated simply by processing a texture pair already stored in the texture memories 411 and 412. Therefore, for the panoramic image that can be seen from the changed viewpoint, an alternative stereo image is generated sufficiently earlier than the processing of steps S901 to S907 shown in FIG. 9 is completed. Therefore, the alternative stereo image can be displayed on the screen 131 during the waiting time of the processing.
  • FIG. 18 is a flowchart of an alternative stereo image generation process in step S1007 shown in FIG.
  • an alternative stereo image is generated by generating a stereo image again from each texture already stored in each texture memory 411, 412.
  • step S1801 as in step S1301 shown in FIG. 13, the control unit 402 obtains the latitude and longitude of each viewpoint before and after the change. Thereafter, processing proceeds to step S1802.
  • step S1802 similarly to step S1302 shown in FIG. 13, the control unit 402 substitutes the latitude and longitude obtained in step S1501 into equation (2), so that the viewpoint before the change and the post-change viewpoint are changed. Find the distance to the viewpoint. Thereafter, processing proceeds to step S1803.
  • step S1803 the control unit 402 moves the virtual camera 830 to the rendering unit 374 by the distance ⁇ D obtained in step S1802 from the center CTR of the spherical model 820 toward the viewing direction VD as shown in FIG.
  • the range included in the angle of view of the virtual camera 830 in the panoramic image projected on the spherical model 820 changes from the area A1 to the center A2 as shown in FIG. Thereafter, processing proceeds to step S1804.
  • step S1804 the control unit 402 causes the rendering unit 374 to convert the left-eye frame into an image photographed by the virtual camera 830 after movement. Specifically, the rendering unit 374 first reads the left eye texture from the left eye texture memory 411. Next, the rendering unit 374 projects the left-eye texture onto the spherical model, and calculates an image when the projected left-eye texture is captured by the moved virtual camera. The rendering unit 374 further overwrites the image obtained by the calculation in the left-eye frame buffer 413 as a new left-eye frame. Thereafter, processing proceeds to step S1805.
  • step S1805 the control unit 402 causes the rendering unit 374 to convert the right-eye frame into an image captured by the virtual camera 830 after movement. Specifically, the rendering unit 374 first reads the right eye texture from the right eye texture memory 412. Next, the rendering unit 374 projects the right-eye texture onto the spherical model, and calculates an image when the projected right-eye texture is captured by the moved virtual camera. The rendering unit 374 further overwrites the right-eye frame buffer 414 with the image obtained by the calculation as a new right-eye frame. Thereafter, processing proceeds to step S1806.
  • step S1806 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413 and reads the right-eye frame from the right-eye frame buffer 414.
  • the output unit 380 multiplexes the read frame pair into an HDMI serial signal and outputs the serial signal to the display device 103.
  • the center portion of the stereo image seen from the viewpoint before the change is enlarged and displayed on the screen 131 of the display device 103 as an alternative stereo image. While the alternative stereo image is displayed on the screen 131, the process for the panoramic image seen from the changed viewpoint is repeated from step S901 shown in FIG.
  • the viewpoint is changed so as to approach the subject. That is, the line of sight is changed in the direction of the avatar 202.
  • the control unit 402 causes the rendering unit 374 to display the virtual camera 830 and You may move from the center CTR in the direction opposite to the viewing direction VD.
  • an area included in the angle of view of the virtual camera 830 is expanded in the panoramic image projected on the spherical model 820.
  • the rendering unit 374 generates an alternative stereo image by converting a region included in the angle of view of the virtual camera 830 after the movement from the left-eye texture and the right-eye texture into a frame pair.
  • the stereo image may be changed from the viewpoint before the change to a stereo image when the direction in which the viewpoint is changed is viewed.
  • the rendering unit moves the virtual camera in the direction in which the viewpoint is changed, and recaptures the panoramic image.
  • the obtained stereo image may be used as a substitute stereo image.
  • the playback device 102 first moves the virtual camera 830 from the center CTR of the spherical model 820 when the change of the viewpoint of the panoramic image is requested.
  • the stereo image that can be seen from is converted into a substitute stereo image.
  • the playback device 102 outputs a substitute stereo image until a panoramic image seen from the changed viewpoint is acquired and a new stereo image is generated. Since the subject of the alternative stereo image is larger in size than the subject of the stereo image seen from the viewpoint before the change, the viewpoint of the alternative stereo image seems to be closer to the subject than the viewpoint before the change. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be suppressed.
  • the alternative stereo image is generated from the already generated texture pair.
  • the alternative stereo image generation process is sufficiently faster than the new stereo image generation process. Therefore, when a change of viewpoint is requested, the alternative stereo image is displayed on the screen 131 of the display device 103 in a short time. In this way, the playback device 102 responds to the request for changing the viewpoint in a short time, so that it is possible to reduce discomfort given to the user.
  • an alternative stereo image generated by moving the virtual camera 830 by the distance between the viewpoints before and after the change is used.
  • the control unit 402 every time the control unit 402 changes the displacement of the virtual camera 830 from 0 to the distance between the viewpoints before and after the change at a predetermined interval, the control unit 402 substitutes the rendering unit 374 for a panoramic image that can be seen from the viewpoint before the change.
  • Stereo images may be repeatedly generated. In that case, it seems to the viewer who views the screen 131 of the display device 103 that the viewpoint of the stereo image displayed on the screen 131 is gradually moving to the changed viewpoint. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be further suppressed.
  • Embodiment 4 The playback apparatus according to the fourth embodiment of the present invention differs from the playback apparatus according to the first embodiment in a method for generating an alternative stereo image. Regarding other features such as configuration, the playback device according to the fourth embodiment is the same as the playback device according to the first embodiment. For the details of the similar components, the description of the first embodiment is cited.
  • a panorama image having a resolution lower than that resolution (hereinafter referred to as a low-resolution panorama image) is prepared in the server 106.
  • the resolution of the panorama image is 3584 pixels ⁇ 1536 pixels
  • the same processing is performed for the low-resolution panoramic image before the processing of steps S901 to S907 is performed for the original resolution panoramic image. Since the low-resolution panoramic image has a smaller amount of data than the original resolution panoramic image, the above processing ends sufficiently early. Therefore, a stereo image converted from a low-resolution panoramic image can be used as an alternative stereo image.
  • FIG. 19 is a table showing a data structure of street view information (hereinafter referred to as low resolution street view information) in which a low resolution panoramic image is incorporated.
  • the low-resolution street view information includes ID, size, latitude, longitude, altitude, direction, and slope, like the street view information shown in FIG.
  • the low resolution street view information is replaced with the low resolution compressed panorama image 1901 and the low resolution compressed panorama depth 1902 instead of the compressed panorama image and the compressed panorama depth. Including.
  • the item “low-resolution compressed panoramic image” 1901 is image data compressed in JPEG format, and represents a low-resolution panoramic image with a point assigned with an ID as a viewpoint.
  • the item “low-resolution compressed panorama depth” 1902 is image data compressed in PNG format, and depth information of the low-resolution compressed panoramic image 1901, that is, an object such as a building wall or road surface reflected in the low-resolution panoramic image. Depth is expressed in pixels.
  • FIG. 20 is a flowchart of the alternative stereo image generation process in step S1007 shown in FIG.
  • the substitute stereo image is generated from the low-resolution panoramic image.
  • the process shown in FIG. 20 is started when the user operates the remote controller 105 to instruct the playback device 102 to change the viewpoint in the street view.
  • step S2001 the control unit 402 detects that the signal from the UO module 330 indicates a change in viewpoint.
  • the control unit 402 decodes the point where the changed viewpoint is located from the signal, searches for the ID assigned to the point, and passes it to the server 106 through the network interface 302.
  • the server 106 provides the playback device 102 with low-resolution street view information including the same ID as the passed ID.
  • the file system 401 downloads the low resolution street view information from the server 106 through the network interface 302. Thereafter, the process proceeds to step S2002.
  • step S2002 the file system 401 separates the low-resolution compressed panoramic image from the low-resolution street view information and passes it to the decoder 371.
  • the decoder 371 decodes the left-eye texture from the low-resolution compressed panoramic image and writes it to the left-eye texture memory 411. Thereafter, the process proceeds to step S2003.
  • step S2003 the file system 401 separates the low resolution compressed panorama depth from the low resolution street view information, and passes it to the depth generation unit 372.
  • the depth generation unit 372 generates a depth map from the low resolution compressed panorama depth and writes the depth map in the depth map memory 410.
  • step S2002 and step S2003 may be executed in parallel. Thereafter, the process proceeds to step S2004.
  • step S2004 the parallax panorama image generation unit 373 first reads the left-eye texture from the left-eye texture memory 411 and reads the depth map from the depth map memory 410. Next, the parallax panoramic image generation unit 373 generates a right eye texture from the left eye texture and the depth map, and writes the right eye texture to the right eye texture memory 412. Thereafter, the process proceeds to step S2005.
  • step S2005 the rendering unit 374 first reads the left-eye texture from the left-eye texture memory 411 and projects a panoramic image represented by the left-eye texture onto the spherical model 820.
  • the rendering unit 374 receives an angle ⁇ representing the viewing direction designated by the user from the control unit 402.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is left eye Convert to frames.
  • the left eye frame is written into the left eye frame buffer 413. Thereafter, the process proceeds to step S2006.
  • step S2006 the rendering unit 374 first reads the right-eye texture from the right-eye texture memory 412 and projects the parallax panoramic image represented by the right-eye texture onto the spherical model 820.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the parallax panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is displayed. Convert to right eye frame.
  • the right eye frame is written into the right eye frame buffer 414. Thereafter, the process proceeds to step S2007.
  • step S2007 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413, and reads the right-eye frame from the right-eye frame buffer 414.
  • the output unit 380 multiplexes the left-eye frame and the right-eye frame into an HDMI serial signal and outputs the serial signal to the display device 103.
  • the stereo image converted from the low-resolution panoramic image is displayed on the screen 131 of the display device 103 as an alternative stereo image. While the alternative stereo image is displayed on the screen 131, the process for the panoramic image seen from the changed viewpoint is repeated from step S901 shown in FIG.
  • the playback device 102 acquires a low-resolution panoramic image before a panoramic image of the original resolution when a change of the viewpoint of the panoramic image is requested, and the low-resolution Convert panoramic images to alternate stereo images.
  • the playback device 102 outputs a substitute stereo image until a panoramic image with the original resolution is acquired and a new stereo image is generated.
  • the alternative stereo image has only a lower resolution than the stereo image seen from the changed viewpoint, and the rough shape of the subject matches. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be suppressed. Further, the data amount of the low-resolution panoramic image is sufficiently smaller than that of the original resolution panoramic image.
  • the alternative stereo image generation process is sufficiently faster than the new stereo image generation process. Therefore, when a change of viewpoint is requested, the alternative stereo image is displayed on the screen 131 of the display device 103 in a short time. In this way, the playback device 102 responds to the request for changing the viewpoint in a short time, so that it is possible to reduce discomfort given to the user.
  • the viewing direction when the low-resolution panoramic image is converted into the alternative stereo image may be different from the viewpoint changing direction. That is, when the viewpoint is changed in a direction different from the direction of the avatar 202, a stereo image that is visible in the direction of the avatar 202 from the changed viewpoint is generated as a substitute stereo image from the low-resolution panoramic image.
  • the stereo image itself converted from the low-resolution panoramic image is used as the alternative stereo image.
  • what changed the stereo image seen from the viewpoint before change by morphing using the stereo image may be used as an alternative stereo image.
  • Morphing is a computer graphics technique that expresses a process in which one image naturally changes to another image as a moving image.
  • the control unit 402 downloads the low-resolution panoramic image from the server 106 and converts the low-resolution panoramic image into a stereo image that can be seen from the changed viewpoint. Convert it.
  • the control unit 402 further causes the rendering unit 374 to substitute a series of stereo images representing a process in which a stereo image seen from the viewpoint before the change is naturally changed to a stereo image seen from the viewpoint after the change as a substitute stereo image. Generate.
  • FIGS. 21A to 21D show a series of alternative stereo images showing a process in which a stereo image seen from the viewpoint before change is naturally changed to a stereo image seen from the viewpoint after change by morphing. It is a schematic diagram.
  • the alternative stereo images are displayed in the order of (a), (b), (c), and (d) in FIG.
  • the alternative stereo image shown in FIG. 21A matches the first stereo image 2101 that can be seen from the viewpoint before the change.
  • the alternative stereo image shown in (d) of FIG. 21 matches the second stereo image 2102 seen from the changed viewpoint.
  • objects 2113 and 2114 having shapes that overlap with the subjects 2111 and 2112 of the second stereo image 2102 appear in the subject of the first stereo image 2101.
  • FIG. 21B objects 2113 and 2114 having shapes that overlap with the subjects 2111 and 2112 of the second stereo image 2102 appear in the subject of the first stereo image 2101.
  • FIG. 21B objects 2113 and 2114 having shapes that overlap with the subjects 2111 and 2112 of the second stereo image 2
  • the shapes of the subjects 2115 and 2116 are closer to the shapes of the subjects 2111 and 2112 of the second stereo image 2102.
  • the series of alternative stereo images represents a process in which the subject gradually changes from the shape in the first stereo image 2101 to the shape in the second stereo image 2102.
  • the uncomfortable feeling that the user receives from the alternative stereo image can be further reduced.
  • FIG. 22 is a flowchart of an alternative stereo image generation process using morphing in step S1007 shown in FIG.
  • the process shown in FIG. 22 is started when the user operates the remote controller 105 to instruct the playback device 102 to change the viewpoint in the street view.
  • step S2201 the control unit 402 detects that the signal from the UO module 330 indicates a change in viewpoint.
  • the control unit 402 decodes the point where the changed viewpoint is located from the signal, searches for the ID assigned to the point, and passes it to the server 106 through the network interface 302.
  • the server 106 provides the playback device 102 with low-resolution street view information including the same ID as the passed ID.
  • the file system 401 downloads the low resolution street view information from the server 106 through the network interface 302. Thereafter, the process proceeds to step S2202.
  • step S2202 the file system 401 separates the low-resolution compressed panoramic image from the low-resolution street view information and passes it to the decoder 371.
  • the decoder 371 decodes the left-eye texture from the low-resolution compressed panoramic image and writes it to the left-eye texture memory 411.
  • the left-eye texture memory 411 stores the first left-eye texture representing the original resolution panoramic image that can be seen from the viewpoint before the change and the second texture that represents the low-resolution panoramic image that can be seen from the viewpoint after the change. Both the left eye texture and the left eye texture are stored. Thereafter, the process proceeds to step S2203.
  • step S2203 the file system 401 separates the low-resolution compressed panorama depth from the low-resolution street view information and passes it to the depth generation unit 372.
  • the depth generation unit 372 generates a depth map from the low resolution compressed panorama depth and writes the depth map in the depth map memory 410.
  • the depth map memory 410 includes a first depth map for a panoramic image having an original resolution that can be seen from the viewpoint before the change, and a second depth map for a panoramic image having a low resolution that can be seen from the viewpoint after the change. Both are stored.
  • step S2202 and step S2203 may be executed in parallel. Thereafter, processing proceeds to step S2204.
  • step S2204 the drawing unit 370 reads a pair of the first left-eye texture and the second left-eye texture from the left-eye texture memory 411, and generates an alternative texture by morphing from the pair.
  • the substitute texture is stored in the left-eye texture memory 411.
  • step S2204 is repeated for the pair of left-eye textures, the drawing unit 370 naturally changes the panorama image represented by the alternative texture from the panorama image seen from the viewpoint before the change to the panorama image seen from the viewpoint after the change. Create alternate textures to change.
  • step S2204 the process proceeds to step S2205.
  • step S2205 the drawing unit 370 reads a pair of the first depth map and the second depth map from the depth map memory 410, and generates an alternative depth map from the pair by morphing.
  • the alternative depth map is stored in the depth map memory 410.
  • step S2205 is repeated for a set of depth maps, the rendering unit 370 changes the depth map of the panoramic image viewed from the viewpoint after the change from the depth map of the panorama image viewed from the viewpoint before the change.
  • An alternative depth map is generated so as to change naturally.
  • step S2206 the parallax panoramic image generation unit 373 first reads the alternative texture from the left-eye texture memory 411 and reads the alternative depth map from the depth map memory 410. Next, the parallax panoramic image generation unit 373 generates a right eye texture from the alternative texture and the alternative depth map, and writes the right eye texture to the right eye texture memory 412. Thereafter, processing proceeds to step S2207.
  • step S2207 the rendering unit 374 first reads an alternative texture from the left-eye texture memory 411, and projects a panoramic image represented by the alternative texture onto the spherical model 820.
  • the rendering unit 374 receives an angle ⁇ representing the viewing direction designated by the user from the control unit 402.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is left eye Convert to frames.
  • the left eye frame is written into the left eye frame buffer 413. Thereafter, processing proceeds to step S2208.
  • step S2208 the rendering unit 374 first reads the right-eye texture from the right-eye texture memory 412, and projects the parallax panoramic image represented by the right-eye texture onto the spherical model 820.
  • the rendering unit 374 directs the virtual camera 830 in the direction of the angle ⁇ from the reference direction C, and among the parallax panoramic images projected on the spherical model 820, the region included in the angle of view ⁇ of the virtual camera 830 is displayed. Convert to right eye frame.
  • the right eye frame is written into the right eye frame buffer 414. Thereafter, the process proceeds to step S2209.
  • step S2209 the output unit 380 first reads the left-eye frame from the left-eye frame buffer 413, and reads the right-eye frame from the right-eye frame buffer 414. Next, the output unit 380 multiplexes the left-eye frame and the right-eye frame into an HDMI serial signal and outputs the serial signal to the display device 103. Thus, the stereo image converted from the alternative texture is displayed on the screen 131 of the display device 103 as the alternative stereo image. Thereafter, processing proceeds to step S2210.
  • step S2210 the control unit 402 determines whether the alternative texture matches the second left-eye texture. If the alternative texture matches the second left-eye texture, the process ends. If the alternative texture does not match the second left-eye texture, the process is repeated from step S2204.
  • the processing of steps S901 to S907 shown in FIG. 9 is executed for the original panoramic image that can be seen from the changed viewpoint. That is, while the original resolution panoramic image that can be seen from the changed viewpoint is converted into a stereo image, the alternative stereo image is displayed on the screen 131 of the display device 103 from the panoramic image that can be seen from the viewpoint before the change, from the viewpoint that has been changed. Naturally changes to a visible panoramic image. Therefore, the uncomfortable feeling given to the user by the insertion of the alternative stereo image can be further suppressed.
  • a series of alternative stereo images are generated using low-resolution panoramic images. Since the low-resolution panorama image has a smaller amount of data than the panorama image at the original resolution, even if morphing processing is added, the process of generating an alternative stereo image from the low-resolution panorama image is a stereo image of the panorama image at the original resolution. It's much faster than converting to an image. Therefore, a series of alternative stereo images can be displayed on the screen 131 of the display device 103 during the process of converting the panoramic image of the original resolution into a stereo image. In this way, the playback device 102 responds to the request for changing the viewpoint in a short time, so that it is possible to reduce discomfort given to the user.
  • the present invention relates to a technique for converting a panoramic image into a stereo image. As described above, when a viewpoint change is instructed, an alternative stereo image is displayed until the panoramic image seen from the changed viewpoint is converted into a stereo image. Is displayed on the screen. Thus, the present invention is clearly industrially applicable.

Abstract

La présente invention porte sur un dispositif d'émission d'image qui acquiert une image panoramique et des informations de profondeur d'un sujet de l'image panoramique, convertit une région prédéterminée de l'image panoramique en une image stéréoscopique, et émet l'image stéréoscopique. Une unité de génération d'image panoramique de disparité génère une image panoramique de disparité par déplacement des coordonnées de chaque pixel de l'image panoramique sur la base de la profondeur indiquée par les informations de profondeur. Une unité de rendu projette l'image panoramique et l'image panoramique de disparité sur un modèle stéréoscopique, convertit une région prédéterminée de l'image projetée en une image stéréoscopique, et émet l'image stéréoscopique à partir d'une unité d'émission. Lorsqu'un changement du point de vue de l'image panoramique est requis, une unité de commande fonctionne de sorte qu'une image stéréoscopique alternative soit émise à partir de l'unité d'émission jusqu'à ce qu'une image panoramique soit visualisée à partir d'un point de vue après que le changement est acquis et convertie en une image stéréoscopique. L'image stéréoscopique alternative est similaire soit à l'image stéréoscopique visualisée à partir du point de vue avant le changement, soit à l'image stéréoscopique visualisée à partir du point de vue après le changement. Un procédé pour générer l'image stéréoscopique alternative est plus rapide que le procédé pour acquérir l'image panoramique visualisée à partir du point de vue après le changement et convertir l'image en une image stéréoscopique.
PCT/JP2012/001676 2011-03-31 2012-03-12 Dispositif d'émission d'image stéréoscopique omnidirectionnelle WO2012132267A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201161470044P 2011-03-31 2011-03-31
US61/470,044 2011-03-31

Publications (1)

Publication Number Publication Date
WO2012132267A1 true WO2012132267A1 (fr) 2012-10-04

Family

ID=46930061

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/001676 WO2012132267A1 (fr) 2011-03-31 2012-03-12 Dispositif d'émission d'image stéréoscopique omnidirectionnelle

Country Status (1)

Country Link
WO (1) WO2012132267A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017014126A1 (fr) * 2015-07-21 2017-01-26 株式会社リコー Système de gestion d'image, procédé de gestion d'image, et programme
WO2018135321A1 (fr) * 2017-01-19 2018-07-26 ソニー株式会社 Dispositif et procédé de traitement d'image
CN111434121A (zh) * 2017-10-20 2020-07-17 交互数字Vc控股公司 体积式视频格式的方法、装置和流
CN114827569A (zh) * 2022-04-24 2022-07-29 咪咕视讯科技有限公司 画面显示方法、装置、虚拟现实设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11205653A (ja) * 1998-01-19 1999-07-30 Canon Inc カメラ制御システム及びカメラ制御方法およびクライアントおよびその動作処理を実行するプログラムを記憶した記憶媒体
JP2009010557A (ja) * 2007-06-27 2009-01-15 National Institute Of Information & Communication Technology 奥行データ出力装置及び奥行データ受信装置
JP2011039358A (ja) * 2009-08-14 2011-02-24 Shibaura Institute Of Technology 立体画像地図

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11205653A (ja) * 1998-01-19 1999-07-30 Canon Inc カメラ制御システム及びカメラ制御方法およびクライアントおよびその動作処理を実行するプログラムを記憶した記憶媒体
JP2009010557A (ja) * 2007-06-27 2009-01-15 National Institute Of Information & Communication Technology 奥行データ出力装置及び奥行データ受信装置
JP2011039358A (ja) * 2009-08-14 2011-02-24 Shibaura Institute Of Technology 立体画像地図

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Google Maps JavaScript API V2 Reference", 19 May 2010 (2010-05-19), Retrieved from the Internet <URL:https://developers.google.com/maps/documentation/javascript/v2/reference> *

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017014126A1 (fr) * 2015-07-21 2017-01-26 株式会社リコー Système de gestion d'image, procédé de gestion d'image, et programme
CN107925740A (zh) * 2015-07-21 2018-04-17 株式会社理光 图像管理系统、图像管理方法及程序
EP3328073A4 (fr) * 2015-07-21 2018-05-30 Ricoh Company, Ltd. Système de gestion d'image, procédé de gestion d'image, et programme
JPWO2017014126A1 (ja) * 2015-07-21 2018-06-14 株式会社リコー 画像管理システム、画像管理方法及びプログラム
WO2018135321A1 (fr) * 2017-01-19 2018-07-26 ソニー株式会社 Dispositif et procédé de traitement d'image
US10944975B2 (en) 2017-01-19 2021-03-09 Sony Corporation Image processing device and image processing method
CN111434121A (zh) * 2017-10-20 2020-07-17 交互数字Vc控股公司 体积式视频格式的方法、装置和流
CN114827569A (zh) * 2022-04-24 2022-07-29 咪咕视讯科技有限公司 画面显示方法、装置、虚拟现实设备及存储介质
CN114827569B (zh) * 2022-04-24 2023-11-10 咪咕视讯科技有限公司 画面显示方法、装置、虚拟现实设备及存储介质

Similar Documents

Publication Publication Date Title
US20160041662A1 (en) Method for changing play mode, method for changing display mode, and display apparatus and 3d image providing system using the same
US8907968B2 (en) Image rendering device, image rendering method, and image rendering program for rendering stereoscopic panoramic images
US6747610B1 (en) Stereoscopic image display apparatus capable of selectively displaying desired stereoscopic image
US6496598B1 (en) Image processing method and apparatus
JP5320524B1 (ja) ステレオ撮影装置
EP2448276B1 (fr) Procédé de fourniture de IUG, appareil d&#39;affichage et système de fourniture d&#39;images 3D l&#39;utilisant
TWI523488B (zh) 處理包含在信號中的視差資訊之方法
JP5592005B2 (ja) フレームパッキングフォーマットでアクティブスペースの知的な使用を行う方法及び装置
JP5833667B2 (ja) マルチビューシステムにおいてモノビジョンを提供する方法および装置
WO2012029298A1 (fr) Dispositif de capture d&#39;images et procédé de traitement d&#39;images
KR20180053367A (ko) 서라운드 뷰를 생성하기 위한 시스템들 및 방법들
US20130038611A1 (en) Image conversion device
JP2010522475A (ja) ２次元／３次元デジタル情報取得及び表示装置
JP2007097129A (ja) ステレオ表示装置及びその制御方法
JP2010273333A (ja) 立体映像合成装置
KR20110129903A (ko) ３ｄ 시청자 메타데이터의 전송
US8866887B2 (en) Computer graphics video synthesizing device and method, and display device
JPH1127703A (ja) 表示装置及びその制御方法
CN113891060A (zh) 自由视点视频重建方法及播放处理方法、设备及存储介质
JP2011139209A (ja) 画像処理装置および方法
WO2012132267A1 (fr) Dispositif d&#39;émission d&#39;image stéréoscopique omnidirectionnelle
WO2008122838A1 (fr) Qualité d&#39;image améliorée dans des dispositifs d&#39;affichage multi-vue stéréoscopiques
KR101987062B1 (ko) Ip를 통한 멀티카메라 비디오 분배 및 결합 시스템 및 그 방법
US20110134226A1 (en) 3d image display apparatus and method for determining 3d image thereof
CN104994370A (zh) 一种3d视频图像采集系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12765034

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12765034

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP