Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/14—Systems for two-way working
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/167—Synchronising or controlling image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/194—Transmission of image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
  • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
  • H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/239—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests
  • H04N21/2393—Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests involving handling client requests
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
  • H04N21/41407—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/16—Analogue secrecy systems; Analogue subscription systems
  • H04N7/173—Analogue secrecy systems; Analogue subscription systems with two-way working, e.g. subscriber sending a programme selection signal
  • H04N7/17309—Transmission or handling of upstream communications
  • H04N7/17318—Direct or substantially direct transmission and handling of requests

Definitions

• Image data distribution system image data transmission device thereof, and image data reception
• the present invention relates to an image data distribution system for distributing image data, and an image
• the present invention relates to a data transmitting device and an image data receiving device.
• a device that displays a stereoscopic image viewed from an arbitrary viewpoint using a multi-viewpoint image captured by a plurality of cameras has been considered.
• a display device for example, there is one as shown in FIG. This is an input device 95 for setting the left and right viewpoints, and a computer 91 1 connected to the input device 95 and creating two images viewed from the left viewpoint. And a three-dimensional display 98 connected to the computer 91 to receive image data of two images, combine them by an internal circuit, and three-dimensionally display them.
• the computer 91 has a memory 92 for storing multi-viewpoint image data, a CPU 93 for executing a process of creating an image display of two images, a stereoscopic display 98, and an input device.
• the input / output interface 94 connected to the input / output device 95 for controlling the input / output of input values and image data from the input device 95, and the CPU 93, the memory 92, and the input / output interface 94 are interconnected.
• bus 96 and including.
• the input device 95 selects a desired one of the multi-view image data stored in the memory 92 and sets the left and right viewpoints for performing the stereoscopic display.
• there is image data viewed from the left viewpoint within a multi-view image It determines whether or not the image data exists, and if it exists, outputs the image data via the input / output interface 94. If it does not exist, the image data viewed from the left viewpoint is created by performing interpolation processing using the image data of a predetermined number of images taken from the viewpoint near the left viewpoint, and the input / output interface Output via 9 4. The same applies to the right viewpoint.
• the present invention has been made in view of such a situation, and an image data distribution system capable of observing a stereoscopic image viewed from an arbitrary viewpoint even on a portable terminal or the like, an image data overnight transmission device, and an image data transmission system To provide a receiving device
• an image data distribution system, an image data transmitting device, and an image data receiving device employ the following means. That is, the image data distribution system of the present invention includes a request information receiving unit that receives client request information transmitted via a network, and a request information receiving unit that receives the request information.
• Request information analysis means for analyzing received request information
• multi-view image supply means for supplying multi-view image data
• An image generating means for generating image data of a predetermined viewpoint that matches the request information input from the multi-viewpoint image supply means, and combines a plurality of image data generated by the image generating means based on the display unit information of the request information.
• Image synthesizing means encoding means for encoding the image data synthesized by the image synthesizing means, transmitting means for transmitting the image data encoded by the encoding means to the network, Receiving means for receiving the image data, decoding means for decoding the encoded image data received by the receiving means, and displaying the image data decoded by the decoding means on the display means
• Image processing means for processing image data, display means for displaying image data processed by the image processing means, request information input means for inputting request information of the client, and request information transmitting means for transmitting request information to the network
• the image data transmitting apparatus further comprises: request information receiving means for receiving client request information transmitted via a network; and request information received by the request information receiving means.
• Image generating means for generating image data of a predetermined viewpoint matching the request information by inputting the image data from a plurality of image data generated by the image generating means.
• Image synthesizing means for synthesizing the image data synthesized by the image synthesizing means, and transmitting means for transmitting the image data encoded by the encoding means to a network. It is characterized by having.
• the image data receiving apparatus of the present invention includes: a receiving unit that receives encoded image data via a network; a decoding unit that decodes the encoded image data received by the receiving unit; Image processing means for processing the image data decoded by the means so that it can be displayed on the display means, and a display means for displaying the image data processed by the image processing means. And request information input means for inputting request information of the client, and request information transmitting means for transmitting the request information to the network.
• the image data distribution system and the image data transmitting device of the present invention have management information adding means for adding management information for enabling access to each viewpoint image data and random access to multi-view image data. It is characterized by the following. Further, the image data distribution system and the image data receiving device of the present invention are characterized in that the image data distribution system and the image data receiving device have a determination unit for determining whether the received image data is two-dimensional image data or stereoscopic image data.
• the image data distribution system, the image data transmitting device, and the image data receiving device of the present invention include information for identifying whether a transmitted or received image data is a two-dimensional image data or a three-dimensional image data. It is characterized by having an identification information adding means for adding an ID.
• the client side does not need a memory having a sufficient capacity and a CPU having a high processing capability. This has the effect that the stereoscopic image viewed from the viewer can be observed.
• FIG. 1 is a block diagram showing an embodiment of an image data distribution system according to the present invention.
• FIG. 2 is a diagram showing an arrangement of a plurality of cameras for creating multi-viewpoint image data.
• FIG. 3 is a diagram showing a left viewpoint L and a right viewpoint R of image data generated by interpolation.
• FIG. 4 is a diagram showing an area cut out from the generated image data according to the resolution of the display unit.
• Figure 5 is an image diagram of a portable terminal serving as a client.
• FIG. 6 is a diagram illustrating an example of a storage image form of multi-view image data.
• FIG. 7 is a diagram showing an example of a generated image data form.
• FIG. 8 is a diagram showing accumulation and extraction of multi-viewpoint image data.
• FIG. 9 is a flowchart illustrating a processing procedure in the server.
• FIG. 10 is a flowchart showing a processing procedure in the client.
• FIG. 11 is a diagram showing an example in which multi-view video data is encoded using MPEG-4.
• FIG. 12 is a diagram showing multi-view image data to which management information has been added.
• FIG. 13 is a diagram illustrating an example of management information.
• FIG. 14 shows a connection form between a server and a client assumed in the present embodiment.
• FIG. 15 is a flowchart showing details of the processing of the image generation unit.
• FIG. 16 shows a conventional example.
• FIG. 1 is a block diagram showing an embodiment of an image data distribution system according to the present invention.
• Server 1 image data transmitting device
• client 11 image data receiving device
• the server 1 stores the multi-view image data 2, and the client 11 sends request information to the server 1, so that an image viewed from a desired viewpoint is displayed in a stereoscopic manner on the display unit 14.
• the multi-view image data does not necessarily need to be stored (recorded) in the server 1 in advance, and may be input in real time from outside.
• the server 1 analyzes the request information transmitted from the client 11 by the request analysis unit 4 (request information analysis means (including request information reception means)), and stores the request information in the multi-view image data 2 (multi-view image supply means).
• the required image data is selected and output to the image generation unit 3 (image generation means), and the image generation unit 3 interpolates and generates the image data of the requested viewpoint (viewpoint information) and outputs the image data to the image synthesis unit 5 (image generation unit).
• the image synthesizing unit 5 synthesizes the input image data into a form suitable for encoding (a form based on the display unit information), and outputs it to the encoding unit 6 (encoding means).
• the encoding unit 6 encodes the input image data at an appropriate bit rate and transmits the encoded image data to the network 7 (transmission means).
• the client 11 receives the encoded image data (receiving means), decodes the data by the decoding unit 12 (decoding means), and sends the decoded image data to the image processing unit 13 (image processing means). Output and convert the image data into an appropriate form according to the stereoscopic display format and display it on the display unit 14 (display means).
• the client 11 also has an input unit 16 (request information input means) for changing the viewpoint, and transmits request information for viewpoint change to the network 7 via the request output unit 15 (request information transmission means). I do.
• the multi-viewpoint image data 2 is a set of image data shot using a plurality of cameras.
• the plurality of cameras are arranged so that the optical axes of the plurality of cameras converge at one point as shown in FIG. 2 (a), for example.
• the force sensors may be arranged on the circumference so that the optical axes of the force lenses are oriented toward the center of the circle. In either case, the cameras need not be arranged at equal intervals, and the distribution may be sparse and dense. Information indicating how the cameras are arranged is recorded together with the image data.
• the image generation unit 3 interpolates and generates image data of a predetermined viewpoint based on the request information from the client 11, It is used when determining whether to create the image based on the image data from the camera.
• camera image data required when generating an image of a requested viewpoint by interpolation.
• the left viewpoint L and the right viewpoint R are set as illustrated for the cameras C1 to C4.
• the image data of the left viewpoint L is generated.
• the image data of C2 and C3 are used.
• FIG. 3 shows an example in which four cameras are used, the number of cameras is not limited to four.
• the generation of intermediate viewpoint images by interpolation is a known technique. For example, see “Tsunashima et al .: Creation of Intermediate Image Data from Twin-Image Stereo Image Data Considering Occlusion-3D Image Conference '95, pp. 174-177 (1995) J and "Azuma et al .: Parallax Estimation Using Edge Information for Intermediate Image Generation, 3D Image Conference '95, pp. 190-194 (1995)” .
• the image synthesizing unit 5 extracts the image data of the required resolution.
• Fig. 4 (a) it is assumed that the size of the generated image data 21 is equal to the size of the image captured by the camera, and the resolution required for display on the client 11 is indicated by the area 22. are doing.
• FIG. 4A only a part of the generated image data is cut out.
• the display data is generated from the generated image data 23 (having the same size as the image captured by the camera). After cutting out the largest area 24 that can be cut out while keeping the spectrum, the resolution may be reduced to the required resolution.
• FIG. 5 is an image diagram of a portable terminal 50 serving as the client 11. It has a three-dimensional display 54, and has a cross key 51 for moving the viewpoint up, down, left and right, and keys 52, 5 3 for moving the viewpoint back and forth. Although not shown, a communication means for communicating with the server is also provided.
• FIG. 6 shows an example of an accumulation form of the multi-viewpoint image data 2. Since moving image data captured by multiple cameras must be synchronized in time, multiple image data C1 to C4 captured by each camera are displayed horizontally as shown in Fig. 6 (a). There is a form in which image data is stored as one image by attaching them side by side. By doing so, it is ensured that C1 to C4 included in the image data to be one image are shot at the same time, and time management becomes easy.
• the attachment method is not limited to a horizontal row as shown in FIG. 6 (a), but may be attached as shown in FIG. 6 (b).
• the multi-viewpoint image data 2 may be compressed and stored, or may be stored in an uncompressed state.
• the image data for each camera The case of compressing and storing will be described with reference to FIG.
• the image data C1 to C4 captured by each camera are input to the encoding unit 31 as shown in FIG.
• the encoding unit 31 encodes each image data and outputs information (such as a frame type and the number of generated pits) necessary for generating management information to the management information generation unit 32.
• the recording unit 33 records management information as accumulated data together with the encoded image data (management information adding means). Details of the management information and the storage form will be described later.
• the selection unit 34 selects only the necessary image data from the stored data and outputs it to the decoding unit 35 for decoding.
• Original image data image data used in the image generation unit 3
• management information recorded together with the image data is used to quickly extract necessary parts.
• FIG. 7 shows an example of an image display mode generated by the image synthesizing unit 5.
• the left-view image data L and the right-view image data R are encoded as shown in Fig. 7 (a). It is preferable to arrange the images side by side (or vertically) to synthesize and encode the image data as one image.
• Fig. 7 (a) by using image data that forms one image, synchronization between the left-viewpoint image data L and the right-viewpoint image data R is guaranteed, which simplifies time management. .
• the image data finally displayed on the display unit 14 may have left-view image data L and right-view image data R for each line as shown in FIG. 7 (c).
• Stripes are alternately lined up (Wrench wrench method Even in such a case, it is desirable that the image data to be encoded is in the form shown in FIG. 7 (a). This is because when block-based coding such as DCT is performed, image data in the form shown in Fig. 7 (c) has a weaker correlation between adjacent pixels and a higher spatial frequency, resulting in poor compression efficiency. It is.
• the same concept can be applied when the required number of viewpoints is larger than two. As shown in Fig.
• the recording means is provided before decoding (before the decoding unit 12) or after image processing (after the image processing unit).
• FIG. 9 is a flowchart showing a processing procedure in the server 1.
• a request from the client is analyzed (step S1).
• necessary image data is selected from the multi-view image data (step S2).
• the image of the requested viewpoint is generated by using it (step S3).
• image data of a size necessary for display is cut out (reduced if necessary) (step S4). Soshi Then, the clipped left viewpoint image data and the right viewpoint image data are combined (step S5).
• the synthesized image data is encoded (step S6). Then, it is output as a bit stream (step S7).
• FIG. 10 is a flowchart showing a processing procedure in the client 11.
• initialization is performed to set the viewpoint position (viewpoint information) and viewpoint-independent information (stereoscopic display format, resolution, etc. (display unit information)) in the initial state (step S11).
• the information is transmitted as a request to the server 1 (step S12).
• a bit stream (image data) satisfying the request is transmitted from the server via the network (step S13).
• it is decrypted (step S14). Since the decoded image data is not in a format that can be displayed three-dimensionally as it is as shown in Fig. 7 (a), by rearranging it, it matches the stereoscopic display format as shown in Fig. 7 (c). Format (step S15).
• step S16 it is displayed on the display unit 14 (step S16).
• step S17 it is determined whether or not to continue the display. If the display is to be continued, it is determined whether there is a request to change the viewpoint (step S18). Next, when there is a change in the viewpoint, the request is output to the server 1 again, and the process returns to the step S12. If the display is not continued in step S18, the process returns to step S13.
• FIG. 14 shows a connection form between a server and a client assumed in the present embodiment.
• Terminals A to C are clients, and are connected to the server 41 via the network 7.
• the server 41 transmits different image data to each terminal in response to a request from each terminal because the stereoscopic display format and the display resolution are different for each terminal, and the position of the viewpoint to be viewed is also different. If all of the multi-viewpoint image data stored (recorded) by the server 41 is transmitted to the network and each terminal selects and displays it, only one type of image data needs to be transmitted. In practice, it is impossible to do so because the viewpoint image data has a huge amount of information and the bandwidth of the network is limited. From this, from the client like the present invention A system in which the server sends appropriate image data according to the requirements of the above is essential in an environment where different types of terminals are connected to the network.
• FIG. 15 is a flowchart showing details of the processing of the image generating unit 3.
• the position of the viewpoint requested by the client is analyzed (step S21).
• the image data may be used as it is, and in other cases, an interpolated image is generated (step S23).
• FIG. 11 shows an example of a case where a multi-view video image is encoded by MPEG-4.
• frames to be encoded have frame numbers that are discrete such as LT f0, LT f3, LTf 5, and LT f10.
• LT f 0 is a frame to be intra-coded (I frame)
• LT fl O is a frame to be predictively encoded from a decoded frame of LT f 0 (P frame)
• LT f 3 and LT f 5 are LT f 0
• the frame is decoded as a frame (B frame) to be bidirectionally predictively encoded from the decoded frame of O, LT fl0.
• FIG. 13 shows an example of the management information added by the management information generation unit 32 in FIG. 8A.
• the encoded data of each camera image can be combined and stored together with the management information as shown in Fig. 12, but at this time, information for enabling access to each camera image data is It is management information.
• information that enables random access to the coded data at a specified time in each camera image data is included together with access to each camera image data.
• FIG. 13A shows an example of management information for accessing the encoded data of each camera image. For example, it indicates that the encoded data of the camera image C2 exists in the B2 byte from the beginning of the data in FIG. Fig. 13 (a) also shows information for accessing the encoded data at the specified time within the digital image data. Pointer is described. In the case of the encoded data of C2, it is indicated that the access table for the encoded data at the designated time is in the address P2 in the management information.
• FIG. 13B is an example of an access table to the encoded data at the specified time. The times tl, t2, t3, ⁇ may be set at equal intervals, or may be at any time intervals.
• the encoded data corresponding to the time t3 is present at the Bt 3rd byte from the beginning of the encoded image of the camera image, and the encoded data of the ⁇ frame is located at a position as far back as It 3 bytes from that position. It is shown that there is.
• the decoder wants to display from time t3, it first decodes the encoded data of the I frame in (Bt3-It3) bytes from the beginning. Then, while decoding P and B frames sequentially, counting the number of bytes decoded, and starting to display when only It 3 bytes have been decoded, the image data at the specified time t 3 is displayed. .
• (A) The sign data is packetized, and there is information indicating whether or not the header information of each packet includes the head of the I frame.
• Figure 13 (b) shows the specified time and the number of bytes up to the corresponding bucket. After accessing the bucket at the specified time t3 with the decoder, it checks whether the bucket includes the beginning of the I-frame and starts decoding and displaying from the packet containing the I-frame. Discard it).
• management information and encoded information are combined and stored, but the management information may be separated and stored as a separate file.
• the information for accessing the designated time may be included not in the management information but in the header information of the encoding image of each camera image. In this case, the third column in Fig. 13 (a) (for accessing the specified time in the camera image)
• the file name of the code image of each camera image is written in the second column of FIG. 13 (a). Access to each camera image is based on the file name.
• the image data distribution system and the image data transmitting device and the image data receiving device according to the present invention can be viewed from an arbitrary viewpoint without requiring a memory having a sufficient capacity and a CPU having a high processing capacity on the client side. It can be applied to mobile terminals that can observe 3D images.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Human Computer Interaction (AREA)
• General Engineering & Computer Science (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (1)

Family

ID=29996848

Family Applications (1)

Country Status (7)

Cited By (10)

Families Citing this family (34)

Citations (6)

Family Cites Families (17)

• 2003
  • 2003-06-30 EP EP03761837A patent/EP1519582A4/en not_active Withdrawn
  • 2003-06-30 AU AU2003244156A patent/AU2003244156A1/en not_active Abandoned
  • 2003-06-30 WO PCT/JP2003/008302 patent/WO2004004350A1/ja active Application Filing
  • 2003-06-30 US US10/519,154 patent/US7734085B2/en not_active Expired - Fee Related
  • 2003-06-30 JP JP2004517332A patent/JP4346548B2/ja not_active Expired - Fee Related
  • 2003-06-30 KR KR1020047021448A patent/KR100742674B1/ko not_active IP Right Cessation
  • 2003-06-30 CN CNB038153718A patent/CN100342733C/zh not_active Expired - Fee Related

Patent Citations (6)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events