JP2004274091A - Image data creating apparatus, image data reproducing apparatus, image data recording system, and image data recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make versatile the image data for three-dimensional display flexible, and to enable efficiently selecting an image at an arbitrary view. <P>SOLUTION: An image signal formed by continuous frames is input into an image data creating apparatus for every one frame. A control section 102 designates the presence or absence of reduction, the presence or absence of coupling and the selection of the 2D image. An image converting section 101 creates image data of the type designated in the the presence or absence of reduction, the presence or absence of coupling. A 3D information creating section 103 formats the presence or absence of reduction and the presence or absence of coupling and the selection of 2D image, and creates 3D information required to display the image as a three-dimensional image. A multiplexing section 104 converts the image data and the 3D information into a predetermined format to output them to the outside. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image data creating apparatus for adding attribute information to image data when creating image data for three-dimensional display, and an image data reproducing apparatus for reproducing the data.
[0002]
[Prior art]
Conventionally, various methods for displaying a three-dimensional image have been proposed. Among them, the one that is generally used is a “binocular type” that uses binocular parallax. That is, stereoscopic viewing is performed by preparing a left-eye image and a right-eye image having binocular parallax and projecting them independently to the left and right eyes.
[0003]
FIG. 16 is a conceptual diagram for explaining a “time division method” which is one of the typical two-lens methods.
[0004]
In this time division method, as shown in FIG. 16, a left-eye image and a right-eye image are arranged alternately every other pixel in the vertical direction, and the display of the left-eye image and the display of the right-eye image are alternately switched. Is displayed. The vertical resolution of the left-eye image and the right-eye image is 比 べ as compared to the normal two-dimensional display. The observer wears shutter-type glasses that open and close in synchronization with the display switching cycle. The shutter used here opens the left eye side and closes the right eye side when the left eye image is displayed, and closes the left eye side and opens the right eye side when the right eye image is displayed. By doing so, the left-eye image is observed only by the left eye and the right-eye image is observed only by the right eye, and stereoscopic viewing can be performed.
[0005]
FIG. 17 is a conceptual diagram for explaining “parallel barrier method” which is another typical binocular method.
[0006]
FIG. 17A is a diagram illustrating the principle of generating parallax. On the other hand, FIG. 17B is a diagram illustrating a screen displayed by the parallax barrier method.
[0007]
In FIG. 17A, an image in which the left-eye image and the right-eye image are alternately arranged every other pixel in the horizontal direction as shown in FIG. 17B is displayed on the image display panel 401, By placing a parallax barrier 402 having slits at intervals smaller than the interval between pixels at the same viewpoint on the front of the image display panel 401, the left eye image can be observed only with the left eye 403 and the right eye image can be observed only with the right eye 404. That is, stereoscopic vision can be performed.
[0008]
By the way, there is a “lenticular method” as a method for displaying an image as shown in FIG. 17B three-dimensionally, similarly to the parallax barrier method. An example of a recording data format for use in the lenticular method is disclosed in Japanese Patent Laid-Open Publication No. H11-15011.
[0009]
FIG. 18 is a conceptual diagram showing an example of such a lenticular recording data format. From the left-eye image 501 shown in FIG. 18 (a) and the right-eye image 502 shown in FIG. 18 (b), each of them is thinned out in half in the horizontal direction to form a single mixed image 503 shown in FIG. 18 (c). Make and record At the time of reproduction, a composite image as shown in FIG. 17B is created by rearranging the mixed image 503.
[0010]
Although not limited to three-dimensional images, Japanese Patent Laid-Open No. 2001-337994 stores additional information for identifying thumbnail images, and displays the additional information on the thumbnail images on a display device. A method is disclosed.
[0011]
[Patent Document 1]
JP-A-11-41627
[Patent Document 2]
JP 2001-337994 A
[0012]
[Problems to be solved by the invention]
As described above, in the conventional three-dimensional display system, recording is performed in a fixed recording data format so as to be suitable for the display method determined by the reproducing apparatus, and the recording data has to have versatility. Is not considered.
[0013]
In addition to the display method, there are various types of information necessary for three-dimensional display, such as the image thinning method and the number of viewpoints in the so-called "multi-view method". Not recorded. If you always use the same display method, there is no need to record such information, but this greatly impairs the versatility of the recorded data. For example, even when data for the parallax barrier method (or lenticular method) is recorded, the left-eye image and the right-eye image can be recorded as separate sequences, and the left and right images as shown in FIG. It is also possible to record a mixed image in which the eye image and the right eye image are arranged on the left and right half of the screen, or a composite image in which the left eye image and the right eye image are arranged every other pixel in the horizontal direction as shown in FIG. Can also be recorded. Of course, if the recording format is different, the processing method for displaying this is also different, but since it is not possible to know which format was recorded from the recorded data, when a third party got the data, There is a problem that it is not possible to determine whether the display should be performed by such processing.
[0014]
Furthermore, in the related art, it has not been considered that image data of each viewpoint is recorded independently of each other, and only an image of a desired viewpoint is simply read and reproduced.
[0015]
Further, the conventional technology does not sufficiently consider compatibility with existing devices. That is, in the system disclosed in Japanese Patent Application Laid-Open No. 2001-337994, only a display system capable of interpreting additional information is considered, and in a display system that cannot interpret the additional information, the additional information is useless.
[0016]
The present invention has been made in order to solve the above-described problems, and an object of the present invention is to provide image data for three-dimensional display with versatility and efficiently convert an image at an arbitrary viewpoint. It is an object of the present invention to provide an image data creation device that enables selection and an image data reproduction device that reproduces the data.
[0017]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides an image data creating apparatus that creates image data of a predetermined data format from a plurality of images corresponding to a plurality of viewpoints, the image data creating device corresponding to each of the plurality of images. Multiplexing means for creating a plurality of image data; and information creating means for creating first information used for displaying the plurality of images and second information for identifying the plurality of image data. The above-mentioned problem is solved by including the first information and the second information in the data format.
[0018]
Also, in the present invention, the information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information and And / or classifying the second information into common information common to the plurality of images and other individual information and creating image data including the common information and image data including only the individual information, To solve the problem.
[0019]
Also, in the present invention, the information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information and And / or classifying the second information into common information common to the plurality of images and other individual information and creating attribute data including the common information and image data including only the individual information, To solve the problem.
[0020]
Further, the present invention solves the above-mentioned problem by providing the plurality of image data with a file name, and the file name includes a common part.
[0021]
Further, the present invention solves the above-mentioned problem by allowing the plurality of image data to belong to the same directory.
[0022]
Further, according to the present invention, in an image data reproducing apparatus for reproducing a plurality of images corresponding to each of a plurality of viewpoints from image data of a predetermined data format, a plurality of image data corresponding to each of the plurality of images are inputted. And analyzing means for analyzing the data format, wherein the analyzing means analyzes first information used for displaying the plurality of images and second information for identifying the plurality of image data. Then, the above-mentioned problem is solved by reproducing a plurality of images corresponding to each of the plurality of image data using the first information and the second information.
[0023]
Further, according to the present invention, the first information and / or the second information exists in correspondence with each of the plurality of images, and includes common information common to the plurality of images and other individual information. The analysis means receives the image data including the common information and the image data including only the individual information as input, and analyzes the first information and / or the second information to solve the above-described problem. Resolve.
[0024]
Further, according to the present invention, the first information and / or the second information exists in correspondence with each of the plurality of images, and includes common information common to the plurality of images and other individual information. The analysis means receives the attribute data including the common information and the image data including only the individual information as input and analyzes the first information and / or the second information to solve the above-described problem. Resolve.
[0025]
Further, the present invention solves the above-mentioned problem by providing the plurality of image data with a file name, and the file name includes a common part.
[0026]
Further, the present invention solves the above-mentioned problem by allowing the plurality of image data to belong to the same directory.
[0027]
In addition, the present invention provides a method for generating image data of a predetermined data format from three-dimensional image display control information generated by encoding parameters for three-dimensionally displaying image data and a plurality of images corresponding to each of a plurality of viewpoints. The image data creation device for creating a three-dimensional image includes a thumbnail image creation unit, and when the image information included in the image data is a three-dimensional image, a symbol indicating that the image information is a three-dimensional image The above-described problem is solved by outputting the embedded image as a thumbnail image.
[0028]
In addition, the present invention provides a method for recording image data including image information corresponding to a plurality of viewpoints and a thumbnail image in which a symbol indicating that the image information is a three-dimensional image is embedded in a medium as a file. To solve the problem.
[0029]
Also, in the present invention, the image information corresponding to the plurality of viewpoints is obtained by combining images of a plurality of viewpoints as one image, and the thumbnail image is obtained by combining an image of one viewpoint among the plurality of viewpoints with a pixel aspect ratio. The above-described problem is solved by using an image obtained by embedding a symbol indicating that the image is a three-dimensional image in an image converted so as to be correct.
[0030]
Further, according to the present invention, the image data including image information corresponding to a plurality of viewpoints and a thumbnail image in which a symbol indicating that the image information is a three-dimensional image is recorded as a file. Solve the problem.
[0031]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0032]
FIG. 1 is a block diagram showing a configuration of an image data creation device according to an embodiment of the present invention. In FIG. 1, an image data creating apparatus 100 determines an arrangement method of an image K from an image 1 of multiple viewpoints (the number of viewpoints K, where K is an integer equal to or greater than 2), and joins them by adjoining them. Image combining unit 101 for creating an image, whether to combine image K from image 1 (with or without combination), whether to reduce image K from image 1 (with or without reduction), used for two-dimensional display The control unit 102 for specifying an image to be selected (2D selection), the number of viewpoints, and the arrangement order of the images, the presence / absence of the reduction, the presence / absence of the combination, the arrangement method of the images, the 2D selection, and the information of the number of viewpoints are formatted to obtain 3D information It comprises a 3D information creation unit 103 for creating, a means for accessing a recording medium and a communication line, and a multiplexing unit 104 for multiplexing image information and 3D information and outputting image data.
[0033]
The operation of the image data creating apparatus 100 configured as described above will be described.
[0034]
An image signal composed of continuous frames is input to the image data creating device for each frame. Here, M imaging devices for inputting an image to the image data creating device 100 are arranged in a matrix in a horizontal direction and in a vertical direction by N in a lattice shape. (Where M and N are integers of 1 or more).
[0035]
FIG. 2 shows an installation example in which the number of viewpoints is eight (a diagram in which the installed imaging device is viewed obliquely downward from behind). Here, the viewpoint numbers are assigned in order from left to right and from top to bottom. That is, the number of imaging devices 301 is 1, the number of imaging devices 302 is 2, the number of imaging devices 303 is 3, and the number of imaging devices 304 is 4. Similarly, the number of imaging devices 305 to 308 is 5 to 8. Note that an image captured by an imaging device with a viewpoint number k throughout this embodiment is referred to as an image k (where k is an integer of 1 or more).
[0036]
The control unit 102 specifies the presence / absence of reduction, the presence / absence of combination, 2D selection, the number M of viewpoints in the horizontal direction, the number N of viewpoints in the vertical direction, and the arrangement order of images. Here, the presence / absence of reduction takes one of values of “with reduction” and “no reduction”, and the presence / absence of combination takes one of a value of “no combination” and “with combination”. The 2D selection takes a viewpoint number or a value of “unspecified”. The arrangement order of the images specifies the arrangement of the images by the viewpoint number. The number of viewpoints is M = 4 and N = 2 in the example of FIG.
[0037]
When the presence or absence of the combination input from the control unit 102 indicates “no combination”, the image combination unit 101 serializes the image K from the image 1 input in parallel according to the arrangement order of the images specified by the control unit 102. Output. Alternatively, when outputting the image of each viewpoint, the viewpoint number specified in the 2D selection may be output such that it is always the first.
[0038]
When the presence or absence of the combination indicates “with connection”, the image combining unit 101 selects the arrangement method from the input image 1 to the input image K. As the arrangement method, there are three types of arrangement, a horizontal arrangement in which the multi-view images are arranged horizontally, a vertical arrangement in which the multi-view images are arranged vertically, and a lattice arrangement in which the multi-view images are arranged in both horizontal and vertical directions.
[0039]
Here, the arrangement method of the image may or may not match the installation method of the imaging device. In the case where the arrangement of the images is matched with the arrangement method of the imaging unit, the arrangement is vertical when M = 1 and N ≧ 2, horizontal when M ≧ 2 and N = 1, and lattice-like otherwise. Further, when they do not match, when M = 1, N ≧ 2 or M ≧ 2, N = 1, either the vertical arrangement or the horizontal arrangement may be selected.
[0040]
When the arrangement method is determined, the images are combined according to the arrangement order of the images input from the control unit 102. FIG. 3 shows an example of the arrangement order of the images when the images photographed by the imaging device shown in FIG. 2 are arranged in a grid pattern. In FIG. 3, one square represents an image, and the numerals are viewpoint numbers. FIG. 3A shows a case where the arrangement order of the images is designated as 1, 2, 3, 4, 5, 6, 7, and 8, and the arrangement is the same as the viewpoint number given to the imaging device. FIG. 3B shows a case where 2, 3, 1, 4, 6, 7, 5, and 8 are designated.
[0041]
When the presence or absence of reduction input from the control unit 102 indicates “with reduction”, the input image of each viewpoint is reduced. The reduction ratio at this time is not fixed, but is determined according to the number of viewpoints. That is, the image is reduced to 1 / M in the horizontal direction and 1 / N in the vertical direction.
[0042]
FIGS. 4 and 5 show examples of the result of combining by the image combining unit 101 at this time. FIG. 4 shows a case where two viewpoints using only viewpoint numbers 1 and 2 are used, for example, among the imaging devices in FIG. 2, and FIG. 5 shows a case where all the imaging devices are used.
[0043]
4A shows the case of “no reduction” and “no combination”, FIG. 4B shows the case of “no reduction” and “combination (horizontal arrangement)”, and FIG. FIG. 4D shows the case of “without reduction” and “without connection”, and FIG. 4E shows the case of “with reduction” and “with connection (horizontal arrangement)”. 4 (f) shows the case of "with reduction" and the case of "with combination (vertical arrangement)". Here, in the case of “with reduction”, the resolution in the horizontal direction or the vertical direction is reduced to に よ り by thinning pixels or the like.
[0044]
FIG. 5 shows an example of a lattice arrangement of multi-viewpoint images. When the presence or absence of reduction indicates "no reduction", the result is as shown in FIG. Here, H and V are the number of horizontal pixels and the number of vertical lines of each viewpoint image before reduction, respectively. When the presence or absence of reduction indicates "with reduction", the result is as shown in FIG. The reduction ratio is １ ／ in the horizontal direction and に in the vertical direction. The size of the image after reduction is H pixels horizontally and V lines vertically, which is the same as the image size of each viewpoint before reduction. The arrangement of the images matches the installation method of the imaging device, and four images are arranged in the horizontal direction and two images are arranged in the vertical direction. Here, the numbers attached to the images indicate viewpoint numbers. The images are arranged in ascending order of viewpoint numbers. The upper left is an image photographed by the imaging unit 301 (viewpoint number 1), and the lower right is an image photographed by the imaging unit 308 (viewpoint number 8).
[0045]
Here, the reduction ratios in the horizontal and vertical directions have been described as fixed, but these may be variable. If variable, the reduction ratio is recorded in the 3D information. When the presence or absence of the combination indicates “no combination”, the designation may be performed for each image of each viewpoint.
[0046]
The 3D information creating unit 103 creates 3D information by formatting the presence or absence of reduction, the presence or absence of combination, the selection of 2D, the number of viewpoints in the horizontal direction and the number of viewpoints in the vertical direction, the arrangement order and arrangement method of images.
[0047]
FIG. 6 shows an example of the 3D information at this time. Here, the image order indicates that the arrangement of the images is “order of viewpoint numbers” or “arbitrary order”. Thereafter, a plurality of viewpoint numbers are recorded. These viewpoint numbers indicate the arrangement of the images in the combined image when the presence or absence of the combination indicates “with combination”. In the example of FIG. 3B, the first viewpoint number is 2, and in the following order, 3, 1, 4, 6, 7, 5, and 8. When the presence or absence of the combination indicates “no combination”, it indicates the order of the multiplexed image information. When the image order indicates “viewpoint number order”, these view numbers may be omitted. When creating the 3D information, the set value may be used as it is, or may be encoded by fixed-length encoding or variable-length encoding.
[0048]
The multiplexing unit 104 converts the image information, the 3D information, and the management information into a predetermined format and outputs the format to the outside. When the images are not combined, the output order from the image combining unit 101 is the arrangement order of the images specified by the control unit 102 as described above. It becomes. Although not shown in FIG. 1, when multiplexing voice and music, those data are also multiplexed by the multiplexing unit 104.
[0049]
Here, the encoded image information may be input to the multiplexing unit 104. FIG. 19 shows the configuration of the image data creating apparatus 110 in such a case. The image data creation device 110 differs from the image data creation device 100 in FIG.
[0050]
The output destination of the multiplexing unit 104 is connected to a recording device such as an IC memory, a magneto-optical disk, a magnetic tape, or a hard disk, or a communication device such as a LAN or a modem. Here, it is assumed that an IC memory is connected to the multiplexing unit 104. Hereinafter, the recording format in this case will be described.
[0051]
Generally, when an IC memory is used as a recording medium, a file system such as a FAT (File Allocation Table) is constructed on the IC memory, and data is recorded as a file. The file format used here may be an existing format or a newly defined unique format.
[0052]
FIG. 7 is a diagram showing a file format for recording image data. In FIG. 7, it is assumed that data is recorded in a file in order from the top to the bottom of the figure. FIG. 7A shows an example in which an existing format is used, and FIG. 7B shows an example in which a new format is used.
[0053]
When an existing format is used, the 3D information is generally recorded as a part of the existing header using a mechanism for expanding a header prepared in the existing format. Here, the extended header is called an extension header. For example, the file header indicates an application data segment in JPEG, defines a new application data segment, and records 3D information. In MPEG-4, a file header indicates a Visual Object Sequence or / and a Video Object Layer, and 3D information is recorded therein as user data.
[0054]
When an existing format is used, a commonly used extension is used as it is. For example, in the case of a JPEG file, generally. If the extension “.jpg” is an MPEG file, generally the extension is “.jpg”. mpg or .mpg. When the extension “mp4” is WMV (Windows (R) Media Video), the extension is generally .mp4. The extension wmv is used. By doing so, even a conventional playback device having no three-dimensional image display function can recognize a file in an existing format and display it as a two-dimensional image.
[0055]
On the other hand, when a new format is used, 3D information is recorded at the head of the file as shown in FIG. 7B, for example. In addition, a unique extension that can be distinguished from an existing format file is attached so that the file is a new format file. The management information shown in FIGS. 7A and 7B is used for recording information that is not directly related to a three-dimensional image, such as a creation date and a creator.
[0056]
First, how to store a multi-viewpoint image when the presence or absence of combination indicates “no combination” will be described. When the existing format shown in FIG. 7A is used as the file format, a plurality of multi-view images are separately recorded in the image information area of FIG. 7A. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. FIG. 8 shows a storage example at this time. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. At this time, each frame may be independently encoded as in Motion JPEG, or the difference may be encoded using inter-frame prediction as in MPEG-4.
[0057]
A case where a new format shown in FIG. 7B is used as the file format will be described. In the new format, an existing format (JPEG, bitmap, etc.) may be used for the file header and image information in FIG. 7B, or an entirely new original format may be used. Therefore, in order to distinguish between these formats, type information (called an image type) is recorded in the 3D information.
[0058]
Regarding the method of recording image data, a plurality of image data of multiple viewpoints are recorded in the image information area of FIG. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. FIG. 9 shows an example of storage and an example of 3D information when the presence or absence of the combination indicates “no connection”. The K file headers and the image information in FIG. 9 are in a format that can be recognized as a file of an existing format by itself. In other words, taking a bitmap file as an example, images 1 to K are recorded as independent bitmap files, and 3D information, management information, and a bitmap file of image K are sequentially connected from the bitmap file of image 1 It has become a format.
[0059]
When the presence / absence of combination indicates “no combination”, multi-view image information may be recorded as separate image data in a file. At this time, the image information of each viewpoint is recorded in the format shown in FIG. 7A in the case of the existing format, and in the format shown in FIG. 7B in the case of the new format. When recording a file, only an existing format may be used, only a new format may be used, or both may be used in combination. FIG. 10 shows an example in the case of using the existing format. Since the number of viewpoints is K, K files are created.
[0060]
At this time, the 3D information creation unit 103 creates 3D information for the number of viewpoints. Here, since one image information is recorded in one file, the image order is omitted, and a viewpoint number is recorded to indicate which viewpoint number each file corresponds to. FIG. 11 shows an example of the 3D information at this time.
[0061]
FIG. 12 is a diagram illustrating an example of the 3D information when the number of viewpoints is two. Here, it is assumed that viewpoint numbers 1 and 2 among the imaging devices shown in FIG. 2 are used as the imaging devices. Since the number of viewpoints is two, two 3D information shown in FIGS. 12A and 12B are created. The left side connected by “=” indicates the item of 3D information, and the right side indicates the set value. Among the items, the same values are recorded in the arrangement method, the number of viewpoints in the horizontal direction, the number of viewpoints in the vertical direction, and the 2D selection in FIGS. 12A and 12B. The other items are different. FIG. 12A shows that the image of viewpoint number 1 is not reduced and FIG. 12B shows that the image of viewpoint number 2 is reduced. I have.
[0062]
Furthermore, when recording multi-view image information as separate files, it is necessary to identify a file of a corresponding viewpoint among multi-view images captured by the same imaging device from a plurality of recorded files. is there. Here, for example, in the multiplexing unit 104 of FIG. 1, information for identifying a file of each viewpoint image captured by the same imaging device may be recorded in the management information described above. FIG. 13 is a diagram showing an example of the management information at this time, in which information as to what name the file of the multi-view image is recorded on the recording medium is recorded. In FIG. 13, the file configuration takes values of “separation” and “integration”. In the case of “separation”, it indicates that each viewpoint image is recorded as a separate file. Is recorded in one file. The viewpoint number and the file name are recorded in association with each other so that the file name of the corresponding image data can be known from the viewpoint number.
[0063]
Alternatively, a file name may be given according to a predetermined naming rule so that the set of multi-view images captured by the imaging device is the same. For example, in the case of the number of viewpoints 2 described above, one set of file names of image 1 and image 2 is “stereo1_1.jpg” and “stereo1_2.jpg”, respectively, and another set of file names is “stereo2_1.jpg”. , "Stereo2_2.jpg".
[0064]
By the way, the 3D information or management information recorded in each file has an overlapping part. For example, in the example of 3D information shown in FIG. 12, the same information is shown except for the presence or absence of reduction and the viewpoint number. In the example of the management information shown in FIG. 13, the contents of the management information are common to all the files.
[0065]
Therefore, the common information (common information) may be recorded in only one file, and only the information (individual information) unique to the image data may be recorded in another file. An example at this time is shown in FIG. FIG. 14A shows a file in which individual information and common information are recorded, and FIG. 14B shows a file in which only individual information is recorded. In this example, the file shown in FIG. 14A is recorded in the existing format, the file shown in FIG. 14B is recorded in the new format, and the file in which the common information is recorded becomes only the other individual information depending on the file extension. The file can be identified from the recorded file. Further, as shown in FIG. 14B, the file name of the file in which the common information is recorded may be recorded as the individual management information. This makes it easier to specify a file in which common information is recorded from a file in which only individual information is recorded.
[0066]
It should be noted that, in addition to the above, a method for identifying these files is to use the above-mentioned naming rule so that the image file of each viewpoint can be identified, and a specific viewpoint number such as a viewpoint number designated by 2D selection can be used. The common information may be recorded in the image file.
[0067]
Alternatively, overlapping information may be combined into one to create a management file, and each image file may record only unique information. It is assumed that a unique extension different from the image file is used for the management file.
[0068]
Further, in a file system such as the FAT described above, a directory is used to manage files collectively. One set of the created image files of each viewpoint (the management file may be included if it exists) may be recorded in the same directory.
[0069]
When the presence or absence of the combination indicates “with combination”, the image information of the image combined into one is recorded in the image information area of FIGS. 7A and 7B. In the case of a moving image, a plurality of images obtained by combining corresponding frames of a multi-viewpoint image are recorded.
[0070]
In the above-described embodiment, the viewpoint number is fixed and the arrangement order of the images can be changed arbitrarily. However, the image arrangement order is fixed and the viewpoint number is changed arbitrarily. It does not matter. Further, the method of installing the image pickup device is not limited to the lattice-like arrangement, but may be any arrangement. In this case, a reference imaging device (a viewpoint number is set to 1) is selected, and the position is represented by a coordinate system using this as the origin. In the 3D information, the position coordinates of the imaging device of each viewpoint are recorded in the order of the viewpoint numbers.
[0071]
Further, in the above embodiment, in the case of a file of a new format, 3D information and management information are recorded at the head of the file. However, the file of the new format is not limited to this, May be after the file header or after the image information, or may be the same as the existing format shown in FIG.
[0072]
Subsequently, a reproducing device for displaying the image data created by the image data creating device 100 as a three-dimensional image will be described.
[0073]
FIG. 15 is a block diagram illustrating a configuration of an image data reproducing device according to an embodiment of the present invention. 15, the image data reproducing apparatus 200 includes a demultiplexing unit 201, a 3D information analysis unit 202, and an image conversion unit 203.
[0074]
The demultiplexing unit 201 reads image data multiplexed into a predetermined format from a recording device or a communication device, and separates the image data into image information, 3D information, and management information. Although not shown in FIG. 15, when voice and music are multiplexed, those data are also separated by the demultiplexer 201.
[0075]
The 3D information analysis unit 202 analyzes the 3D information in a predetermined format and extracts a set value for each item.
[0076]
The image conversion unit 203 is connected to display devices having different display formats, such as a two-dimensional display device using a normal cathode ray tube or a liquid crystal panel, a three-dimensional display device using a lenticular method, a parallax barrier method, a time division method, and the like. You.
[0077]
The operation of the image data reproducing device 200 configured as described above will be described. Here, it is assumed that an IC memory is connected to the demultiplexing unit 201. As described above, the image file in the existing format and the new format and the management file are recorded in the IC memory. The distinction between the image file and the management file can be made by the file extension. Here, it is assumed that the user selects one image file or one management file by a designation unit (not shown).
[0078]
First, a case where the selected file is an image file will be described. In this case, the existing format and the new format can be distinguished from each other by the extension of the file. Therefore, when the file to be reproduced is a file of the existing format shown in FIG. 3D information is read from the extended area of. In the case of the new format shown in FIG. 7B, 3D information is read from the beginning of the file.
[0079]
The 3D information analysis unit 202 analyzes the 3D information and extracts setting values such as presence / absence of combination, presence / absence of reduction, number of viewpoints, arrangement method, and 2D selection. Further, a viewpoint number of an image to be displayed as a three-dimensional image is determined. In order to display multi-viewpoint image data as a three-dimensional image, two viewpoints having parallax may be selected from the multi-viewpoint images and used as a left-eye image and a right-eye image. For example, in the case of the image data recorded by the photographing device of FIG. 2, the image data can be selected from 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4, etc. . In addition, if the image is rotated by 90 ° at the time of display, a three-dimensional image can be displayed even when a set of 1, 5, 2, 6, 3, 7, and 4 and 8 is selected.
[0080]
When the presence / absence of combination indicates “no combination”, only the image information of one viewpoint is recorded in the selected file as shown in FIG. 10, or each of the viewpoints not combined as shown in FIG. Either image information is recorded.
[0081]
In order to distinguish them, as a result of analyzing the 3D information, it is sufficient to check whether or not the image order information is included in the 3D information. If the image order is included in the 3D information, it is the latter. An arbitrary combination of viewpoints that can be stereoscopically viewed is selected from the first viewpoint number to the K-th viewpoint number recorded in the 3D information. Otherwise, the former is selected, and a viewpoint number that enables stereoscopic viewing is selected by combining with the viewpoint number i recorded in the 3D information (where i is an integer of 1 or more). The selected viewpoint number is output to demultiplexing section 201.
[0082]
The demultiplexing unit 201 reads out the image information of the input viewpoint number from the file and outputs it to the image conversion unit 203. When the image information of the input viewpoint number is not recorded in the file, the file in which the image information is recorded is found out from the management information and the naming rule as described above and read out.
[0083]
Alternatively, the file configuration in the management information may be referred to discriminate whether only one viewpoint or all viewpoints are recorded in the file. If the file configuration is “separated”, image information of one viewpoint is recorded, and if the file configuration is “integrated”, image information of all viewpoints is recorded.
[0084]
On the other hand, when the presence or absence of the combination is “with combination”, since only one combined image is recorded in the file, an arbitrary viewpoint number can be selected. In this case, the viewpoint number is output to the image conversion unit 203.
[0085]
Here, when the image information is encoded, decoding is performed after demultiplexing. FIG. 20 shows the configuration of the image data reproducing device 210 in such a case. The image data reproducing device 210 is different from the image data reproducing device 200 in FIG.
[0086]
The image conversion unit 203 converts the image information separated by the demultiplexing unit 201 according to the presence / absence of connection, presence / absence of reduction, number of viewpoints, arrangement method, 2D selection, and viewpoint number input by the 3D information analysis unit 202. Convert to display format. At this time, if the presence or absence of the combination indicates “no combination”, the conversion is performed when the image information specified by the viewpoint number is completed. If the image information is not available due to the file being deleted or the like, the image of the viewpoint number closest to the arrangement may be used instead, or two-dimensional display may be performed. When the presence or absence of the combination is “with combination”, the image of the viewpoint specified by the viewpoint number is cut out from the combined image and then converted.
[0087]
The conversion into the display format is performed. For example, when a display device of the parallax barrier system is connected to the image data reproducing device 200, the above-described format of FIG. The left and right images need only be arranged every other pixel in the horizontal direction. In the case of FIG. 4B, the left and right images are thinned out to 1/2 in the horizontal direction before being arranged every other pixel in the horizontal direction. In any case, since the 3D information is added to the image data, the image data can be converted into a display format suitable for the display device.
[0088]
When coding image information, taking the parallax barrier method as an example, even if an image to be displayed is as shown in FIG. 17B, an image having an arrangement as shown in FIG. The coding efficiency is greatly improved. This is because the correlation between adjacent pixels is higher in the state of FIG. 4E than in the state of FIG. 17B. In order to use the encoded image of FIG. 4E in the parallax barrier method, the image data reproducing device 210 of FIG. 20 decodes the image by the decoding unit 204, and then the image conversion unit 203 decodes the image as shown in FIG. What should I do?
[0089]
When the display is switched to the two-dimensional display during the display of the three-dimensional image, the image of the viewpoint number designated by the 2D selection is displayed. At this time, when the presence / absence of reduction is “no reduction”, the image is enlarged and displayed twice if “reduced”. However, when the image is not being displayed, any one of the displayed images may be selected and displayed. The selection method is used for, for example, the smallest viewpoint number, the largest viewpoint number, the arrangement closest to the viewpoint number specified by the 2D selection in the arrangement of the imaging device and the arrangement of the combined image, and the left eye image. And those used for the right eye image, and are not particularly limited here. When the 2D selection does not specify an image to be used for two-dimensional display, an image to be displayed is selected by a predetermined method.
[0090]
When a two-dimensional display device is connected, an image specified by 2D selection is displayed. The display method is the same as the two-dimensional display in the three-dimensional display device.
[0091]
When the file selected by the user is a management file, the presence / absence of combination recorded in the 3D information indicates “no combination”, but any combination of stereoscopically viewable viewpoint numbers may be selected. Just fine. The operation from the reading of the image data from the image file to the conversion to the display format is the same as described above, and the description is omitted here.
[0092]
Further, in the above-described embodiment, a case has been described in which the viewpoint number and the file name are recorded in the management information. That is, in the image data creating apparatus 100, an identification number indicating that the image information is a set of multi-view images captured by the same image capturing apparatus is recorded in the common information and the individual information, and the image data reproducing apparatus 200 , The image information is read only when the file name and the identification number match. As the identification number, one number may be assigned to a set of multi-view images, or a different number may be assigned to each viewpoint. By doing so, it is possible to prevent malfunction due to falsification of the file name.
[0093]
As described above, in a conventional reproducing apparatus which does not have a function of displaying a three-dimensional image by uniformly handling data created by various three-dimensional image shooting methods, a two-dimensional image can be displayed normally. Since it can be used, versatility can be provided.
[0094]
By the way, there is a case where a thumbnail image is recorded in an image file as shown in FIG. In the DCF (Design rule for Camera File system) standard formulated to ensure the compatibility of image files, even if image information cannot be reproduced, in order to maintain the minimum reproduction compatibility, thumbnail images must be reproduced. Require storage. Although the DCF standard does not specify the number of pixels of image information, the number of pixels of a thumbnail image is limited to one type of 160 pixels × 120 pixels.
[0095]
In order to record image information for three-dimensional display in a file based on such a concept, a form as shown in FIG. 21B can be considered. That is, it is assumed that 3D information is added to the image file in the format of FIG. 21A, and the image information in FIG. 21B is a 3D image.
[0096]
By the way, any image data reproducing apparatus that can interpret 3D information as described so far can properly process a file as shown in FIG. 21B. Consider the image data reproducing apparatus of FIG. It is assumed that neither the 3D information analysis unit 202 nor the image conversion unit 203 shown in FIG. Since such an image data reproducing apparatus cannot interpret 3D information, it is not known whether the image information stored in this file is a 2D image or a 3D image, and it is temporarily determined that the image information is a 3D image by some means. Even if the image information is not provided, the image information cannot be correctly reproduced.
[0097]
Even in such a case, it can be expected that only the thumbnail image can be displayed if the thumbnail image is recorded in accordance with the concept such as the DCF standard. Here, consider a thumbnail image. Now, assuming that, for example, an image as shown in FIG. 4E is stored as the image information in FIG. 21B, first, this image is directly reduced to 160 pixels × 120 pixels as shown in FIG. Such an image is recorded as a thumbnail image. When the image in FIG. 22A is used as a thumbnail, images similar to the left and right are arranged side by side, and the size of the subject is halved in the horizontal direction. For users who do not know that they are for 3D display, there is a sense of incompatibility. On the other hand, there is a method of recording, as a thumbnail image, an image as shown in FIG. 22B obtained by enlarging one image in FIG. 4E twice in the horizontal direction. In this case, the image on the left may be used, the image on the right may be used, or the image specified by “2D selection” in the 3D information may be used. Although the image in FIG. 22B has no problem as a thumbnail, there is a problem in that it is difficult to distinguish whether the image is a 2D file or a 3D file even when looking at the image. That is, although a 3D image is actually stored as image information even though the thumbnail image in FIG. 22B is visible, correct image reproduction cannot be performed even if an attempt is made to reproduce the image information. In an image data reproducing apparatus that cannot perform three-dimensional display, it is very convenient if such a file can be identified in advance.
[0098]
Therefore, in the present invention, a thumbnail image in which a symbol indicating that this file stores a 3D image is embedded is recorded as a thumbnail image. For example, as shown in FIG. 23A, a thumbnail image may be obtained by superimposing a pictogram “3D” on the lower right of an image as shown in FIG. 22B, or as shown in FIG. Thumbnail images may be obtained by embedding the letters “3D” in the center of the screen in a watermark-like manner. Also, as shown in FIG. 23 (c), even if an image as shown in FIG. 22 (a) combined with the character "3D" is recorded as a thumbnail image, at least this file is a file containing a 3D image. I can recognize that.
[0099]
The symbol itself may be text, a mark, or a specific image as long as it represents 3D. Further, there is no limitation on the position of the symbol in the thumbnail image. However, in any case, it must be recorded as a part of the thumbnail image. Conversely, these symbols are not recorded separately from the thumbnail images.
[0100]
This is the important point. If the image data reproducing apparatus is capable of interpreting the 3D information, the thumbnail image as shown in FIG. 22B and the read 3D information are combined before display to generate a thumbnail image, and as shown in FIG. Thumbnail display is possible. This can be said to be an application of the method disclosed in JP-A-2001-337994. However, such a method cannot avoid the above-mentioned inconvenience in the thumbnail display in the image data reproducing apparatus which cannot interpret the 3D information. In order to display a meaningful thumbnail in an image data reproducing apparatus that cannot interpret 3D information, it is necessary to previously record an image as shown in FIG. 23 as a thumbnail image in a file. In this way, the image data reproducing apparatus can check the contents of the image by simply reproducing the thumbnail image in the same manner as the 2D image file, even if the file is a 3D image file. 3D image files can be distinguished.
[0101]
FIG. 24 shows an example of an image data creating apparatus that records such thumbnail images in a file. The image data creating apparatus 120 of FIG. 24 includes a thumbnail image creating unit 106, and multiplexes the thumbnail data along with encoded data (or uncompressed image information) and 3D information in the multiplexing unit 104 to create a file. The output destination of the file may be a recording device such as an IC memory, a magneto-optical disk, a magnetic tape, or a hard disk, or a communication device such as a LAN or a modem. In the DCF standard, the size of a thumbnail image is defined as 160 pixels × 120 pixels, but a general thumbnail image is not limited to this size.
[0102]
【The invention's effect】
According to the present invention, in an image data creating apparatus that creates image data of a predetermined data format from a plurality of images corresponding to each of a plurality of viewpoints, a plurality of image data corresponding to each of the plurality of images is created. Multiplexing means, and information creating means for creating first information used for displaying the plurality of images and second information for identifying the plurality of image data, wherein the data format is By including the first information and the second information, it is advantageous that image data for three-dimensional display can be provided with versatility and an image at an arbitrary viewpoint can be efficiently selected. The effect is obtained.
[0103]
According to the present invention, the information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information. By classifying the information and / or the second information into common information common to the plurality of images and other individual information, image data including the common information and image data including only the individual information are created. Advantageously, image data for three-dimensional display is made versatile, and an image at an arbitrary viewpoint can be efficiently selected.
[0104]
According to the present invention, the information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information. By classifying the information and / or the second information into common information common to the plurality of images and other individual information, creating attribute data including the common information and image data including only the individual information. Advantageously, image data for three-dimensional display is made versatile, and an image at an arbitrary viewpoint can be efficiently selected.
[0105]
Further, according to the present invention, the plurality of image data has a file name, and the file name includes a common part, so that the image data for three-dimensional display can have general versatility and can have an arbitrary viewpoint. This has the advantageous effect that the image can be efficiently selected.
[0106]
Further, according to the present invention, the plurality of image data belong to the same directory, so that the image data for three-dimensional display has versatility, and an image at an arbitrary viewpoint can be efficiently selected. The advantageous effect that it can be obtained is obtained.
[0107]
Further, according to the present invention, in an image data reproducing apparatus that reproduces a plurality of images corresponding to each of a plurality of viewpoints from image data of a predetermined data format, a plurality of image data corresponding to each of the plurality of images are provided. And analyzing means for analyzing the data format, wherein the analyzing means includes first information used for displaying the plurality of images and second information for identifying the plurality of image data. Is analyzed to reproduce a plurality of images corresponding to each of the plurality of image data using the first information and the second information, thereby reproducing image data for three-dimensional display. , An advantageous effect that an image at an arbitrary viewpoint can be efficiently selected.
[0108]
Further, according to the present invention, the first information and / or the second information exist in correspondence with each of the plurality of images, and include common information common to the plurality of images and other individual information. Information, and the analysis means inputs image data including the common information and image data including only the individual information, and analyzes the first information and / or the second information to obtain a three-dimensional image. When reproducing image data for display, an advantageous effect that an image at an arbitrary viewpoint can be efficiently selected can be obtained.
[0109]
Further, according to the present invention, the first information and / or the second information exist in correspondence with each of the plurality of images, and include common information common to the plurality of images and other individual information. Information, and the analysis means inputs attribute data including the common information and image data including only the individual information, and analyzes the first information and / or the second information, thereby obtaining a three-dimensional image. When reproducing image data for display, an advantageous effect that an image at an arbitrary viewpoint can be efficiently selected can be obtained.
[0110]
Also, according to the present invention, the plurality of image data has a file name, and the file name includes a common part, so that when reproducing image data for three-dimensional display, an image of an arbitrary viewpoint Has an advantageous effect that can be efficiently selected.
[0111]
Further, according to the present invention, since the plurality of image data belong to the same directory, it is possible to efficiently select an image at an arbitrary viewpoint when reproducing image data for three-dimensional display. An advantageous effect is obtained.
[0112]
Further, according to the present invention, an appropriate thumbnail display can be performed even in an existing image data reproducing apparatus that does not support three-dimensional display, and a 2D image file and a 3D image file can be distinguished.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of an image data creation device according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of installation of an imaging device in a case of multiple viewpoints.
FIG. 3 is a diagram illustrating an example of how to assign viewpoint numbers.
FIG. 4 is a diagram illustrating an example of a combination when the number of viewpoints is two;
FIG. 5 is a diagram showing a lattice arrangement of multi-viewpoint images.
FIG. 6 is a diagram showing a format example of 3D information.
FIG. 7 is a diagram showing a file format of image data.
FIG. 8 is a diagram showing an example of a format when image data is stored in a file of an existing format.
FIG. 9 is a diagram illustrating an example of a format when image data is stored in a file of a new format.
FIG. 10 is a diagram showing a storage example of image data when recording multi-view image data in separate files.
FIG. 11 is a diagram showing a format example of 3D information.
FIG. 12 is a diagram illustrating an example of set values of 3D information.
FIG. 13 is a diagram illustrating a format example of management information.
FIG. 14 is a diagram illustrating an example when recording multi-viewpoint image data in separate files.
FIG. 15 is a diagram showing a configuration of an image data reproducing device according to an embodiment of the present invention.
FIG. 16 is a diagram showing a display format of an image in a time division system.
FIG. 17 is a diagram for explaining the concept of the parallax barrier method.
FIG. 18 is a diagram for explaining a display format of an image in the parallax barrier method.
FIG. 19 is a diagram showing a configuration of an image data creation device according to another embodiment of the present invention.
FIG. 20 is a diagram showing a configuration of an image data reproducing device according to another embodiment of the present invention.
FIG. 21 is a diagram showing an image file format for recording thumbnail image data for three-dimensional display.
FIG. 22 is a diagram showing a thumbnail image reduced to 160 pixels × 120 pixels.
FIG. 23 is a diagram showing a thumbnail image in which a symbol indicating that data of a 3D image is stored is embedded.
FIG. 24 is a diagram showing a configuration of an image data creating apparatus for recording thumbnail images in a file according to another embodiment of the present invention.
[Explanation of symbols]
100 Image data creation device
101 Image combining unit
102 control unit
103 3D information creation unit
104 Multiplexer
200 image data reproducing apparatus
201 Demultiplexer
202 3D information analysis unit
203 Image conversion unit
301 to 308 imaging device
401 Image display panel
402 Parallax Barrier
403 Left eye
404 right eye
501 Left eye image
502 Right eye image

Claims (14)

In an image data creating apparatus that creates image data of a predetermined data format from a plurality of images corresponding to each of a plurality of viewpoints,
Multiplexing means for creating a plurality of image data corresponding to each of the plurality of images,
An information creating unit that creates first information used for displaying the plurality of images and second information for identifying the plurality of image data,
The image data creation device according to claim 1, wherein the data format includes the first information and the second information. The image data creation device according to claim 1,
The information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information and / or the second information. Wherein the image data is classified into common information common to the plurality of images and other individual information, and image data including the common information and image data including only the individual information are generated. The image data creation device according to claim 1,
The information creating means creates a plurality of the first information and / or the second information in correspondence with each of the plurality of images, and creates a plurality of the created first information and / or the second information. Wherein the image data is classified into common information common to the plurality of images and other individual information, and image data including only the individual information and the attribute data including the common information is generated. The image data creation device according to claim 1,
The plurality of image data have a file name, and the file name includes a common part. The image data creation device according to claim 1,
An image data creating apparatus, wherein the plurality of image data belong to the same directory. An image data reproducing apparatus that reproduces a plurality of images corresponding to each of a plurality of viewpoints from image data of a predetermined data format,
Inputting a plurality of image data corresponding to each of the plurality of images, comprising an analysis unit for analyzing the data format,
The analysis means analyzes first information used for displaying the plurality of images and second information for identifying the plurality of image data,
An image data reproducing apparatus for reproducing a plurality of images corresponding to each of the plurality of image data using the first information and the second information. The image data reproducing device according to claim 6,
The first information and / or the second information is provided corresponding to each of the plurality of images, and includes common information common to the plurality of images and other individual information,
The image data reproducing apparatus, wherein the analysis means receives image data including the common information and image data including only the individual information as input, and analyzes the first information and / or the second information. . The image data reproducing device according to claim 6,
The first information and / or the second information is provided corresponding to each of the plurality of images, and includes common information common to the plurality of images and other individual information,
The image data reproducing apparatus, wherein the analysis unit receives, as input, attribute data including the common information and image data including only the individual information, and analyzes the first information and / or the second information. . The image data reproducing device according to claim 6,
The image data reproducing device, wherein the plurality of image data has a file name, and the file name includes a common part. The image data reproducing device according to claim 6,
An image data reproducing apparatus, wherein the plurality of image data belong to a same directory. Image data generation for generating image data of a predetermined data format from three-dimensional image display control information generated by encoding parameters for displaying image data three-dimensionally and a plurality of images corresponding to each of a plurality of viewpoints In the device,
A thumbnail image creation unit that outputs, when the image information included in the image data is a three-dimensional image, an image in which a symbol indicating that the image data is a three-dimensional image is embedded as a thumbnail image An image data creation device characterized by performing the following. An image data recording method, wherein image data including image information corresponding to a plurality of viewpoints and a thumbnail image in which a symbol indicating that the image information is a three-dimensional image is embedded is recorded on a medium as a file. The image data recording method according to claim 12,
The image information corresponding to the plurality of viewpoints is obtained by combining images of a plurality of viewpoints as one image, and the thumbnail image is obtained by converting an image of one of the plurality of viewpoints so that a pixel aspect ratio is correct. An image data recording method, characterized in that the image is an image in which a symbol indicating a three-dimensional image is embedded. An image data recording medium, wherein image data including image information corresponding to a plurality of viewpoints and a thumbnail image in which a symbol indicating that the image information is a three-dimensional image is embedded is recorded as a file.

Images