JP4251907B2 - Image data creation device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image data creation apparatus that attaches attribute information to image data when creating image data for three-dimensional display, and an image data reproduction apparatus that reproduces the data.
[0002]
[Prior art]
Conventionally, various methods for displaying a three-dimensional image have been proposed. Among them, what is generally used is a so-called “binocular type” that uses binocular parallax. That is, a left-eye image and a right-eye image having binocular parallax are prepared, and stereoscopic vision is performed by projecting them to the left and right eyes independently.
[0003]
FIG. 16 is a conceptual diagram for explaining a “time division method”, which is one of the typical two-lens methods.
[0004]
In this time division method, as shown in FIG. 16, the left eye image and the right eye image are alternately arranged every other pixel in the vertical direction, and the display of the left eye image and the display of the right eye image are alternately switched. Is displayed. The left-eye image and the right-eye image have a vertical resolution that is ½ that of normal two-dimensional display. The observer wears shutter-type glasses that open and close in synchronization with the display switching cycle. The shutter used here opens the left eye side and closes the right eye side when the left eye image is displayed, and closes the left eye side and opens the right eye side when the right eye image is displayed. By doing so, the left eye image is observed only by the left eye, and the right eye image is observed only by the right eye, and stereoscopic viewing can be performed.
[0005]
FIG. 17 is a conceptual diagram for explaining a “parallax barrier method” which is another representative method of the two-lens method.
[0006]
FIG. 17A is a diagram illustrating the principle of generating parallax. On the other hand, FIG. 17B is a diagram showing a screen displayed by the parallax barrier method.
[0007]
In FIG. 17A, an image in which the left eye image and the right eye image as shown in FIG. 17B are alternately arranged every other pixel in the horizontal direction is displayed on the image display panel 401. By placing a parallax barrier 402 having slits at intervals smaller than the interval between pixels of the same viewpoint on the front surface of the image display panel 401, the left eye image is observed only by the left eye 403, and the right eye image is observed only by the right eye 404. That is, stereoscopic viewing can be performed.
[0008]
Incidentally, as in the parallax barrier method, there is a “lenticular method” as a method for three-dimensionally displaying an image as shown in FIG. An example of a recording data format for use in this lenticular method is disclosed in Patent Document 1 described later.
[0009]
FIG. 18 is a conceptual diagram showing an example of such a lenticular recording data format. From the left eye image 501 shown in FIG. 18 (a) and the right eye image 502 shown in FIG. 18 (b), each is thinned by half in the horizontal direction, and one mixed image 503 shown in FIG. 18 (c). Make and record. At the time of reproduction, the mixed image 503 is rearranged to create a composite image as shown in FIG.
[0010]
Although not limited to a three-dimensional image, Patent Document 2 discloses a method of storing additional information for identifying a thumbnail image and displaying the additional information on the thumbnail image on a display device. ing.
[0011]
[Patent Document 1]
Japanese Patent Laid-Open No. 11-41627
[Patent Document 2]
JP 2001-337994 A
[0012]
[Problems to be solved by the invention]
As described above, in the conventional three-dimensional display system, recording with a fixed recording data format is performed so as to be suitable for the display method determined on the reproducing apparatus side, and the recording data should be versatile. Is not considered.
[0013]
In addition to the display method, there are various information necessary for three-dimensional display such as the image thinning method and the number of viewpoints in the so-called “multi-view method”, but when the display method is single, such information is recorded as recorded data. Not recorded. If the same display method is always used, it is not necessary to record such information. However, the versatility of recorded data is greatly impaired. For example, even when recording data for the parallax barrier method (or lenticular method), the left eye image and the right eye image can be recorded as separate sequences, or the left as shown in FIG. It is also possible to record a mixed image in which the eye image and the right eye image are arranged on the left and right sides of the screen half, or a composite image in which the left eye image and the right eye image are arranged every other pixel in the horizontal direction as shown in FIG. Can also be recorded. Of course, if the recording format is different, the processing method for displaying this will also be different, but since it is not possible to know which format was recorded from the recorded data, when a third party obtains the data, which There is a problem that it is not known whether the display should be performed by such processing.
[0014]
Further, in the conventional technique, it has not been considered that image data of each viewpoint is recorded independently of each other, and only an image of a desired viewpoint is easily read and reproduced.
[0015]
Further, the conventional technology does not sufficiently consider compatibility with existing devices. That is, in the system disclosed in Patent Document 2, only a display system capable of interpreting the additional information is considered, and the additional information is not useful in a display system that cannot interpret the additional information.
[0016]
The present invention has been made to solve the above-described problems, and its purpose is to provide versatility to image data for three-dimensional display and to efficiently image an arbitrary viewpoint. An object of the present invention is to provide an image data creation device that enables selection and an image data reproduction device that reproduces the data.
[0017]
[Means for Solving the Problems]
  The present invention inputs image information taken from a plurality of viewpoints in a predetermined order, creates a main image for generating a main image for stereoscopic display, and the number of viewpoints as information when the main image is stereoscopically displayed, Whether or not the image information is combinedas well asWith reductionNothingA three-dimensional control information generating unit that generates three-dimensional control information including at least one of them, a thumbnail image generating unit that generates a thumbnail image from the main image, the main image, the thumbnail image, and the three-dimensional control information And a multiplexing unit that multiplexes the image, and the thumbnail image generation unit extracts the image for one viewpoint from the main image to generate the thumbnail image.
[0018]
  Here, the thumbnail image creation unitAboveThumbnail imageInEmbed a symbol indicating that the image is for stereoscopic display.AndFeatures.
  In addition, the present invention inputs image information taken from a plurality of viewpoints in a predetermined order and creates a main image creation unit that creates a main image for stereoscopic display, and information that is information when stereoscopically displaying the main image. A three-dimensional control information creating unit that creates three-dimensional control information including at least one of a score, presence / absence of combination of image information, and presence / absence of reduction; a thumbnail image creating unit that creates a thumbnail image from the main image; A multiplexing unit that multiplexes the main image, the thumbnail image, and the three-dimensional control information, wherein the thumbnail image creation unit extracts a plurality of viewpoint images from the main image and combines them. And a second image obtained by extracting an image for one viewpoint from the main image, and creating a thumbnail image in the form of a picture-in-picture.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0021]
FIG. 1 is a block diagram showing a configuration of an image data creation apparatus according to an embodiment of the present invention. In FIG. 1, an image data creation device 100 determines a layout method of an image K from an image 1 of multiple viewpoints (number of viewpoints K, where K is an integer equal to or larger than 2), and combines these adjacently combined. Image combining unit 101 for creating an image, whether to combine image K from image 1 (whether or not combined), whether or not to reduce image K from image 1 (whether or not to reduce), used for two-dimensional display The control unit 102 that designates the image to be selected (2D selection), the number of viewpoints and the arrangement order of the images, the presence / absence of reduction, the presence / absence of combination, the image arrangement method, the information on 2D selection and the number of viewpoints, and 3D information The 3D information creating unit 103 to be created and a unit for accessing a recording medium and a communication line are provided, and the multiplexing unit 104 multiplexes image information and 3D information and outputs image data.
[0022]
The operation of the image data creation device 100 configured as described above will be described.
[0023]
An image signal composed of continuous frames is input to the image data creation device for each frame. Here, M image pickup devices for inputting an image to the image data creation device 100 are arranged in a grid pattern in a horizontal direction with M pieces in the horizontal direction and N pieces in the vertical direction, and each image pickup device has a number (viewpoint). Number is assigned (where M and N are integers of 1 or more).
[0024]
FIG. 2 shows an installation example in the case where the number of viewpoints is eight (a diagram in which the installed imaging device is looked down obliquely downward). Here, viewpoint numbers are assigned from left to right and from top to bottom. That is, the imaging device 301 is 1, the imaging device 302 is 2, the imaging device 303 is 3, the imaging device 304 is 4, and similarly, the imaging devices 305 to 308 are 5 to 8. Note that an image photographed by the imaging device of viewpoint number k through this embodiment is referred to as an image k (where k is an integer equal to or greater than 1).
[0025]
The control unit 102 specifies the presence / absence of reduction, the presence / absence of combination, 2D selection, the number of viewpoints M in the horizontal direction, the number N of viewpoints in the vertical direction, and the arrangement order of the images. Here, the presence / absence of reduction takes one of the values “with reduction” or “without reduction”, and the presence / absence of connection takes the value of “without connection” or “with connection”. The 2D selection takes a viewpoint number or a value of “no designation”. The arrangement order of images specifies the sequence of images by viewpoint number. The number of viewpoints is M = 4 and N = 2 in the example of FIG.
[0026]
When the presence / absence of the combination input from the control unit 102 indicates “no connection”, the image combining unit 101 serializes the image K from the image 1 input in parallel according to the arrangement order of the images specified by the control unit 102. Shall be output. Alternatively, when an image of each viewpoint is output, the viewpoint number designated by 2D selection may always be output first.
[0027]
Further, when the presence / absence of combination indicates “with combination”, the image combining unit 101 selects an arrangement method from the input image 1 to the image K. The arrangement method can be selected from three arrangements: a horizontal arrangement in which multi-viewpoint images are arranged horizontally, a vertical arrangement in which multi-viewpoint images are arranged up and down, and a lattice arrangement in which both horizontal and vertical directions are arranged.
[0028]
Here, the image arrangement method may or may not coincide with the imaging device installation method. When matching the image arrangement with the installation method of the imaging unit, the vertical arrangement is obtained when M = 1 and N ≧ 2, the horizontal arrangement is obtained when M ≧ 2 and N = 1, and the lattice arrangement is obtained otherwise. If they are not matched, when M = 1, N ≧ 2 or M ≧ 2, N = 1, either vertical arrangement or horizontal arrangement may be selected.
[0029]
When the arrangement method is determined, the images are combined according to the arrangement order of the images input from the control unit 102. FIG. 3 shows an example of the arrangement order of images when the images photographed by the imaging apparatus shown in FIG. 2 are arranged in a grid pattern. In FIG. 3, one square represents an image, and the number is a viewpoint number. FIG. 3A shows a case where the arrangement order of images is designated as 1, 2, 3, 4, 5, 6, 7, and 8, and the arrangement is the same as the viewpoint number assigned to the imaging device. FIG. 3B shows a case where 2, 3, 1, 4, 6, 7, 5, 8 are designated.
[0030]
If the presence / absence of reduction input from the control unit 102 indicates “with reduction”, the input image of each viewpoint is reduced. The reduction ratio at this time is not fixed and is determined according to the number of viewpoints. That is, it is reduced to 1 / M in the horizontal direction and 1 / N in the vertical direction.
[0031]
Examples of the result of combining by the image combining unit 101 at this time are shown in FIGS. FIG. 4 shows a case of two viewpoints using only viewpoint numbers 1 and 2, for example, of FIG. 2, and FIG. 5 shows a case where all of the imaging apparatuses are used.
[0032]
4A shows the case of “no reduction” and “no connection”, FIG. 4B shows the case of “no reduction” and “with connection (horizontal arrangement)”, and FIG. 4C shows “reduction”. FIG. 4D shows the case of “with reduction” and “no connection”, and FIG. 4E shows the case of “with reduction” and “with connection (horizontal)”. 4 (f) shows the case of “with reduction” and “with connection (vertical arrangement)”. Here, in the case of “with reduction”, the resolution in the horizontal direction or the vertical direction is halved by thinning out pixels or the like.
[0033]
FIG. 5 shows an example of a grid arrangement of multi-viewpoint images. When the presence / absence of reduction indicates “no reduction”, the result is as shown in FIG. Here, H and V are the number of pixels in the horizontal direction and the number of vertical lines in each viewpoint image before reduction, respectively. When the presence or absence of reduction indicates “with reduction”, the result is as shown in FIG. The reduction ratio is 1/4 in the horizontal direction and 1/2 in the vertical direction. The image size after reduction is horizontal H pixels and vertical V lines, and is the same as the image size of each viewpoint before reduction. The arrangement of the images coincides with the installation method of the imaging apparatus, and four images are arranged in the horizontal direction and two images are arranged in the vertical direction. Here, the numbers attached to the images indicate viewpoint numbers. The images are arranged in ascending order of viewpoint numbers. The upper left is an image captured by the imaging unit 301 (viewpoint number 1), and the lower right is an image captured by the imaging unit 308 (viewpoint number 8).
[0034]
Here, the reduction ratio in the horizontal direction and the vertical direction has been described as being fixed, but these may be variable. If variable, the reduction ratio is recorded in 3D information. Further, when the presence / absence of combination indicates “no combination”, the image may be specified for each viewpoint image.
[0035]
The 3D information creation unit 103 creates 3D information by formatting presence / absence of reduction, presence / absence of combination, 2D selection, the number of viewpoints in the horizontal direction and the number of viewpoints in the vertical direction, the arrangement order of images, and the arrangement method.
[0036]
An example of the 3D information at this time is shown in FIG. Here, the image order indicates that the arrangement of images is “viewpoint number order” or “arbitrary order”. After this, a plurality of viewpoint numbers are recorded. These viewpoint numbers indicate how the images are arranged in the combined image when the presence / absence of the combination indicates “with combination”. In the example of FIG. 3B, the first viewpoint number is 2, and the order is 3, 1, 4, 6, 7, 5, 8 in the following. Further, when the presence / absence of combination indicates “no combination”, the order of multiplexed image information is indicated. When the image order indicates “order of viewpoint number”, these viewpoint numbers may be omitted. When creating 3D information, the set value may be used as it is, or may be encoded by fixed length encoding or variable length encoding.
[0037]
The multiplexing unit 104 converts the image information, 3D information, and management information into a predetermined format and outputs the converted information. When images are not combined, the output order from the image combining unit 101 is the image arrangement order specified by the control unit 102 as described above, so the order in which the image information is multiplexed is also the order in which the images are arranged. It becomes. Although not shown in FIG. 1, when voice and music are multiplexed, those data are also multiplexed by the multiplexing unit 104.
[0038]
Here, the encoded image information may be input to the multiplexing unit 104. FIG. 19 shows the configuration of the image data creation device 110 in such a case. The image data creation device 110 is different from the image data creation device 100 of FIG.
[0039]
An output destination of the multiplexing unit 104 is connected to a recording device such as an IC memory, a magneto-optical disk, a magnetic tape, and a hard disk, and a communication device such as a LAN and a modem. Here, it is assumed that an IC memory is connected to the multiplexing unit 104. Hereinafter, a recording format in this case will be described.
[0040]
In general, when an IC memory is used as a recording medium, a file system such as FAT (File Allocation Table) is constructed on the IC memory, and data is recorded as a file. As the file format used here, an existing format may be used, or a newly defined unique format may be used.
[0041]
FIG. 7 is a diagram showing a file format for recording image data. In FIG. 7, data is recorded in a file in the order from the top to the bottom of the figure. FIG. 7A shows an example in which an existing format is used, and FIG. 7B shows an example in which a new format is used.
[0042]
When an existing format is used, 3D information is generally recorded as a part of an existing header portion using a mechanism for extending a header portion prepared in the existing format. Here, the extended header is called an extension header. For example, the file header indicates an application data segment in JPEG, defines a new application data segment, and records 3D information. In MPEG-4, the file header indicates Visual Object Sequence or / and Video Object Layer, and 3D information is recorded as user data in these.
[0043]
If an existing format is used, a commonly used extension is used as it is. For example, in the case of a JPEG file, generally. If the extension jpg is an MPEG file, generally mpg or. When the extension of mp4 is WMV (Windows (R) Media Video), generally. The extension wmv is used. In this way, even a conventional playback device that does not have a three-dimensional image display function can recognize the file as an existing format and display it as a two-dimensional image.
[0044]
On the other hand, when a new format is used, for example, 3D information is recorded at the head of the file as shown in FIG. In addition, a unique extension that can be distinguished from an existing format file is attached so that the file is a new format file. The management information in FIGS. 7A and 7B is used for recording information that is not directly related to the three-dimensional image, such as the creation date and the creator.
[0045]
First, a method of storing multi-viewpoint images will be described in the case where the presence / absence of combination indicates “no combination”. When the existing format shown in FIG. 7A is used as the file format, a plurality of multi-viewpoint images are separately recorded in the image information area of FIG. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. A storage example at this time is shown in FIG. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. At this time, each frame may be encoded independently as in Motion JPEG, or the difference may be encoded using inter-frame prediction as in MPEG-4.
[0046]
A case where the new format shown in FIG. 7B is used as the file format will be described. In the new format, there are cases where an existing format (JPEG, bitmap, etc.) is used for the file header and the image information portion in FIG. 7B, and a completely new unique format is used. Therefore, in order to distinguish between these formats, type information (referred to as an image type) is recorded in the 3D information.
[0047]
As for the recording method of the image data, a plurality of multi-viewpoint image data is recorded in the image information area of FIG. In the case of a moving image, a plurality of pieces of frame data are recorded for each viewpoint. FIG. 9 shows an example of storage and 3D information in the case where the presence / absence of combination indicates “no combination”. The K file headers and the image information in FIG. 9 are in a format that can be recognized as a single file in an existing format. In other words, taking a bitmap file as an example, images 1 to K are recorded as independent bitmap files, and 3D information, management information, and the bitmap file of image K are connected in order from the bitmap file of image 1 It has become a format.
[0048]
When the presence or absence of combination indicates “no combination”, multi-viewpoint image information may be recorded in a file as separate image data. At this time, the image information of each viewpoint is recorded in the format shown in FIG. 7A for the existing format and in the format shown in FIG. 7B for the new format. When recording the file, only the existing format may be used, only the new format may be used, or both may be used in combination. FIG. 10 shows an example when the existing format is used. Since the number of viewpoints is K, K files are created.
[0049]
At this time, the 3D information creation unit 103 creates 3D information corresponding to the number of viewpoints. Here, since there is only one image information recorded in one file, the image order is omitted, and the viewpoint number is recorded to indicate which viewpoint number each file corresponds to. An example of the 3D information at this time is shown in FIG.
[0050]
FIG. 12 is a diagram illustrating an example of 3D information when the number of viewpoints is two. Here, it is assumed that viewpoint numbers 1 and 2 are used as the imaging device among the imaging devices shown in FIG. Since the number of viewpoints is 2, two pieces of 3D information shown in FIGS. 12A and 12B are created. The left side connected by “=” indicates an item of 3D information, and the right side indicates the set value. Among the items, the same values are recorded in FIG. 12A and FIG. 12B for the arrangement method, the number of viewpoints in the horizontal direction, the number of viewpoints in the vertical direction, and 2D selection. The other items are different. FIG. 12A shows the image of viewpoint number 1 and is not reduced, and FIG. 12B shows the image of viewpoint number 2 and is reduced. Yes.
[0051]
Furthermore, when recording multi-viewpoint image information as separate files, it is necessary to identify the corresponding viewpoint file among the multi-viewpoint images captured by the same imaging device from among the plurality of recorded files. is there. Here, for example, in the multiplexing unit 104 of FIG. 1, information for identifying the file of each viewpoint image captured by the same imaging device may be recorded in the management information described above. FIG. 13 is a diagram showing an example of the management information at this time, in which information on what name the multi-viewpoint image file is recorded on the recording medium is recorded. In FIG. 13, the file structure takes values of “separated” and “integrated”, and “separated” indicates that each viewpoint image is recorded as a separate file. Indicates that it is recorded in one file. The viewpoint number and the file name are recorded in association with each other so that the file name of the corresponding image data can be known from the viewpoint number.
[0052]
Alternatively, a file name may be assigned according to a predetermined naming rule so that it can be seen that the set of multi-viewpoint images captured by the imaging device is the same. For example, when the number of viewpoints is 2, the set file names of image 1 and image 2 are “stereo1_1.jpg” and “stereo1_2.jpg”, respectively, and the file names of another set are “stereo2_1.jpg”. , “Stereo2_2.jpg”.
[0053]
By the way, 3D information or management information recorded in each file has an overlapping part. For example, in the example of 3D information shown in FIG. 12, the same information is shown except for the presence / absence of reduction and the viewpoint number. In the example of management information shown in FIG. 13, the contents of the management information are common to all files.
[0054]
Therefore, these pieces of common information (common information) may be recorded only in one file, and only information specific to image data (individual information) may be recorded in other files. An example at this time is shown in FIG. 14A shows a file in which individual information and common information are recorded, and FIG. 14B shows a file in which only individual information is recorded. In this example, the file in FIG. 14 (a) is recorded in the existing format, the file in FIG. 14 (b) is recorded in the new format, and the file in which the common information is recorded is recorded only by other individual information according to the file extension. It can be identified from the recorded file. Further, as shown in FIG. 14B, the file name of a file in which common information is recorded may be recorded as individual management information. This makes it easy to specify a file in which common information is recorded from a file in which only individual information is recorded.
[0055]
In addition to the above, the method for identifying these files allows the image file of each viewpoint to be identified using the above-mentioned naming rules, and allows a specific viewpoint number such as a viewpoint number designated by 2D selection. Common information may be recorded in the image file.
[0056]
Alternatively, overlapping information may be combined into a management file, and only unique information may be recorded in each image file. The management file uses a unique extension different from that of the image file.
[0057]
Furthermore, in the above-described file system such as FAT, a directory is used to collectively manage files. One set of created image files for each viewpoint (a management file may be included if it exists) may be recorded in the same directory.
[0058]
When the presence / absence of the combination indicates “with combination”, the image information of the combined image is recorded in the image information area of FIGS. 7A and 7B. In the case of a moving image, a plurality of images obtained by combining corresponding frames of a multi-view image are recorded.
[0059]
In the above embodiment, the way of assigning the viewpoint numbers is fixed and the image arrangement order can be arbitrarily changed. However, the arrangement order of the images is fixed and the viewpoint number is preferentially changed. It doesn't matter. Furthermore, the installation method of the imaging device can be not only a lattice arrangement but also an arbitrary arrangement. In this case, a reference imaging device (with a viewpoint number of 1) is selected, and the position is expressed in a coordinate system with this as the origin. In the 3D information, the position coordinates of the imaging device at each viewpoint are recorded in the order of viewpoint numbers.
[0060]
In the above embodiment, in the case of a new format file, 3D information and management information are recorded at the beginning of the file. However, the new format file is not limited to this, and the storage location thereof is not limited to this. May be after the file header or after the image information, or may be the same as the existing format shown in FIG.
[0061]
Next, a playback device for displaying the image data created by the image data creation device 100 as a three-dimensional image will be described.
[0062]
FIG. 15 is a block diagram showing a configuration of the image data reproducing apparatus according to the embodiment of the present invention. In FIG. 15, the image data reproduction device 200 includes a demultiplexer 201, a 3D information analyzer 202, and an image converter 203.
[0063]
The demultiplexer 201 reads image data multiplexed in a predetermined format from a recording device or a communication device, and separates it into image information, 3D information, and management information. Although not shown in FIG. 15, when voice and music are multiplexed, those data are also separated by the demultiplexer 201.
[0064]
The 3D information analysis unit 202 analyzes 3D information in a predetermined format and extracts a setting value for each item.
[0065]
The image conversion unit 203 is connected to a display device having a different display format, such as a two-dimensional display device using a normal cathode ray tube or a liquid crystal panel, a stereoscopic display device using a lenticular method, a parallax barrier method, a time division method, or the like. The
[0066]
The operation of the image data reproducing apparatus 200 configured as described above will be described. Here, it is assumed that an IC memory is connected to the demultiplexer 201. As described above, the image file in the existing format and the new format, and the management file are recorded in the IC memory. The image file and the management file can be distinguished by the file extension. Here, it is assumed that the user selects one image file or management file by a designation unit (not shown).
[0067]
First, a case where the selected file is an image file will be described. In this case, since the existing format and the new format can be distinguished by the extension of the file, the demultiplexer 201 determines that the file header is the file header when the file to be reproduced is the existing format file shown in FIG. 3D information is read from the extended area. In the case of the new format shown in FIG. 7B, 3D information is read from the beginning of the file.
[0068]
The 3D information analysis unit 202 analyzes the 3D information and extracts setting values such as the presence / absence of coupling, the presence / absence of reduction, the number of viewpoints, the arrangement method, and 2D selection. Further, the viewpoint number of an image to be displayed as a three-dimensional image is determined. In order to display multi-viewpoint image data as a three-dimensional image, two viewpoints with parallax may be selected from the multi-viewpoint images to form a left-eye image and a right-eye image. For example, in the case of the image data recorded by the photographing apparatus of FIG. 2, it is possible to select from among 1 and 2, 1 and 3, 1 and 4, 2 and 3, 2 and 4 arranged in the horizontal direction. . Further, if the display is rotated by 90 °, it is possible to display it as a three-dimensional image even if a set of 1 and 5, 2, and 6, 3, and 7, and 4 and 8 is selected.
[0069]
When the presence / absence of combination indicates “no combination”, the image information of only one viewpoint is recorded in the selected file as shown in FIG. 10, or each viewpoint that is not combined as shown in FIG. Either all of the image information is recorded.
[0070]
In order to distinguish these, as a result of analyzing the 3D information, it is only necessary to check whether the image order information is included in the 3D information. If the image order is included in the 3D information, it is the latter. A combination of arbitrary viewpoints that can be stereoscopically viewed is selected from the first viewpoint number to the Kth viewpoint number recorded in the 3D information. Otherwise, it is the former, and the viewpoint number that enables stereoscopic viewing is selected by combining with the viewpoint number i (where i is an integer of 1 or more) recorded in the 3D information. The selected viewpoint number is output to the demultiplexer 201.
[0071]
The demultiplexing unit 201 reads out the image information of the input viewpoint number from the file and outputs it to the image conversion unit 203. When the image information of the input viewpoint number is not recorded in the file, the file in which the image information is recorded is found from the management information and the naming rule as described above and read out.
[0072]
Alternatively, the file configuration in the management information may be referred to in order to distinguish whether the file recording is only one viewpoint or all viewpoints. If the file structure is “separated”, image information of one viewpoint is recorded, and if it is “integrated”, image information of all viewpoints is recorded.
[0073]
On the other hand, when the presence / absence of combination is “with combination”, since only one combined image is recorded in the file, an arbitrary viewpoint number can be selected. In this case, the viewpoint number is output to the image conversion unit 203.
[0074]
Here, when image information is encoded, decoding is performed after demultiplexing. FIG. 20 shows the configuration of the image data reproducing apparatus 210 in such a case. The image data reproducing apparatus 210 is different from the image data reproducing apparatus 200 of FIG.
[0075]
The image conversion unit 203 converts the image information separated by the demultiplexing unit 201 according to the presence / absence of combination, the presence / absence of reduction, the number of viewpoints, the arrangement method, 2D selection, and viewpoint number input by the 3D information analysis unit 202. Convert to display format. At this time, if the presence / absence of the combination indicates “no combination”, the conversion is performed when the image information designated by the viewpoint number is ready. If image information is not available due to reasons such as deletion of a file, the image with the closest viewpoint number may be substituted, or two-dimensional display may be performed. When the presence / absence of the combination is “with combination”, the image of the viewpoint designated by the viewpoint number is cut out from the combined image and then converted.
[0076]
For example, when a parallax barrier display device is connected to the image data reproducing device 200, the format shown in FIG. 4E is the easiest to handle. It is only necessary to arrange the left and right images every other pixel in the horizontal direction. In the case of FIG. 4B, the left and right images are thinned out in half in the horizontal direction before being arranged every other pixel in the horizontal direction. In either case, since 3D information is added to the image data, it can be converted into a display format suitable for the display device.
[0077]
When encoding image information, taking the parallax barrier method as an example, even if an image to be displayed is as shown in FIG. 17B, an image arranged as shown in FIG. 4E is encoded. Encoding efficiency is greatly improved. This is because the correlation between adjacent pixels is higher in the state of FIG. 4E than in the state of FIG. In order to use the encoded image of FIG. 4E for the parallax barrier method, the image conversion unit 203 uses the image conversion unit 203 as shown in FIG. You can rearrange them.
[0078]
When the display is switched to the two-dimensional display during the display of the three-dimensional image, the image of the viewpoint number designated by the 2D selection is displayed. At this time, when the presence / absence of reduction is “no reduction”, the image is displayed as it is, and when it is “reduction”, the image is enlarged twice. However, when the image is not being displayed, any one of the displayed images may be selected and displayed. The selection method is used for, for example, the one with the smallest viewpoint number, the largest one, the one closest to the viewpoint number specified by 2D selection in the installation of the imaging device or the layout of the combined image, and the left eye image. And those used for the right-eye image, and are not particularly limited here. When the 2D selection does not specify an image to be used for 2D display, the image to be displayed is selected by a predetermined method.
[0079]
When a 2D display device is connected, an image designated by 2D selection is displayed. The way of display is the same as the two-dimensional display in the three-dimensional display device.
[0080]
In addition, when the file selected by the user is a management file, the presence / absence of the combination recorded in the 3D information indicates “no connection”, but any combination of stereoscopic view point numbers can be selected. That's fine. Since the operation from reading the image data from the image file to conversion into the display format is the same as described above, the description thereof is omitted here.
[0081]
In the above embodiment, the case where the viewpoint number and the file name are recorded in the management information has been described. However, the following may be used. That is, in the image data creation device 100, an identification number indicating that the image information is a set of multi-viewpoint images captured by the same imaging device is recorded in the common information and the individual information, and the image data reproduction device 200 is recorded. The image information is read out only when the file name matches the identification number. One identification number may be assigned to a set of multi-viewpoint images, or a different number may be assigned to each viewpoint. By doing so, it is possible to prevent malfunctions due to fake file names.
[0082]
As described above, in a conventional reproducing apparatus that handles data created by various 3D image photographing methods in a unified manner and does not have a 3D image display function, it is possible to display a 2D image normally. Since it becomes possible, it can have versatility.
[0083]
By the way, as shown in FIG. 21A, a thumbnail image may be recorded in an image file. The DCF (Design Rule for Camera File system) standard established to ensure image file compatibility is intended to maintain minimum playback compatibility even when the main image (image information) cannot be played back. , Storage of thumbnail images is obligatory. In the DCF standard, the number of pixels of the main image is not specified, but the number of pixels of the thumbnail image is limited to one type of 160 × 120 pixels.
[0084]
In order to record information for three-dimensional display in a file in accordance with such an idea, a form as shown in FIG. That is, it is assumed that 3D information is added to the image file in the format of FIG. 21A, and the main image in FIG. 21B is a 3D image.
[0085]
Here, as the main image in FIG. 21 (b), for example, as shown in FIG. 4 (e), an image in which two viewpoint images are reduced in half in the horizontal direction and combined is stored. It is assumed that the number of pixels is 640 pixels × 480 pixels. Further, assuming that an image obtained by reducing a main image to 160 pixels × 120 pixels as it is is used as a thumbnail image, an image included in this file is as shown in FIG. In this case, it can be said that both the main image and the thumbnail image are 3D images.
[0086]
Therefore, when there is an image as shown in FIG. 22 (a), in an image data reproducing apparatus capable of interpreting 3D information and capable of 3D display, information indicating that this file stores 3D. Is overwritten on the thumbnail image, thumbnail display as shown in FIG. 23C becomes possible. In addition, thumbnails can be displayed in 3D by performing image conversion performed when the main image is displayed in 3D on the thumbnail image. Furthermore, it is possible to further improve the visibility by giving 3D display by giving parallax to a symbol (for example, a character “3D” in FIG. 23C) itself indicating a 3D image.
[0087]
When thumbnails are displayed in 3D, a thumbnail image as shown in FIG. 22A can be used to perform drawing at a higher speed than when the decoded main image is reduced and displayed in 3D. This is because the thumbnail image is small, so that it is not necessary to once decode a large image such as the main image, and the decoding process can be performed at high speed.
[0088]
Here, since the thumbnail image does not have to be a reduced version of the main image as it is, a combination of the main image and the thumbnail image as shown in FIG. The main image in FIG. 22B is the same as that in FIG. 22A, but the thumbnail image is obtained by extracting only one viewpoint image from the main image and reducing the whole to 160 pixels × 120 pixels. Yes. In FIG. 22B, the main image is reduced in half in the horizontal direction, that is, the ratio of the length in the horizontal direction to the vertical direction is 1: 2, so the thumbnail image is an image for one viewpoint. It is assumed that it has been doubled in the horizontal direction after being taken out.
[0089]
When there is an image as shown in FIG. 22B, 3D information can be interpreted, and in an image data reproducing apparatus capable of 3D display, information indicating that this file stores 3D is displayed as a thumbnail image. By overwriting on the thumbnail, thumbnail display as shown in FIG. 23B is possible. As shown in FIG. 22B, only the image for one viewpoint in the main image is extracted and used as a thumbnail image, so that thumbnail display without distortion can be quickly performed.
[0090]
Note that even in the case of a thumbnail image as shown in FIG. 22A, information indicating that 3D is stored after taking out only one viewpoint when thumbnails are displayed and enlarged twice in the horizontal direction is displayed as thumbnails. By overwriting the image, thumbnail display as shown in FIG. 23B is possible.
Incidentally, FIG. 22 shows the case where the presence / absence of the main image is “with connection”, but the main image may be “without connection” as shown in FIG. When the presence / absence of the combination of the main images is “no combination”, a reduced version of one of the plurality of images recorded as the main image may be recorded as a thumbnail image. In addition, the “2D selection” information has been described above. However, whether the main image is combined or not is “with connection” or “without connection”, the information for one viewpoint specified by “2D selection” is used. Images may be taken out to create thumbnail images, or thumbnails may be displayed.
[0091]
Now, if it is an image data reproducing apparatus capable of interpreting 3D information as described above, it can be processed appropriately even if a file as shown in FIG. 21B is received, but an old type that cannot interpret 3D information here. Consider an image data reproducing apparatus. It is assumed that the 3D display cannot be connected to the image data reproducing apparatus without the 3D information analysis unit 202 and the image conversion unit 203 in FIG. Since such an image data reproducing apparatus cannot interpret 3D information, it does not know whether the image information stored in this file is a 2D image or a 3D image, and if it is a 3D image by some means. Even if it is understood, the image information cannot be correctly reproduced because the image conversion unit is not provided.
[0092]
Even in such a case, if thumbnail images are recorded in accordance with the idea such as the DCF standard, it can be expected that only thumbnail images can be displayed.
Therefore, in the present invention, a thumbnail image in which a symbol indicating that this file stores a 3D image is embedded as a thumbnail image is recorded as a thumbnail image. For example, as shown in FIG. 23 (a), an image in which a pictograph “3D” is superimposed on the lower right of the image may be used as a thumbnail image, or as shown in FIG. 23 (b), the character “3D” is placed at the center of the screen. A thumbnail image embedded in the form of a watermark may be used. Further, even if an image as shown in FIG. 23C is recorded as a thumbnail image, it can be recognized that at least this file stores a 3D image.
When a symbol representing 3D is embedded in a thumbnail image, the position and size to be embedded may be designated by the user at the time of recording. Alternatively, a plurality of symbols may be prepared so that an arbitrary symbol can be selected from them. Further, when creating a thumbnail image, the background area of the image may be automatically determined, and a symbol may be written in the background area. In addition, when a symbol representing 3D is automatically embedded in a thumbnail image as described above, the recorded thumbnail image is confirmed, and if the position or size of the symbol is not preferable, the thumbnail image is recreated. You may be able to do that.
[0093]
An example of the combination of the main image and thumbnail image of the image file recorded in this way is shown in FIG. Here, as in the case of FIG. 22, as the main image, for example, as shown in FIG. 4 (e), it is assumed that an image in which two viewpoint images are reduced in the horizontal direction by ½ and combined is stored. It is assumed that the number of pixels is 640 pixels × 480 pixels. Further, as the thumbnail image, if an image obtained by embedding a symbol indicating that this file is a 3D image stored in an image obtained by reducing the main image to 160 pixels × 120 pixels is used, the thumbnail image of FIG. When storing such thumbnail images, the combination is as shown in FIG. 24A, and when storing thumbnail images as in FIG. 23B, the combination is as shown in FIG.
[0094]
The symbol itself may be text, a mark, or a specific image as long as it indicates 3D. There is no limitation on the position of the symbol in the thumbnail image. However, in any case, it needs to be recorded as a part of the thumbnail image. Conversely, these symbols are not recorded separately from the thumbnail images.
[0095]
This is an important point. In order to perform meaningful thumbnail display in an image data reproducing apparatus that cannot interpret 3D information, it is necessary to record an image as shown in FIG. 23 in a file in advance as a thumbnail image. In this way, the image data reproducing apparatus can check the contents of the 2D image by reproducing the thumbnail image just like the 2D image file, even if it is a 3D image file. Files and 3D image files can be distinguished.
By the way, even an image data reproducing apparatus capable of interpreting 3D information does not necessarily support all data formats. That is, for example, an image data reproduction apparatus that supports only 4 eye 3D images cannot display correctly even if 2 eye 3D image data is received. In such a case, the image content can be confirmed by displaying the thumbnail image. Since the image data reproducing apparatus capable of interpreting 3D information can determine whether or not the main image included in the file can be correctly reproduced by interpreting the 3D information, the thumbnail image is displayed when the data format cannot be correctly reproduced. In addition to the display, a message such as “It is an unsupported 3D data format” or the like may be displayed.
[0096]
Here, FIG. 28 shows an example of an image data creation apparatus that records thumbnail images in a file. The image data creation device 120 of FIG. 28 includes a thumbnail image creation unit 106, which multiplexes thumbnail data together with encoded data (or uncompressed image information) and 3D information in the multiplexing unit 104 to create a file. The output destination of the file may be a recording device such as an IC memory, a magneto-optical disk, a magnetic tape, or a hard disk, or a communication device such as a LAN or a modem. In the DCF standard, the size of a thumbnail image is defined as 160 pixels × 120 pixels, but a general thumbnail image is not limited to this size.
[0097]
Next, FIG. 26 shows an example of an image data reproducing apparatus that performs thumbnail display as shown in FIG. 23 from thumbnail images as shown in FIG. In FIG. 26, parts having the same functions as those of the image data reproducing apparatus 210 shown in FIG. In the image data reproduction device 220 shown in FIG. 26, the demultiplexer 221 separates the thumbnail data. When the data to be reproduced is a 3D image, in response to an instruction from the 3D information analysis unit 202, the thumbnail generation unit 225 decodes the thumbnail data (it is not necessary to decode in the case of non-compressed data) A thumbnail indicating that the image is a 3D image is displayed in a thumbnail. When the thumbnail is displayed in 3D, the thumbnail generation unit 225 performs the same processing on the thumbnail image as the conversion performed by the image conversion unit 203 on the main image.
[0098]
FIG. 27 shows an example of an image data reproducing apparatus that can switch and output a 3D display image and a 2D display image. In FIG. 27, parts having the same functions as those of the image data reproducing device 220 shown in FIG. In the image data reproducing device 230 shown in FIG. 27, when image data obtained by encoding the main image and the thumbnail image as shown in FIG. 24 is input, the image information decoded by the decoding unit 204 is the image conversion unit 203. And is also sent to the thumbnail generation unit 225. The control unit 226 gives an instruction to the image conversion unit 203 and the thumbnail generation unit 225 to output a 3D display image or a 2D display image. When performing 2D display of thumbnails, the thumbnail generation unit 225 may output a thumbnail image as shown in FIG. 24 as it is (in this case, a 3D symbol is also displayed), or the input main image is reduced. Then, a thumbnail image that does not include a symbol indicating 3D may be output by performing the same processing as the image conversion unit 203. When performing 3D display of thumbnails, the input main image may be reduced and processing similar to that performed by the image conversion unit 203 may be performed.
[0099]
In the above description, in the case of a 3D file, a symbol indicating 3D is displayed so as to be superimposed on the thumbnail image. Conversely, for a 3D file, the thumbnail image is displayed as it is. In the case of a 2D file, a symbol indicating 2D may be displayed over the thumbnail image.
[0100]
Further, not only a 3D symbol for a 3D file is displayed so as to be superimposed on a thumbnail image, but also the information included in the 3D information such as the number of viewpoints and the viewpoint number is displayed so as to be superimposed on the thumbnail image. May be. Further, the information included in the 3D information and the symbol indicating 3D may be displayed not only on the thumbnail image but also at a predetermined position near the thumbnail image.
[0101]
FIG. 25 shows another example of thumbnail images stored in a file. A first image obtained by combining two or more viewpoint images and a second image obtained by extracting only one viewpoint from the first image are formed into a picture-in-picture form. FIG. 25A shows the first image as a parent image and the second image as a child image, but the parent image and the child image may be interchanged as shown in FIG. 25C and 25D are obtained by embedding a symbol indicating 3D in the images of FIGS. 25A and 25B, respectively. With regard to FIGS. 25C and 25D, a symbol indicating 3D may not be recorded in the thumbnail image, but may be output in a superimposed manner when displaying thumbnails. By using the thumbnail display as shown in FIG. 25, it is possible to simultaneously confirm both the image content using an image without distortion and the shape of the image actually recorded as the main image.
[0102]
【The invention's effect】
According to the present invention, when the main image is a three-dimensional image by creating three-dimensional control information for three-dimensional display of the main image and multiplexing the main image, the thumbnail image, and the three-dimensional control information. In addition, it is possible to output a thumbnail image for appropriately confirming the image content.
[0103]
According to the present invention, it is possible to display a thumbnail image three-dimensionally by creating a thumbnail image obtained by reducing the main image as it is.
[0104]
According to the present invention, it is possible to display a thumbnail image without distortion by extracting an image for one viewpoint from the main image and creating a thumbnail image.
[0105]
According to the present invention, even in a conventional image data reproducing apparatus that cannot interpret 3D control information by embedding a symbol indicating a 3D image in a thumbnail image, a thumbnail indicating that the selected file is 3D. Can be determined.
[0106]
According to the present invention, a thumbnail image in the form of a picture-in-picture is created by generating a reduced image of a main image and a reduced image obtained by extracting an image for one viewpoint from the main image. Both confirmation and confirmation of the shape of the image actually recorded as the main image can be performed simultaneously.
[0107]
According to the present invention, when the main image data is a three-dimensional image, the thumbnail data is output by superimposing a symbol indicating that it is a three-dimensional image as a thumbnail, so that the selected file is 2D or 3D. Can be identified by thumbnails.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of an image data creation device according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an installation example of an imaging apparatus in the case of multiple viewpoints.
FIG. 3 is a diagram illustrating an example of how to assign viewpoint numbers.
FIG. 4 is a diagram illustrating a coupling example in the case of 2 viewpoints.
FIG. 5 is a diagram illustrating a grid arrangement of multi-viewpoint images.
FIG. 6 is a diagram illustrating a format example of 3D information.
FIG. 7 is a diagram illustrating a file format of image data.
FIG. 8 is a diagram showing an example of a format when image data is stored in an existing format file.
FIG. 9 is a diagram showing an example of a format when image data is stored in a new format file.
FIG. 10 is a diagram illustrating an example of storing image data when multi-viewpoint image data is recorded in separate files.
FIG. 11 is a diagram illustrating a format example of 3D information.
FIG. 12 is a diagram illustrating an example of setting values of 3D information.
FIG. 13 is a diagram illustrating a format example of management information.
FIG. 14 is a diagram illustrating an example when multi-viewpoint image data is recorded in separate files.
FIG. 15 is a diagram illustrating a configuration of an image data reproduction device according to an embodiment of the present invention.
FIG. 16 is a diagram illustrating a display format of an image in a time division method.
FIG. 17 is a diagram for explaining the concept of the parallax barrier method;
FIG. 18 is a diagram for explaining an image display format in the parallax barrier method;
FIG. 19 is a diagram showing a configuration of an image data creation device according to another embodiment of the present invention.
FIG. 20 is a diagram showing a configuration of an image data reproducing device according to another embodiment of the present invention.
FIG. 21 is a diagram showing an image file format for recording thumbnail image data for three-dimensional display.
FIG. 22 is a diagram illustrating a combination of a main image and thumbnail images reduced to 160 pixels × 120 pixels.
FIG. 23 is a diagram illustrating a thumbnail image in which a symbol indicating that 3D image data is stored is embedded.
FIG. 24 is a diagram illustrating combinations of thumbnail images in which symbols indicating that main image and 3D image data are stored are embedded.
FIG. 25 is a diagram illustrating an example of thumbnails expressed in picture-in-picture.
FIG. 26 is a diagram illustrating a configuration of an image data reproduction device that reproduces a file in which thumbnail images are recorded.
FIG. 27 is a diagram illustrating a configuration of an image data reproduction device capable of switching between 3D display and 2D display.
FIG. 28 is a diagram showing a configuration of an image data creation device for recording thumbnail images in a file according to another embodiment of the present invention.
[Explanation of symbols]
100 Image data creation device
101 Image combiner
102 Control unit
103 3D information creation unit
104 Multiplexer
200 Image data playback device
201 Demultiplexer
202 3D information analysis unit
203 Image converter
301 to 308 Imaging device
401 Image display panel
402 parallax barrier
403 left eye
404 right eye
501 Left eye image
502 Right eye image

Claims (3)

A main image creation unit that inputs image information captured from a plurality of viewpoints in a predetermined order and creates a main image for stereoscopic display;
The main number of viewpoints is image information at the time of stereoscopic display and a three-dimensional control information creator for creating 3-dimensional control information including at least one of the binding presence and reduction of existence of the image information,
A thumbnail image creation unit for creating a thumbnail image from the main image;
A multiplexing unit that multiplexes the main image, the thumbnail image, and the three-dimensional control information,
The thumbnail image creation unit creates the thumbnail image by extracting an image for one viewpoint from the main image. The thumbnail image creating unit in the thumbnail image, the image data generating apparatus according to claim 1, wherein the embedding symbols indicating the image for stereoscopic display. A main image creation unit that inputs image information captured from a plurality of viewpoints in a predetermined order and creates a main image for stereoscopic display;
A three-dimensional control information creating unit that creates three-dimensional control information including at least one of the number of viewpoints, which is information when stereoscopically displaying the main image, the presence / absence of combining of the image information, and the presence / absence of reduction;
A thumbnail image creation unit for creating a thumbnail image from the main image;
A multiplexing unit that multiplexes the main image, the thumbnail image, and the three-dimensional control information,
The thumbnail image creating unit extracts a first image obtained by extracting and combining a plurality of viewpoint images from the main image, and a second image obtained by extracting one viewpoint image from the main image . An image data creation apparatus for creating a thumbnail image in a shape.

Images